Cassandra Reaper repairs failed after the Storage Node decommission
Applies to
- NetApp StorageGRID
- StorageGRID Storage Node
Issue
- The below output can be found by running
spreaper list-runs
on any storage node.
"creation_time": "2020-11-24T23:05:08Z",
"current_time": "2020-12-08T18:59:39Z",
"datacenters": [],
"duration": "7 days 0 hours 2 minutes 13 seconds",
"end_time": "2020-12-01T23:07:22Z",
"estimated_time_of_arrival": null,
"id": "7f8d00b0-2ea9-11eb-b76b-d7a5b22a5393",
"incremental_repair": false,
"intensity": 1.000,
"keyspace_name": "storagegrid",
"last_event": "",
"nodes": [],
"owner": "auto-scheduling",
"pause_time": null,
"repair_parallelism": "PARALLEL",
"repair_thread_count": 4,
"repair_unit_id": "dc8dbfa0-17c7-11eb-b890-676ddd59fc8a",
"segments_repaired": 0,
"start_time": "2020-11-24T23:05:08Z",
"state": "ABORTED",
The Cassandra system.log
shows consistency not being met.
ERROR[ReadRepairStage: 5361] 2024-08-21 13:43:18,370 CassandraDaemon.java (line 231) Exception in thread Thread[ReadRepairStage: 5361, 5, main]org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 3 responses.
at org.apache.cassandra.service.DataResolver$RepairMergeListener.close (DataResolver.java: 202)
at org.apache-cagsandra.db.partitions.Unt11teredPartitionIteratorg$2.c1ose (UntilteredPartitlonIterators.Java:203)
at org.apache.cassandra.db.transform.BaseIterator.close (BaseIterator.java: 92)
ac org.apache.cassandra.service.DataResolver.compareResponses (DataResolver.java: 79)
at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow (AsyncRepairCallback.java: 50)
at org.apache.cassandra.utils.WrappedRunnable.run (WrappedRunnable.java: 28)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java: 1149)
at java.util.concurrent.ThreadP001Executor$Worker.run (ThreadPoolExecutor-java: 624)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0 (NamedThreadFactory-java: 81)
at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable-java: 30)