StorageGRID cassandra service not starting after storage node crash or reboot
Applies to
- NetApp StorageGRID 11.3
- NetApp StorageGRID 11.4
Issue
- StorageGRID Cassandra service not starting after storage node crash/reboot
- Cassandra
/var/local/log/cassandra/system.log
shows file preventing service from starting:
ERROR [main] 2020-10-27 08:53:26,730 CommitLogReplayer.java (line 399) Replay stopped. If you wish to overrid
e this error and continue starting the node ignoring commit log replay problems, specify -Dcassandra.commitlog.ignorereplayerrors=true on the command line
ERROR [main] 2020-10-27 08:53:26,737 JVMStabilityInspector.java (line 202) JVM state determined to be unstable. Exiting forcefully due to:
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 505591 of commit log /var/local/lib/cassandra/commitlog/CommitLog-6-1603730646351.log, with invalid CRC. The end of segment marker should be zero.
ERROR [main] 2022-08-23 14:52:38,221 CommitLogReplayer.java (line 399) Replay stopped. If you wish to override this error and continue starting the node ignoring commit log replay problems, specify -Dcassandra.commitlog.ignorereplayerrors=true on the command line
ERROR [main] 2022-08-23 14:52:38,236 JVMStabilityInspector.java (line 202) JVM state determined to be unstable. Exiting forcefully due to: org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 4857568 in Next section at 4845614 in CommitLog-6-1659952412249.log
- The
servermanager.log
shows:
2022-08-23 14:55:11 +0000 | cassandra | starting cassandra
2022-08-23 14:55:29 +0000 | ade-exporter | waiting for dds, waiting 30s to try again
2022-08-23 14:55:59 +0000 | ade-exporter | waiting for dds, waiting 30s to try again
2022-08-23 14:56:29 +0000 | ade-exporter | waiting for dds, waiting 30s to try again
2022-08-23 14:56:59 +0000 | ade-exporter | waiting for dds, waiting 30s to try again
2022-08-23 14:57:29 +0000 | ade-exporter | waiting for dds, waiting 30s to try again
2022-08-23 14:57:38 +0000 | cassandra | cassandra ended
2022-08-23 14:57:41 +0000 | cassandra | Too many failed attempts, entering error state
2022-08-23 14:57:41 +0000 | cassandra | cassandra ended