Skip to main content
NetApp Knowledge Base

StorageGRID cassandra service not starting after storage node crash or reboot

Views:
948
Visibility:
Public
Votes:
1
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

  • NetApp StorageGRID 11.3
  • NetApp StorageGRID 11.4

Issue

  • StorageGRID Cassandra service not starting after storage node crash/reboot
  • Cassandra /var/local/log/cassandra/system.log shows file preventing service from starting:
Example 1:

ERROR [main] 2020-10-27 08:53:26,730 CommitLogReplayer.java (line 399) Replay stopped. If you wish to overrid
e this error and continue starting the node ignoring commit log replay problems, specify -Dcassandra.commitlog.ignorereplayerrors=true on the command line

 
ERROR [main] 2020-10-27 08:53:26,737 JVMStabilityInspector.java (line 202) JVM state determined to be unstable.  Exiting forcefully due to:
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 505591 of commit log /var/local/lib/cassandra/commitlog/CommitLog-6-1603730646351.log, with invalid CRC. The end of segment marker should be zero.
 
Example 2:
 
ERROR [main] 2022-08-23 14:52:38,221 CommitLogReplayer.java (line 399) Replay stopped. If you wish to override this error and continue starting the node ignoring commit log replay problems, specify -Dcassandra.commitlog.ignorereplayerrors=true on the command line

ERROR [main] 2022-08-23 14:52:38,236 JVMStabilityInspector.java (line 202) JVM state determined to be unstable.  Exiting forcefully due to: org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 4857568 in Next section at 4845614 in CommitLog-6-1659952412249.log

  • The servermanager.log shows:

2022-08-23 14:55:11 +0000 | cassandra                 | starting cassandra
2022-08-23 14:55:29 +0000 | ade-exporter              | waiting for dds, waiting 30s to try again
2022-08-23 14:55:59 +0000 | ade-exporter              | waiting for dds, waiting 30s to try again
2022-08-23 14:56:29 +0000 | ade-exporter              | waiting for dds, waiting 30s to try again
2022-08-23 14:56:59 +0000 | ade-exporter              | waiting for dds, waiting 30s to try again
2022-08-23 14:57:29 +0000 | ade-exporter              | waiting for dds, waiting 30s to try again
2022-08-23 14:57:38 +0000 | cassandra                 | cassandra ended
2022-08-23 14:57:41 +0000 | cassandra                 | Too many failed attempts, entering error state
2022-08-23 14:57:41 +0000 | cassandra                 | cassandra ended

 

 

 

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.