StorageGRID foreground verification causes Cassandra out of memory and CASA alarm
Applies to
- StorageGRID 11.5 and earlier
- Foreground verification is running
- StorageGRID contains huge objects
Issue
CASA Data Store Down
alarm is detected on multiple nodesbycast.log
frequently shows the following error:
ERROR 0229 d8d1145c62625c26 FGVS: FGVerification_Module::execute Cassandra UnavailableException (Failed to execute cql: SELECT DISTINCT uuid, TOKEN(uuid) AS t FROM storagegrid.object_by_uuid WHERE TOKEN(uuid) > -XXXXXXXXXXXXXXXXXXX - Cassandra Driver Error(Unavailable):'Cannot achieve consistency level ALL')
servermanager.log
shows cassandra service repeats to down immediately after startup in a short time
cassandra | cassandra ended
cassandra | starting cassandra
cassandra | cassandra ended
cassandra | starting cassandra
cassandra | cassandra ended
cassandra | starting cassandra
cassandra | cassandra ended