StorageGRID bucket experiencing high latency issue due to high temperature on nodes
Applies to
- StorageGRID 11.9.0.6
- SG5712, SG5812
Issue
- StorageGRID S3 bucket experiencing high latency issue due to high temperature on nodes.
- Grid Manager > Support > Diagnostics report
- Cassandra message drops
- high metadata read/ write latencies
- Significant increase in the Cassandra pending messages queue.
- One of affected Node holds the metadata for the affected bucket.
- SPUT requests directed to this node began to experience failures and higher latency due to
Cassandra driver timeouts
Jul1709:15:35 <NODE> ADE:|124579332013610680 S3RQ%CEA 2025-07-17T09:15:35.870550|NOTICE 125258bc6c3f27e0d1bd S3RQ: EVENT_PROCESS_CREATE-connection=1752743735847193 method=PUT name= auth= clientIP=Jul1709:15:42 <NODE> ADE:|124579332013610680S3RQ^RDY 2025-07-17T09:15:42.812258| WARNING 027758bc6c3f27e0d1bd S3RQ: S3Request_Module::process CreateCassandraTimeoutException(posSuc:SELECT * FROMstoragegrid.bucketsWHEREbucket='<Bucket name>'-CassandraDriverError(Request timedout):'Request timedout'Jul1709:15:42 <NODE> ADE:| 124579332013610680 S3RQ^RDY2025-07-17T09:15:42.812326| NOTICE 009258bc6c3f27e0d1bd S3RQ: S3errorresponse: RequestId=1752743735847193, TraceId=58bc6c3f27e0d1bd, Resource=/<bucket/object_path>, HTTP Method=PUT, HTTP Status Code=503, X-Forwarded-For:'<IP>', ErrorMsg = ServiceUnavailable, ErrorType=Internal, CustomErrorMessage={None}, Details={posSuc:SELECT * FROMstoragegrid.buckets WHEREbucket='<Bucket name>'- CassandraDriverError (Requesttimedout): 'Requesttimedout'DetailedInfo:[N/A]}