StorageGRID responds error with Cannot achieve strong consistency during update while one site or replication link is down
Applies to
- StorageGRID
- Multi-site StorageGRID deployments with inter-site replication links down
Issue
- Applications performing GET or HEAD requests to specific S3 buckets in StorageGRID receive HTTP 500 Internal Server Error responses, particularly during periods when the replication link between sites is down or unstable.
- While enabling legal hold on an object (PutObjectLegalHold), updating object retention (PutObjectRetention) or setting a bucket’s object lock configuration (PutBucketObjectLockConfiguration), client software receives error "Cannot achieve strong consistency during update error"
bycast.log shows:
Feb 14 04:09:54 sg-sn01-cdc ADE: |12534264 2081719201 S3RQ ^RDY 2024-02-14T04:09:54.585354| NOTICE 0103 ec975d8c514f5703 S3RQ: S3 error response: RequestId=1707883790968210, TraceId=ec975d8c514f5703, Resource=/daxxx-atnf/000061xxv0xx/data/?legal-hold, HTTP Method=PUT, HTTP Status Code=500, X-Forwarded-For: '1xx.83.xxx.xx', ErrorMsg=InternalError, ErrorType=Client, CustomErrorMessage={Cannot achieve strong consistency during update.}, Details={BEGIN UPDATE storagegrid.object_by_uuid USING TIMESTAMP ? SET object_lock_legal_hold = ? WHERE uuid = ?; UPDATE storagegrid.object_by_uuid SET last_access_time = ? WHERE uuid = ?; APPLY BATCH - Cassandra Driver Error(Unavailable):'Cannot achieve consistency level EACH_QUORUM in DC group20' Detailed Info:[consistency: EACH_QUORUM, responses_received: 0, responses_required: 2]}
