HTTP 500 Errors and Object Unavailability with StorageGRID

• NetApp StorageGRID 11.9.0.12
• Object-based Storage (SW-SG-WEBSCALE-SW)
• Multi-site StorageGRID deployments with Strong Site Consistency
• Environments performing site/network isolation testing

During a planned site isolation test, customers observed a large number of HTTP 500 Internal Server Errors and object-not-available responses when accessing objects in a specific S3 bucket (e.g., “s3-prd1”, object: “synchronous-client-access-keys”). The issue occurred despite the bucket being configured with Strong Site Consistency. The following errors and log entries were recorded:

Symptoms and Log Output:

• S3 client requests failed with HTTP 500 (InternalError) and SF 403 errors during the isolation event.
• bycast.log and related logs showed:

  ERROR OBDR: RefreshLocations CassandraUnavailableException(SELECT ... FROM storagegrid.object_by_uuid WHERE uuid=? - CassandraDriverError(Unavailable): 'Cannot achieve consistency level EACH_QUORUM in DC group' DetailedInfo:[consistency:EACH_QUORUM, responses_received:0, responses_required:2])ERROR OBDR: RefreshLocations CassandraUnavailableException(SELECT ... FROM storagegrid.object_by_uuid WHERE uuid=? - CassandraDriverError(Unavailable): 'Cannot achieve consistency level QUORUM' DetailedInfo:[consistency:QUORUM, responses_received:3, responses_required:4])ERROR S3RQ: S3requesterror(afterresponse): connection=..., Resource=..., HTTPStatusCode(notsent)=500, ErrorMsg=InternalError, Details={asyncPusherror:noconsumer}