StorageGRID returns HTTP 503 to S3 PUT request after Cassandra TimeoutException
Applies to
- StorageGRID
- Larger than normal API requests are processed
Issue
- S3 PUT request fails by HTTP 503
bycast.logshows HTTP 503 is returned afterCassandra TimeoutException
OBDI ^RDY 2023-03-31T07:40:42.290936| WARNING 0236 7521f267ce24b9c OBDI: checkForPreExistingObject Cassandra TimeoutException (Failed to execute cql at consistency TWO: SELECT event_time, event, last_access_time, object_lock_mode, object_lock_retain_until_time, object_lock_legal_hold, user_metadata, writetime(user_metadata), content_type, writetime(content_type), restore_start_time, restore_expiry_time, retier_time, object_partially_tiered FROM storagegrid.object_by_uuid WHERE uuid = <UUID> - Cassandra Driver Error(Read timeout):'Operation timed out - received only 2 responses.' Detailed Info:[consistency: ALL, responses_received: 2, responses_required: 3, data_present: 1])
OBDI ^RDY 2023-03-31T07:40:42.290959| WARNING 1271 7521f267ce24b9c OBDI: Failed to lookup object 'Failed to execute cql at consistency TWO: SELECT event_time, event, last_access_time, object_lock_mode, object_lock_retain_until_time, object_lock_legal_hold, user_metadata, writetime(user_metadata), content_type, writetime(content_type), restore_start_time, restore_expiry_time, retier_time, object_partially_tiered FROM storagegrid.object_by_uuid WHERE uuid = <UUID> - Cassandra Driver Error(Read timeout):'Operation timed out - received only 2 responses.' Detailed Info:[consistency: ALL, responses_received: 2, responses_required: 3, data_present: 1]'
S3RQ %DED 2023-03-31T07:40:44.087205| NOTICE 0089 7521f267ce24b9c S3RQ: S3 error response: RequestId=1680248429898354, TraceId=7521f267ce24b9c, Resource=<PATH>, HTTP Method=PUT, HTTP Status Code=503, X-Forwarded-For: '<IP>', ErrorMsg=ServiceUnavailable, ErrorType=Client, CustomErrorMessage={<none>}, Details={<none>}
- Cassandra
system.logindicatesReadTimeoutExceptionmany times
INFO [ReadStage-52] 2023-03-31 07:40:07,218 NoSpamLogger.java (line 94) Maximum memory usage reached (33554432), cannot allocate chunk of 1048576
ERROR [ReadRepairStage:3806] 2023-03-31 07:40:42,020 CassandraDaemon.java (line 231) Exception in thread Thread[ReadRepairStage:3806,5,main]
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.