S3 Browser reports Server unavailable when uploading large file
Applies to
- NetApp StorageGRID
- Bare-metal based nodes
- S3 Browser (may affect other applications)
Issue
When trying to upload a large file on S3 browser, S3 browser reports "
Server unavailable
"Bycast.log
reports:Jul 1317:23:20 <nodename> ADE: |12563451 0477510131 S3RQ CSEN2023-07-13T17:23:20.726909| NOTICE 0089 e7693aca94f3d63 S3RQ: S3error response: RequestId=1689181912355014, TraceId=e7693aca94f3d63,Resource=/<filename>?partNumber=1&uploadId=0L2rrN9oJwNo9PRW58zM0vZAafdv5gcFtolZ0vL1u8v8m5UDqP0fQuU_jw,HTTP Method=PUT, HTTP Status Code=503, X-Forwarded-For: '<ip address>',ErrorMsg=ServiceUnavailable, ErrorType=Internal,CustomErrorMessage={<none>}, Details={Failed to execute cql at consistency TWO: SELECT * FROM storagegrid.s3_multipart_uploads_by_bucket WHEREbucket_id = '245C45D6-219D-11EE-88A1-D3E000B7D9E5' AND key ='<filename>' AND upload_id ='0L2rrN9oJwNo9PRW58zM0vZAafdv5gcFtolZ0vL1u8v8m5UDqP0fQuU_jw' - Cassandra DriverError(Read timeout):'Operation timed out - received only 5 responses.' DetailedInfo:[consistency: ALL, responses_received: 5, responses_required: 6,data_present: 1]}
Cassandra
system.log
reports nodes going up/downINFO [GossipTasks:1] <date> <time> Gossiper.java (line 1260) InetAddress /<ip address> is now DOWN
INFO [GossipTasks:1] <date> <time> Gossiper.java (line 1260) InetAddress /<ip address> is now UP
When running Support > Diagnostics, serveral nodes report high disk latency, up to seconds