StorageGRID 11.5 returns HTTP 500 against zero byte S3 ingest using AWS SDK for Java
Applies to
- NetAppStorageGRID 11.5
- AWS SDK for Java
- Client sends request by HTTP to StorageGRID port 18084
- Uploading object is 0 byte
- Hadoop version 2.6
- Java version 1.7
Issue
- Ingest failure with
HTTP 500
error after upgrading StorageGRID 11.5. - Run the command “
hadoop distcp -m 50 /source-path s3a://bucket-name/destination-path”
to ingest the objects with the below errors on the Hadoop application side.
Caused by: org.apache.hadoop.fs.s3a.AWSS3IOException: innerMkdirs on s3a://bucketname/datapath/views/checkNumber/2020-11/testpath
com.amazonaws.services.s3.model.AmazonS3Exception: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: 1624352367639862), Extended Request ID: null: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: 1624352367639862)
bycast-err.log
shows the below error on the StorageGRID side:
MMM YY hh:mm:ss <nodename> ADE: |12345678 0123456789 S3RQ POPD YYYY-MM-DDThh:mm:ss.uuuuuu| ERROR 1878 <traceid> S3RQ: S3 request error: connection=<connectionid>, Resource=/bucket1/prefix1/object1, HTTP Status Code=500, ErrorMsg=InternalError, ErrorType=Client CustomErrorMessage={<none>}, Details={asyncPop error: no producer}
Or:
MMM YY hh:mm:ss <nodename> ADE: |12345678 0123456789 S3RQ %DED YYYY-MM-DDThh:mm:ss.uuuuuu| NOTICE 0090 <traceid> S3RQ: S3 error response: RequestId=<requestid>, TraceId=<traceid>, Resource=/bucket1/prefix1/object1, HTTP Status Code=500, ErrorMsg=InternalError, ErrorType=Client, CustomErrorMessage={<none>}, Details={unexpected error (GERR). details: ,}