Hadoop application query fails to execute and write to storageGRID bucket and timeout
Applies to
- StorageGRID 11.6.0.13
- Hadoop
Issue
- Hadoop application query is failing to run and getting timed out as their queries are failing to write to bucket.
- From Application error:
botocore.exceptions.Client
2024-01-24 17:11:54.335 DEBUG com.amazonaws.thirdparty.apache. http.wire: 'main': http-outgoing-1 << "<Error><Code>ServiceUnavailable</Code><Message>Please reduce your request rate.</Message><Resource>/<BUCKET>/?uploads&prefix=daXXXrmediate%2Fquery_test2%2F</Resource><RequestId></RequestId></Error>"
2024-01-24 17:11:54.335 DEBUG com.amazonaws.thirdparty.apache. http.headers: 'main': http-outgoing-1 << HTTP/1.1 503 Service Unavailable- From StorageGRID logs:
Jan 25 08:46:07 XXXNODE04 ADE: |12330832 2577485089 S3RQ %CEA 2024-01-25T08:46:07.470023| NOTICE 1448 3018ddf97d08def3 S3RQ: EVENT_PROCESS_CREATE - connection=17061XXX127 method=GET name=</<BUCKET>/?uploads&prefix=datalake_intermediate%2Fquery_test%2F> auth=<V4> clientIP=<>
Jan 25 08:47:01 XXXNODE04 ADE: |12330832 2577485089 S3RQ CSEN 2024-01-25T08:47:01.869914| WARNING 0236 3018ddf97d08def3 S3RQ: S3Request_Module::processCreate Cassandra TimeoutException (posSuc: Failed to execute statement while paging. - Cassandra Driver Error(Request timed out):'Request timed out' Detailed Info:[N/A])
Jan 25 08:47:01 XXXNODE04 ADE: |12330832 2577485089 S3RQ CSEN 2024-01-25T08:47:01.869996| NOTICE 0089 3018ddf97d08def3 S3RQ: S3 error response: RequestId=1706XXX65127, TraceId=3018ddf97d08def3, Resource=/<BUCKET>/?uploads&prefix=daXXXmediate%2Fquery_test%2F, HTTP Method=GET, HTTP Status Code=503, X-Forwarded-For: 'XX.XX.XXX.91', ErrorMsg=ServiceUnavailable, ErrorType=Internal, CustomErrorMessage={<none>}, Details={posSuc: Failed to execute statement while paging. - Cassandra Driver Error(Request timed out):'Request timed out' Detailed Info:[N/A]}