StorageGRID - Client-side timeout during S3 Select queries

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:

Visibility:: Public

Votes:: 0

Category:: storagegrid

Specialty:: sgrid

Last Updated:

Applies to

StorageGRID 11.8 and 11.9
Amazon Web Services (AWS) SDK for Java v1
S3 Select API

Issue

S3 Select queries against large csv objects time out on the client side, and result in error 500, especially for objects with millions of rows or large file sizes (e.g., 5GB, 14 million rows).
The issue is observed when using AWS SDK for Java v1.
The client application reaches a timeout and disconnects before the query completes and response from GRID is returned.

Examining bycast-err.log on storage nodes reveals the following error:

Oct 28 15:51:45 nodename ADE: |12470336 1213087561 S3RQ PSHD 2025-10-28T15:51:45.519425| ERROR    1828 73b605873a4948b1 S3RQ: S3 request error (after response): connection=connection_id, Resource=/bucket/object_name.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<somecredential>%2F20251028%2Fus-east-1%2Fs3%2Faws4_requestX-Amz-Expires=10800x-ntap-bypass-host-check=1&X-Amz-Signature=<somesignature>, HTTP Method=GET, HTTP Status Code (not sent)=500, ErrorMsg=InternalError, ErrorType=Internal, CustomErrorMessage={<none>}, Details={asyncPush error: no consumer}

Example error message usig aws cli v1:

aws s3api select-object-content --bucket demo --key object_name.csv --endpoint http://enpoint.url:port --expression "SELECT * FROM S3Object WHERE product = 'someproduct'" --expression-type SQL --input-serialization '{"CSV": {"FileHeaderInfo": "USE"}}' --output-serialization '{"CSV": {}}'  output.csv  --no-verify-ssl

Read timeout on endpoint URL: "http://endpoint.url:port/bucket/object_name.csv?select&select-type=2"

In podman logs notice that the gap between 'NOTICE - Parsing query' and 'INFO - Starting HTTP response with code 200' is > 60 seconds (default aws cli client side timeout).

podman -r logs <container_id>

[2025-10-28T15:49:51] [/build/src/libs/sql_processing/Parser.cc:141] NOTICE - Parsing query 'SELECT * FROM S3Object WHERE product = 'someproduct''

[...]

[2025-10-28T15:51:45] [/build/src/libs/output_stream/OutputStream.cc:222] INFO - Starting HTTP response with code 200