Intermittent Error 504 Gateway Time-out during ingest/ retrieve on storageGRID bucket

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 85

Visibility:: Public

Votes:: 0

Category:: storagegrid

Specialty:: sgrid

Last Updated:

Applies to

StorageGRID OS 11.7.0

Issue

Intermittent error observed with ingest/ retrieve of objects from storageGRID bucket:

Error: reading blob <>: fetching blob: received unexpected HTTP status: 504 Gateway Time-out

From storageGRID Graphana, we observed 504 errors coinciding with 499 errors on the gateway node which matches with the issue observed from application side:

Failed to pull docker image from bucket: Error timed out

Enabled debug level logging
From storageGRID logs, we identified the issue with SSL handshake timeout during upstream of request from the Client to Storage Node via Gateway node:

Gateway node - /var/local/log/nginx-gw/access.log

2024/04/15 06:40:08 [warn] 1327055#1327055: *580890363 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 23 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-01-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 06:40:13 [warn] 1327099#1327099: *580934305 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 31 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:19:56 [warn] 1327060#1327060: *581032520 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 24 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata, host: "<Gateway node>:10443"

2024/04/15 06:40:08 [error] 1327055#1327055: *580890363 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:34:50 [error] 1327060#1327060: *581032520 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"