StorageGRID admin node fails to respond /grid/metric-query API during decommission

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 74

Visibility:: Public

Votes:: 0

Category:: storagegrid-webscale

Specialty:: sgrid

Last Updated:

Applies to

StorageGRID 11.6
Decommission task is processing

Issue

Both the Primary Admin node and Secondary Admin node fail to respond /grid/metric-query API at the same time.
500 error code (Internal Server error) can be observed by nginx-gw\nginx-gw\access.log.*
- Normal
  unix: - - [13/Feb/2023:04:30:03 +0000] "GET /api/v2/grid/metric-query?query=storagegrid_tenant_usage_data_bytes{instance=\x22admin-container1\x22} HTTP/1.1" 200 361743 "-" "curl/xx.29.0" 10.145.222.111- - [13/Feb/2023:04:30:03 +0000] "GET /api/v2/grid/metric-query?query=storagegrid_tenant_usage_data_bytes{instance=\x22admin-container1\x22} HTTP/1.1" 200 361743 "-" "curl/xx.29.0"
- Abnormal
  10.145.222.111 - - [13/Feb/2023:04:50:03 +0000] "GET /api/v2/grid/metric-query?query=storagegrid_tenant_usage_data_bytes{instance=\x22admin-container1\x22} HTTP/1.1" 500 1273 "-" "curl/xx.29.0" unix: - - [13/Feb/2023:04:50:04 +0000] "GET /api/v2/grid/metric-query?query=storagegrid_tenant_usage_data_bytes{instance=\x22admin-container1\x22} HTTP/1.1" 500 1273 "-" "curl/xx.29.0"

Unable to parse proxy results error message can be observed by nms.log.

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.592| WARN    Received call to deprecated v2 API at GET "/api/v2/grid/metric-query"

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.608| ERROR Unable to parse proxy results.

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   765: unexpected token at 'Service Unavailable' (JSON::ParserError)

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   /usr/lib/ruby/vendor_ruby/json/common.rb:156:in `parse'

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   /usr/lib/ruby/vendor_ruby/json/common.rb:156:in `parse'

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   /usr/local/mgmt-api/app/controllers/proxy.rb:418:in `respond_wrapped'

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   Invalid result: "Service Unavailable" (RuntimeError)

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   /usr/local/mgmt-api/app/controllers/proxy.rb:423:in `rescue in respond_wrapped'

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   /usr/local/mgmt-api/app/controllers/proxy.rb:417:in `respond_wrapped'

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   /usr/local/mgmt-api/app/controllers/proxy.rb:142:in `block (2 levels) in registered'

Feb 13 04:50:03 admin-container1 NMS: |2023-02-13T04:50:03.609| ERROR   {"message":{"text":"Invalid result: \"Service Unavailable\""},"code":500,"status":"error","responseTime":"2023-02-13T04:50:03.609Z","apiVersion":"2.7","deprecated":true,"data":{"type":"RuntimeError","trace":["/usr/local/mgmt-api/app/controllers/proxy.rb:423:in `rescue in respond_wrapped'","/usr/local/mgmt-api/app/controllers/proxy.rb:417:in `respond_wrapped'","/usr/local/mgmt-api/app/controllers/proxy.rb:142:in `block (2 levels) in registered'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1635:in `call'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1635:in `block in compile!'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:992:in `block (3 levels) in route!'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1011:in `route_eval'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:992:in `block (2 levels) in route!'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1040:in `block in process_route'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1038:in `catch'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1038:in `process_route'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:990:in `block in route!'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:989:in `each'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:989:in `route!'","/usr/lib/ruby/vendor_ruby/sinatra/base.rb:1097:in `block in dispatch!'"]}}