High Volume of Active Proxy Delete Requests/High HTTP Session Count
Applies to
StorageGRID 11.X and 12.X
Description
StorageGRID is experiencing an unusually high number of active proxy delete requests and HTTP sessions.
This can be confirmed via the following methods:
1. From GMI --> Support --> Metrics --> Prometheus Interface (link ending with /metrics/graph)
2. Can also be configured as custom alerts so you may see an alert on the GridManager UI
Look for the following:
storagegrid_private_serialization_active_proxy_delete_object_requests has exceeded 1500
storagegrid_http_sessions_incoming_currently_established{service="ldr"} has surpassed 15,000
Please note, these conditions may also trigger alerts related to open file descriptors (FOPN), but the primary indicators of the issue referenced in this article are the two metrics above. If you are not seeing these symptoms use the FOPN KB for further troubleshooting.
Impact: This situation can halt delete operations, potentially causing application outages and resulting in 504 errors. Applications dependent on delete operations may experience significant disruptions and case priority should be defined based on customer impact.
Cause: Typically triggered by problems on network, such as site isolation or switch/router/firewall changes.
