S3 GETs on LATM (last access time) enabled buckets may cause memory exhaustion and crash StorageGRID nodes
Applies to
- StorageGRID 11.2.0.3 or earlier
- StorageGRID 11.1.1.9 or earlier
Description
Only if Last Access Time (LATM) on a bucket is enabled may this article be helpful.
Symptoms
- S3 Client side slow performance
- Storage Node Memory depletion
- Possible Storage Nodes crashing (Blue state)
- Possible Storage Node LDR service crashing (Blue state)
- High CPU Usage on Storage Node
- "Worst Case Blocking (MSBX)" over 100s
- GMI -> Storage Node -> LDR -> Resources
Cause
When LATM is enabled for a bucket
- S3 Request will attempt to push all object segments to ILM when OBDR is destroyed
- The pushWithSegments looks up the segment UUID metadata to enqueue segments into ILM queue
- As ADE kernel considers the OBDR dead, the C* operation response is never delivered to it and OBDR never exits