High ingest latency due to prefix splitting failed
Applies to
- NetApp StorageGRID 11.5.0.2 or later
- NetApp StorageGRID 11.6
Issue
- Ingest latency increase sharply while S3 operations rate doesn't increase or even drops
- In below statistics sample, latency increase to 8 seconds at 18:08 (GMT):
% grep SPUT audit.log | audit-sum -gt 1M
message group count min(sec) max(sec) average(sec)
============= ===== ======== ======== ============
...
YYYY-MM-DDT18:03 8819 0.011 1.297 0.031
YYYY-MM-DDT18:04 8579 0.011 1.337 0.035
YYYY-MM-DDT18:05 7826 0.011 1.381 0.036
YYYY-MM-DDT18:06 8062 0.011 1.721 0.036
YYYY-MM-DDT18:07 7399 0.011 7.720 0.119
YYYY-MM-DDT18:08 1520 0.073 12.596 8.728
YYYY-MM-DDT18:09 1941 0.207 12.570 8.074
YYYY-MM-DDT18:10 2233 0.020 12.000 8.063
YYYY-MM-DDT18:11 2559 0.028 11.190 8.745
YYYY-MM-DDT18:12 2617 0.014 11.320 8.927
...
%- Grid network traffic increase significantly after 02:00 AM (local time, GMT+8):
