Disk latency with extCache.io.writeError and insert_reject_throttle_io
Applies to
- ONTAP 9
- All FAS systems with NVMe based Flash Cache
Issue
- Increased latency on
Disk
Delay Center
cluster01::> qos statistics volume latency show -vserver vserver01 -volume volume01
Workload ID Latency Network Cluster Data Disk QoS Max QoS Min NVRAM Cloud FlexCache SM Sync VA
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total- - 24.06ms 104.00us 53.00us 556.00us 23.35ms 0ms 0ms 3.00us 0ms 0ms 0ms 0ms
volume01-wid51230 51230 22.49ms 104.00us 52.00us 526.00us 21.81ms 0ms 0ms 3.00us 0ms 0ms 0ms 0ms
-total- - 13.60ms 143.00us 60.00us 188.00us 13.21ms 0ms 0ms 2.00us 0ms 0ms 0ms 0ms
volume01-wid51230 51230 13.19ms 143.00us 59.00us 185.00us 12.80ms 0ms 0ms 2.00us 0ms 0ms 0ms 0ms
- Multiple events
wafl_exempt09: ems.engine.suppressed:debug]: Event 'extCache.io.writeError' suppressed 67626 times in last 601 seconds.
wafl_exempt09: extCache.io.writeError:notice]: WAFL external cache I/O write error: Insertq_save: unable to set up context chain, code 0
- FlashCache no longer inserts
cluster01::> node run -node node02 stats show -p flexscale-access
Cache Reads Writes Disk Reads
Usage Hit Meta Miss Hit Evict Inval Insert Chain Blocks Chain Blocks Replaced
% /s /s /s % /s /s /s /s /s /s /s /s
45 703 11 39171 1 0 0 0 0 0 0 0 109
45 320 26 37607 0 0 0 0 0 0 0 0 98
45 903 51 38018 2 0 0 0 0 0 0 0 114
- Reboot fixes it temporarily