High disk latency with low IOPS in Azure CVO
- Views:
- 240
- Visibility:
- Public
- Votes:
- 0
- Category:
- cloud-volumes-ontap-cvo
- Specialty:
- perf
- Last Updated:
- 8/19/2024, 3:28:58 PM
Applies to
- ONTAP 9
- CVO
- Azure
Issue
- The latency is seen from disk latency from QOS stats.
Cluster::> qos statistics volume latency show -vserver vs0 -volume vol0
Workload ID Latency Network Cluster Data Disk QoS NVRAM Cloud
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
vol0 6762 137.73ms 1.91ms 1.00us 22.93ms 112.89ms 0ms 0ms 0ms
vol0 6762 133.29ms 1.82ms 0ms 2.84ms 128.62ms 0ms 0ms 0ms
vol0 6762 78.48ms 1.96ms 0ms 7.69ms 68.83ms 0ms 0ms 0ms
vol0 6762 50.51ms 239.00us 0ms 995.00us 49.28ms 0ms 0ms 0ms
- The statit command indicates the IOPS are dropped while the disk utilization is increased.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/data_aggr1/plex0/rg0:
0a.00.18 100 280.18 280.18 64.00 67 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.00.10 100 280.18 280.18 64.00 64 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.00.19 100 280.18 280.18 64.00 71 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.00.12 100 280.18 280.18 64.00 69 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.00.17 100 280.18 280.18 64.00 75 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
0a.00.16 100 280.18 280.18 64.00 77 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
- The Disk IOPS decreased from >3K to ~200, and disk utilization increased to 100% with 200_ pending, and queued IOPS.
- EMS logs show connectivity errors as below
Tue Jul 16 14:47:05 +1000 [azpcvoause101c-01: pha_main000: scsi.cmd.floatTimeout:notice]: "Disk" device "0d.3L0": The page blob I/O operation timeout increased to "20" seconds: HA status "0x4": OSC Error "2": retry count "1": elapsed time "9712" milliseconds: cdb 0x88:00000000000272728820:0009. Disk 0d.3 S/N [sa82p4kn1zbcontainerotqmyqedc7iw] UID [sa82p4kn1zbcontainerotqmyqedc7iw] Target Address