Skip to main content
NetApp Knowledge Base

High disk latency with low IOPS in Azure CVO

  1. Last updated
    Aug 19, 2024
  2. Save as PDF
Views:
250
Visibility:
Public
Votes:
0
Category:
cloud-volumes-ontap-cvo
Specialty:
perf
Last Updated:
8/19/2024, 3:28:58 PM

Applies to

  • ONTAP 9
  • CVO
  • Azure

Issue

  • The latency is seen from disk latency from QOS stats.
Cluster::> qos statistics volume latency show -vserver vs0 -volume vol0
Workload    ID    Latency    Network    Cluster       Data       Disk        QoS      NVRAM      Cloud
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
vol0      6762   137.73ms     1.91ms     1.00us    22.93ms   112.89ms        0ms        0ms        0ms
vol0      6762   133.29ms     1.82ms        0ms     2.84ms   128.62ms        0ms        0ms        0ms
vol0      6762    78.48ms     1.96ms        0ms     7.69ms    68.83ms        0ms        0ms        0ms
vol0      6762    50.51ms   239.00us        0ms   995.00us    49.28ms        0ms        0ms        0ms
  • The statit command indicates the IOPS are dropped while the disk utilization is increased.
disk             ut%  xfers  ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/data_aggr1/plex0/rg0:    
0a.00.18          100 280.18  280.18  64.00  67   0.00   ....     .  0.00   ....     .   0.00   ....     . 0.00   ....     .
0a.00.10          100 280.18  280.18  64.00  64   0.00   ....     .  0.00   ....     .   0.00   ....     . 0.00   ....     .
0a.00.19          100 280.18  280.18  64.00  71   0.00   ....     .  0.00   ....     .   0.00   ....     . 0.00   ....     .
0a.00.12          100 280.18  280.18  64.00  69   0.00   ....     .  0.00   ....     .   0.00   ....     . 0.00   ....     .
0a.00.17          100 280.18  280.18  64.00  75   0.00   ....     .  0.00   ....     .   0.00   ....     . 0.00   ....     .
0a.00.16          100 280.18  280.18  64.00  77   0.00   ....     .  0.00   ....     .   0.00   ....     . 0.00   ....     .
 
  • The Disk IOPS decreased from >3K to ~200, and disk utilization increased to 100% with 200_ pending, and queued IOPS.
  • EMS logs show connectivity errors as below 
Tue Jul 16 14:47:05 +1000 [azpcvoause101c-01: pha_main000: scsi.cmd.floatTimeout:notice]: "Disk" device "0d.3L0": The page blob I/O operation timeout increased to "20" seconds: HA status "0x4": OSC Error "2": retry count "1": elapsed time "9712" milliseconds: cdb 0x88:00000000000272728820:0009. Disk 0d.3 S/N [sa82p4kn1zbcontainerotqmyqedc7iw] UID [sa82p4kn1zbcontainerotqmyqedc7iw] Target Address

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.