Higher than expected SSD latency due to 6Gb SAS shelves and cables

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 283

Visibility:: Public

Votes:: 0

Category:: ontap-9

Specialty:: perf

Last Updated:

Applies to

ONTAP 9
AFF
SAS 6Gb HBA

Issue

qos statistics command shows disk latency:

cluster1::> qos statistics volume latency show -vserver svm1 -volume vol1
Workload            ID    Latency    Network    Cluster       Data       Disk    QoS Max    QoS Min      NVRAM  
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total-              -     4.71ms   185.00us     7.00us   468.00us     4.04ms        0ms        0ms    11.00us
vol1-wid32140    32140    22.22ms    86.00us   125.00us   904.00us    21.11ms        0ms        0ms     1.00us
-total-              -     4.58ms   253.00us     7.00us   991.00us     3.32ms        0ms        0ms    11.00us
vol1-wid32140    32140    29.69ms    90.00us   162.00us     2.61ms    26.82ms        0ms        0ms     1.00us
-total-              -     2.55ms   229.00us     8.00us   589.00us     1.71ms        0ms        0ms    13.00us
vol1-wid32140    32140    18.84ms    61.00us   314.00us   136.00us    18.32ms        0ms        0ms     1.00us
-total-              -     4.33ms   198.00us     9.00us  1221.00us     2.88ms        0ms        0ms    13.00us
vol1-wid32140    32140   177.16ms   113.00us   226.00us   108.03ms    68.79ms        0ms        0ms     2.00us

The io_pending and the io_queued values are high
The SSD disks show high latency and utilization in statit

Cluster::> set -privilege diag

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}: y

Cluster::*> statistics start -object disk -counter "io_queued|io_pending"
Statistics collection is being started for sample-id: sample_148


Cluster::*> statistics show -filter "io_queued>10|io_pending>10"


Object: disk
Instance: 0d.23.13
Start-time: 12/5/2022 16:48:26
End-time: 12/5/2022 16:51:58
Elapsed-time: 212s
Scope: node1
Number of Constituents: 1 (complete_aggregation)
    Counter                                                     Value
    -------------------------------- --------------------------------
    io_queued                                                      19
    io_pending                                                      8

...

Cluster::*> node run -node node1 -command statit -b
Cluster::*> node run -node node1 -command statit -e
                       Disk Statistics (per second)  
        ut% is the percent of time the disk was busy.  
        xfers is the number of data-transfer commands issued per second.  
        xfers = ureads + writes + cpreads + greads + gwrites  
        chain is the average number of 4K blocks per command.  
        usecs is the average disk round-trip time per 4K block.  
  disk            ut%  xfers   ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites--chain-usecs
/data_aggr1/plex0/rg0:                                                                  
0c.01.2           80 2510.99  1525.62   5.95   763 389.58  11.11  2682 595.79   5.22  1318   0.00   ....     .   0.00   ....     .
0c.01.3           81 2502.32  1510.75   5.96   758 393.43  11.27  2706 598.14   5.15  1348   0.00   ....     .   0.00   ....     .
0c.01.4           81 2518.91  1528.01   5.93   762 392.18  11.45  2661 598.72   5.12  1364   0.00   ....     .   0.00   ....     .
...