Lower than expected performance while utilizing a single volume
Applies to
ONTAP 9
Issue
- Lower than expected performance while utilizing a single volume
- You may notice high latency on a volume, and CPU utilization is low.
Example:
- The output of commands
sysstat, qos statistics volume performance show, andqos statistics volume latency showhas high latency on vol1 only and CPU is low for the node. - Other volumes are not having latency
Cluster::> qos statistics volume latency show
Workload ID Latency Network Cluster Data Disk QoS NVRAM Cloud
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total- - 1057.00us 118.00us 86.00us 210.00us 609.00us 34.00us 0ms 0ms
vol1-wid25471 25471 20.76ms 598.00us 2.44ms 7.37ms 0ms 10.35ms 0ms 0ms
vol2-wid15658 15658 428.00us 155.00us 0ms 273.00us 0ms 0ms 0ms 0ms
-total- - 1.53ms 521.00us 153.00us 111.00us 158.00us 591.00us 0ms 0ms
vol1-wid25471 25471 48.54ms 932.00us 2.37ms 780.00us 0ms 44.46ms 0ms 0ms
vol2-wid15658 15658 316.00us 128.00us 0ms 188.00us 0ms 0ms 0ms 0ms
-total- - 830.00us 134.00us 75.00us 125.00us 444.00us 52.00us 0ms 0ms
vol1-wid25471 25471 37.61ms 698.00us 1.92ms 0ms 0ms 34.99ms 0ms 0ms
vol2-wid15658 15658 471.00us 130.00us 0ms 341.00us 0ms 0ms 0ms 0ms
cluster1::> qos statistics volume performance show
Workload ID IOPS Throughput Latency
--------------- ------ -------- ---------------- ----------
-total- - 4993 859.84MB/s 1044.00us
vol1-wid25471 25471 81 569.00MB/s 522.28ms
vol2-wid47655 47655 80 68.33MB/s 2.30ms
vol3-wid43753 43753 72 61.00MB/s 2.12ms
vol4-wid27831 27831 70 60.67MB/s 2.12ms
vol5-wid17972 17972 62 8.42MB/s 829.00us
vol6-wid14310 14310 62 1.80MB/s 623.00us
vol7-wid51942 51942 62 3.14MB/s 689.00us
vol8-wid631 631 61 3.46MB/s 648.00us
vol9-wid43632 43632 59 3.09MB/s 633.00us
vol10-wid52828 52828 58 7.09MB/s 1065.00us
-total- - 4604 872.46MB/s 1106.00us
vol1-wid25471 25471 70 560.67MB/s 152.15ms
vol2-wid47655 47655 70 59.33MB/s 2.15ms
vol3-wid43753 43753 70 60.33MB/s 2.33ms
vol4-wid27831 27831 70 60.67MB/s 2.14ms
vol5-wid17972 17972 65 15.11MB/s 1485.00us
vol6-wid14310 14310 62 8.45MB/s 977.00us
vol7-wid51942 51942 60 5.75MB/s 826.00us
vol8-wid631 631 58 4.42MB/s 613.00us
vol9-wid43632 43632 57 8.38MB/s 982.00us
vol10-wid52828 52828 56 4.41MB/s 574.00us
Cluster::> node run -node node1 -command sysstat -x 1
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP_Ty CP_Ph Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time [T--H--F--N--B--O--#--:] [n--v--p--f] util in out in out
13% 261 3 0 2927 222768 258592 104 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 1% 2663 0 0 0 0 0 0
15% 403 6 0 3040 255451 255197 116 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 2% 2631 0 0 0 0 0 0
5% 348 10 0 811 6094 9894 232 24 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 2% 453 0 0 0 0 0 0
2% 172 14 0 222 2795 535 840 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 2% 36 0 0 0 0 0 0
2% 172 10 0 196 2358 1024 296 32 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 2% 14 0 0 0 0 0 0
4% 179 4 0 185 1252 1725 30312 7788 0 0 12 96% 17% 1--0--0--0--0--0--0--0 0--0--0--1 7% 2 0 0 0 0 0 0
6% 191 7 0 1243 7036 3093 892 35628 0 0 12 100% 9% 0--0--0--0--0--0--0--0 0--0--0--0 6% 1045 0 0 0 0 0 0
11% 331 16 0 1884 211723 259196 508 3100 0 0 12 99% 5% 0--0--0--0--0--0--0--0 0--0--0--0 5% 1537 0 0 0 0 0 0
11% 270 15 0 2163 233854 259417 560 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 2% 1878 0 0 0 0 0 0
12% 380 5 0 2168 219803 258504 736 32 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 3% 1783 0 0 0 0 0 0
12% 146 1 0 1791 211641 257702 84 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 1% 1644 0 0 0 0 0 0
12% 614 3 0 2339 212750 259381 104 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 2% 1722 0 0 0 0 0 0
13% 419 2 0 2508 213848 251174 232 24 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 3% 2087 0 0 0 0 0 0
13% 262 3 0 2130 213556 259212 144 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 2% 1865 0 0 0 0 0 0
14% 778 5 0 2433 226228 284261 2024 32 0 0 12 99% 0% 0--0--0--0--0--0--0--0 0--0--0--0 8% 1650 0 0 0 0 0 0
13% 631 4 0 2454 224345 221650 104 0 0 0 12 100% 0% 0--0--0--0--0--0--0--0 0--0--0--0 1% 1819 0 0 0 0 0 0