Write latency due to imbalanced Raid group
Applies to
- ONTAP
Issue
- The application or VM reported slowness.
- Latency is reported from the data layer in the QOS statistics command or from WAFL Susp CP in AIQ volume latency (component) view
cluster1::> qos statistics volume latency show -vserver vs0 -volume vs0_vol0
Workload ID Latency Network Cluster Data Disk QoS NVRAM Cloud
--------------- ------ ---------- ---------- ---------- ---------- ---------- ------- ---------- ----------
vs0_vol0-wid1.. 15658 42.8ms 155.00us 0ms 28.7ms 14.1ms 0ms 0ms 0ms
vs0_vol0-wid1.. 15658 31.6ms 128.00us 0ms 23.5ms 8.1.ms 0ms 0ms 0ms
vs0_vol0-wid1.. 15658 47.1ms 130.00us 0ms 32.8ms 14.3ms 0ms 0ms 0ms
vs0_vol0-wid1.. 15658 30.2ms 137.00us 0ms 19.2ms 11.0ms 0ms 0ms 0ms
vs0_vol0-wid1.. 15658 42.4ms 143.00us 0ms 32.8ms 9.60ms 0ms 0ms 0ms
- systat -x output shows evidence of CP type CP_Ty B (Back-to-Back) CPs.
- There is no direct indication of a disk bottleneck.
- Statit shows that one of the raid groups contains a single/very few data disks and the disks utilization is not evenly distributed across raid group.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0: 0a.03.0 1 5.76 0.00 .... . 3.79 47.02 21 1.97 18.21 31 0.00 .... . 0.00 .... . 0b.03.1 1 5.76 0.00 .... . 3.80 46.91 21 1.96 18.09 31 0.00 .... . 0.00 .... . 0b.00.1 83 228 0.07 10.00 44 6.16 39.95 19 0.06 38.16 38 0.00 .... . 0.00 .... . 0b.00.2 80 215 58.04 2.85 243 6.14 40.02 19 0.28 5.26 85 0.00 .... . 0.00 .... . /aggr0/plex0/rg1: 0a.01.0 1 120.54 0.00 .... . 54.37 57.82 14 66.17 44.91 20 0.00 .... . 0.00 .... . 0b.00.8 1 120.54 0.00 .... . 54.38 57.81 19 66.16 44.91 23 0.00 .... . 0.00 .... . 0a.01.1 43 185.28 2061.95 4.04 135 57.25 32.25 20 66.09 12.07 60 0.00 .... . 0.00 .... . 0b.00.9 55 180.00 2057.09 4.04 150 57.25 32.29 23 65.66 12.11 74 0.00 .... . 0.00 .... . 0a.01.2 44 176.64 2054.20 4.02 141 56.59 32.40 20 65.85 12.20 62 0.00 .... . 0.00 .... . 0b.00.10 56 166.30 2043.93 4.03 163 56.86 32.45 24 65.50 12.31 73 0.00 .... . 0.00 .... . 0a.01.3 73 179.01 2056.35 4.02 136 57.05 32.43 20 65.62 12.19 60 0.00 .... . 0.00 .... . 0b.00.11 75 162.22 2039.31 4.03 149 57.16 32.20 24 65.74 12.25 71 0.00 .... . 0.00 .... . 0a.01.4 63 177.86 2055.72 4.00 138 56.69 32.51 21 65.45 12.40 61 0.00 .... . 0.00 .... . 0b.00.22 55 169.34 2046.18 4.02 156 56.96 32.21 25 66.20 12.36 73 0.00 .... . 0.00 .... . 0a.01.6 74 174.07 2051.05 4.03 139 57.32 32.47 21 65.70 12.02 62 0.00 .... . 0.00 .... . 0b.00.13 25 163.94 2041.44 4.02 150 56.64 32.27 26 65.86 12.38 75 0.00 .... . 0.00 .... . <<<<<< 0b.00.14 21 165.87 2044.10 4.01 149 56.39 32.49 27 65.38 12.20 72 0.00 .... . 0.00 .... . 0b.00.23 15 170.28 2047.49 4.00 150 56.80 32.12 28 65.99 12.13 73 0.00 .... . 0.00 .... .