Latency increases during volume move cutover phase
Applies to
ONTAP 9
Issue
All the volumes on the filer witness a short slowness during the volume move cutover phase of a volume on this node.
Example: Below is the performance statisitcs observed on one impacted volume from both the client and the filer during the cutover phase.
Note: This is not the volume going through the volume move cutover phase
VDbentch log:11 18, 2020 interval i/o MB/sec bytes read resp read write resp resp queue cpu% cpu%
rate 1024**2 i/o pct time resp resp max stddev depth sys+u sys
13:32:50.043 501 17260.00 269.69 16384 79.79 0.921 0.910 0.965 7.402 0.311 15.9 44.2 35.6
13:32:51.043 502 15972.00 249.56 16384 79.66 0.996 1.033 0.850 25.002 0.889 15.9 44.8 35.6
13:32:52.042 503 8553.00 133.64 16384 80.09 1.860 1.868 1.829 8.574 1.722 15.9 22.0 16.5
13:32:53.042 504 3407.00 53.23 16384 80.83 4.693 4.690 4.704 9.185 1.325 16.0 8.9 7.3
13:32:54.044 505 3351.00 52.36 16384 79.14 4.768 4.769 4.765 9.339 1.131 16.0 11.4 9.0
13:32:55.042 506 3406.00 53.22 16384 80.59 4.679 4.694 4.616 10.017 1.302 15.9 10.8 8.2
13:32:56.042 507 3339.00 52.17 16384 81.28 4.796 4.786 4.836 9.095 1.194 16.0 10.6 8.1
13:32:57.042 508 3399.00 53.11 16384 80.94 4.700 4.697 4.714 9.183 1.204 16.0 9.7 8.2
13:32:58.043 509 3570.00 55.78 16384 81.32 4.462 4.473 4.417 9.443 1.454 16.0 10.3 8.7
13:32:59.042 510 6971.00 108.92 16384 80.10 2.297 2.315 2.225 13.173 1.853 15.9 21.1 16.6
13:33:00.044 511 18303.00 285.98 16384 79.92 0.868 0.870 0.858 4.062 0.234 15.9 46.5 39.6
13:33:01.043 512 18439.00 288.11 16384 80.43 0.863 0.874 0.814 6.378 0.353 15.9 48.5 39.3
13:33:02.042 513 17853.00 278.95 16384 79.77 0.891 0.901 0.851 9.553 0.419 15.9 53.1 36.9
qos statistics volume latency show:
2020/11/18 13:32:48 Workload ID Latency Network Cluster Data Disk QoS Max QoS Min NVRAM Cloud FlexCache SM Sync VA
2020/11/18 13:32:48 --------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
2020/11/18 13:32:48 volume2 51455 360.00us 261.00us 0ms 26.00us 69.00us 0ms 0ms 4.00us 0ms 0ms 0ms 0ms
2020/11/18 13:32:51 volume2 51455 372.00us 264.00us 0ms 28.00us 76.00us 0ms 0ms 4.00us 0ms 0ms 0ms 0ms
2020/11/18 13:32:54 volume2 51455 1004.00us 160.00us 0ms 682.00us 159.00us 0ms 0ms 3.00us 0ms 0ms 0ms 0ms
2020/11/18 13:32:57 volume2 51455 4.21ms 138.00us 0ms 3.95ms 122.00us 0ms 0ms 4.00us 0ms 0ms 0ms 0m
2020/11/18 13:33:00 volume2 51455 3.81ms 137.00us 0ms 3.54ms 127.00us 0ms 0ms 4.00us 0ms 0ms 0ms 0ms
2020/11/18 13:33:03 volume2 51455 366.00us 143.00us 0ms 128.00us 92.00us 0ms 0ms 3.00us 0ms 0ms 0ms 0ms
2020/11/18 13:33:06 volume2 51455 322.00us 132.00us 0ms 105.00us 82.00us 0ms 0ms 3.00us 0ms 0ms 0ms 0ms