High latency lasting for 20s after re-inserting the NVMe disks
Applies to
- ONTAP 9
- NS224NSM100 Shelf
- NSM100.0210.SFW or lower versions
Issue
- Pulling out a disk physically doesn't cause any high latency
- Re-inserting the disk back to where it was causes a significant latency(300ms+), which lasts for at least 20 seconds
Aug 08, 2023 interval i/o MB/sec bytes read resp read write read write resp queue cpu% cpu%
rate 1024**2 i/o pct time resp resp max max stddev depth sys+u sys
11:05:52.002 7330 162810.0 635.98 4096 50.04 0.784 0.794 0.774 11.56 7.74 0.619 127.6 4.4 3.7
11:05:53.002 7331 65962.0 257.66 4096 49.95 1.833 2.804 0.865 148.01 130.48 8.057 127.9 2.1 1.7
11:05:54.001 7332 6505.0 25.41 4096 50.10 13.984 27.135 0.779 564.42 237.50 39.614 127.9 0.2 0.2 <<<< latency starts to increase
11:05:55.002 7333 5183.0 20.25 4096 50.74 18.038 34.668 0.906 1669.83 134.46 77.484 128.0 0.2 0.1
11:05:56.001 7334 4021.0 15.71 4096 50.86 18.663 35.809 0.918 1925.62 323.81 85.525 127.9 0.1 0.1
11:05:57.001 7335 2029.0 7.93 4096 50.32 61.915 116.416 6.710 3475.53 2788.37 295.145 128.0 0.1 0.1
11:05:58.002 7336 1849.0 7.22 4096 51.65 70.809 129.283 8.346 4165.34 3871.72 351.452 128.0 0.1 0.0
11:05:59.001 7337 654.0 2.55 4096 52.75 87.737 164.500 2.031 4719.96 545.98 380.376 127.9 0.0 0.0
11:06:00.003 7338 248.0 0.97 4096 52.42 505.870 960.016 5.540 5977.56 624.00 1307.227 128.0 0.0 0.0
11:06:01.002 7339 82.0 0.32 4096 53.66 435.734 784.257 32.180 5011.87 1213.09 1222.068 128.0 0.0 0.0
11:06:02.001 7340 272.0 1.06 4096 49.63 534.932 1028.842 48.232 5878.59 4575.01 1333.964 127.9 0.0 0.0
11:06:03.001 7341 553.0 2.16 4096 50.09 291.319 555.145 26.538 4947.73 4331.76 930.171 128.0 0.0 0.0
11:06:04.001 7342 103.0 0.40 4096 43.69 356.931 637.607 139.164 5498.16 5793.68 1103.215 127.9 0.0 0.0
11:06:05.002 7343 124.0 0.48 4096 48.39 557.411 1151.677 0.286 5852.79 0.43 1310.418 128.0 0.0 0.0
11:06:06.002 7344 512.0 2.00 4096 45.90 365.526 732.140 54.500 7148.55 6949.17 1178.939 128.1 0.0 0.0
11:06:07.001 7345 104.0 0.41 4096 48.08 445.630 772.286 143.171 8443.86 7717.69 1519.741 127.9 0.0 0.0
11:06:08.001 7346 203.0 0.79 4096 56.16 636.207 1132.672 0.286 9444.99 0.42 1683.053 128.0 0.0 0.0
11:06:09.001 7347 237.0 0.93 4096 50.21 573.815 1058.516 85.007 9709.71 10000.52 1700.200 127.9 0.0 0.0
11:06:10.001 7348 342.0 1.34 4096 44.74 779.534 1711.994 24.686 10430.75 4597.19 2287.474 128.0 0.0 0.0
11:06:11.002 7349 265.0 1.04 4096 48.68 526.416 1081.143 0.242 8386.50 0.39 1540.784 128.0 0.0 0.0
11:06:12.001 7350 167.0 0.65 4096 53.29 441.639 828.474 0.251 6304.54 0.43 1329.067 127.9 0.0 0.0
11:06:13.002 7351 223.0 0.87 4096 51.12 648.932 1225.246 46.181 7417.14 5006.28 1612.059 128.0 0.0 0.0
11:06:14.002 7352 362.0 1.41 4096 46.69 385.074 824.525 0.269 6449.28 5.12 1240.424 128.0 0.0 0.0
11:06:15.001 7353 166.0 0.65 4096 45.78 594.156 1297.434 0.277 6448.72 0.43 1518.776 127.9 0.0 0.0
11:06:16.001 7354 901.0 3.52 4096 48.61 338.495 671.356 23.608 7463.71 7254.85 1163.933 128.0 0.0 0.0
11:06:17.001 7355 338.0 1.32 4096 52.37 256.145 482.927 6.825 5800.68 1033.72 801.015 127.9 0.0 0.0
11:06:18.001 7356 38103.0 148.84 4096 50.43 8.047 14.994 0.980 7092.49 1692.23 128.940 128.0 1.3 1.0 <<<< latency gets back to normal
- This happens to both the disks in an Aggregate and the spare disks