Persistent latency on a single‑node Cloud Volumes ONTAP system in AWS
Applies to
- ONTAP 9
- Cloud Volumes ONTAP (single node)
- AWS
Issue
- Severe and persistent latency (200-400ms) despite very low I/O load (~250–287 operations) and moderate CPU utilization (~53–57%)
- device.latency.threshold exceeded the threshold seen on all disks
[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd2" and the associated target device "n/a". The threshold is "25000", execution latency is "28991", total latency is "29941", and the number of operations for this period is "2316".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd7" and the associated target device "n/a". The threshold is "25000", execution latency is "60773", total latency is "62108", and the number of operations for this period is "8329".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd8" and the associated target device "n/a". The threshold is "25000", execution latency is "52614", total latency is "53674", and the number of operations for this period is "8552".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd10" and the associated target device "n/a". The threshold is "25000", execution latency is "60209", total latency is "61568", and the number of operations for this period is "8349".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd9" and the associated target device "n/a". The threshold is "25000", execution latency is "40269", total latency is "41193", and the number of operations for this period is "9944".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd4" and the associated target device "n/a". The threshold is "25000", execution latency is "28189", total latency is "28996", and the number of operations for this period is "12541".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd5" and the associated target device "n/a". The threshold is "25000", execution latency is "28932", total latency is "29158", and the number of operations for this period is "12247".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd6" and the associated target device "n/a". The threshold is "25000", execution latency is "25476", total latency is "26996", and the number of operations for this period is "12836".[node-01: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "nvd1" and the associated target device "n/a". The threshold is "25000", execution latency is "24324", total latency is "26472", and the number of operations for this period is "10101".- NVRAM Transfer latency up to 200ms
