CVO upgrade paused due to plex resync
Applies to
- Cloud Volumes ONTAP (CVO)
- Amazon Web Services (AWS)
Issue
During a scheduled CVO upgrade, after node takeover (as part of the upgrade), aggregates on each node entered a degraded state and began plex resync operations:
Error:Failed to upgrade the ONTAP with status: paused_on_errorCause
- Backend storage latency on AWS EBS volumes during the upgrade process
- CVO detected device latencies exceeding the 25,000 microsecond threshold, leading to temporary marking of disks as failed or missing
- This triggered aggregate degraded states and plex resyncs, which in turn paused the upgrade until the resyncs finished
Sun Oct 12 09:51:22 +0000 [CVO_CVO_Cluster: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "isp25" and the associated target device "/dev/nvd9". The threshold is "25000", execution latency is "26743", total latency is "26766", and the number of operations for this period is "69".Sun Oct 12 09:51:22 +0000 [CVO_Cluster: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "isp28" and the associated target device "/dev/nvd8". The threshold is "25000", execution latency is "25692", total latency is "25711", and the number of operations for this period is "70".Solution
- The upgrade will proceed after the plex resyncs completed, and both nodes should be upgraded successfully.
- Use the plex show command to check on the status of the plex resync
Partner Notes
partnerNotes_text
Additional Information
For additional details about the disk latency, reach out to AWS
Internal Notes
internalNotes_text