Skip to main content
NetApp Knowledge Base

CVO upgrade paused with error "paused_on_error" due to plex resync

  1. Last updated
  2. Save as PDF
Views:
101
Visibility:
Public
Votes:
0
Category:
cloud-volumes-ontap-cvo
Specialty:
amazon_web_service
Last Updated:

Applies to

  • Cloud Volumes ONTAP  (CVO)
  • Amazon Web Services (AWS)
  • Google Cloud Platform (GCP0

Issue

During a scheduled CVO upgrade, after node takeover (as part of the upgrade), aggregates on each node entered a degraded state and began plex resync operations:
Error:Failed to upgrade the ONTAP with status: paused_on_error

Cause

  • Backend storage latency on AWS EBS/GCP volumes during the upgrade process
  • CVO detected device latencies exceeding the 25,000 microsecond threshold, leading to temporary marking of disks as failed or missing
  • This triggered aggregate degraded states and plex resyncs, which in turn paused the upgrade until the resyncs finished
Sun Oct 12 09:51:22 +0000 [CVO_CVO_Cluster: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "isp25" and the associated target device "/dev/nvd9". The threshold is "25000", execution latency is "26743", total latency is "26766", and the number of operations for this period is "69".
 
Sun Oct 12 09:51:22 +0000 [CVO_Cluster: kernel: device.latency.threshold:notice]: The device latency in microseconds exceeded the threshold value for device "isp28" and the associated target device "/dev/nvd8". The threshold is "25000", execution latency is "25692", total latency is "25711", and the number of operations for this period is "70".

Solution

  • The upgrade will proceed after the plex resyncs completed, and both nodes should be upgraded successfully.
  • Use the plex show command to check on the status of the plex resync

Partner Notes

partnerNotes_text
 

Additional Information

For additional details about the disk latency, reach out to AWS/GCP

Internal Notes

internalNotes_text

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.