Skip to main content
NetApp Knowledge Base

The CVO is in failed status and panic frequently after the upgrade

  1. Last updated
  2. Save as PDF
Views:
206
Visibility:
Public
Votes:
1
Category:
netapp_console
Specialty:
ds_cvo
Last Updated:

Applies to

  • NetApp Cloud Volume ONTAP(CVO)  9.7
  • Microsoft Azure

Issue

The CVO is in failed status and panic frequently happens after upgrade to ONTAP9.7P22

  • CVO working environment error:
Failed - Show Failure Message
Cloud Manager cannot communicate with Cloud Volumes
ONTAP because there is no connectivity or because the Cloud Volumes ONTAP system is not available.
 
  • EMS log of panic :
Wed Jul 12 07:39:02 +0000 [nodename-01: nodewatchdog: nodewatchdog.node.panic:alert]: Data ONTAP has experienced a serious internal error: Process vldb unresponsive for 210 seconds. This might cause the node experiencing the problem to become unresponsive to data access. The node has been panicked to prevent this condition from continuing.
Wed Jul 12 07:39:02 +0000 [nodename-01: nodewatchdog: sk.panic:alert]: Panic String: Process vldb unresponsive for 210 seconds in process nodewatchdog on release 9.7P22 (C)
Wed Jul 12 07:39:02 +0000 [nodename-01: nodewatchdog: coredump.shutdown.trace:notice]: coredump shutdown times: kmod dumper start 7, panic_info set 14, sf_dumpcore sent 15, sparecore id sent 59, fmot_outage info sent 0, disk_dump_start done 0, coredump begin 24 (all in msec).

Cause

Solution

Upgrade to CVO 9.8P19 or later

Partner Notes

partnerNotes_text
 

Additional Information

additionalInformation_text

Internal Notes

Run the "pciconf -lbv" command from the systemshell, and search for ConnectX-5 (or 6).
In case the upgrade is not possible due to nonstop panics, the infinite loop is probably because of the mlx5 issuing FW dump here.
You can kill this process and upgrade to the system to latest version.
 
NOTE:  The monitoring daemon needs to be disabled every time the node reboots.
 
To verify if the monitoring daemon is running :
::*> systemshell -node * ps -auxw | grep mlx
 
To disable the monitoring daemon, if running:
::*> system shell -node <node_name> sudo /bin/pkill -f netapp_mlx5dump.sh

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.