Skip to main content
NetApp Knowledge Base

CVO upgrade results in root volume recovery

  1. Last updated
  2. Save as PDF
Views:
401
Visibility:
Public
Votes:
0
Category:
netapp_console
Specialty:
ds_cvo
Last Updated:

Applies to

Cloud Volume ONTAP (CVO)

Issue

  • EMS reports error:
[xxxxxx:mgmtgwd.rootvolrec.low.space:EMERGENCY]: The root volume on node xxxxxx is dangerously low on space. Less than 10 MB of free space remaining.
 
  • Upgrade causes MGMT cluster app to go down on one node, and all cluster apps down on another.
  • One node of a two node cluster cannot run any clustershell commands:
cluster::> cluster show

Error: "show" is not a recognized command

 

Cause

  • The root volume "vol0" was almost full due to vol0 snapshots and old core files.
  • When vol0 is full, using Cloud Manager to "getAPI" the image from a web server causes the image to be written to vol0. vol0 cannot accept the whole image and takes MGMT app to go down and sets rdb flag on the node that isn't Cluster Master.

Solution

Please contact NetApp Technical Support or log into the NetApp Support Site to create a case. Reference this article for assistance with the solution. 

  • The solution requires Diagnostic-level recovery. 
  • The use of Diagnostic commands and recovery steps is potentially disruptive and should only be used with guidance from NetApp Support. 

Additional Information

additionalInformation_text

Internal Notes

 1. Clear the rdb flag on the node that has all cluster apps down:

  • Confirm the flag is set:
system configuration recovery node mroot-state clear -recovery-state
 
RDB Recovery State: 5550055000
VLDB: RDB auto-recovery failed to find master (5)
Management: RDB auto-recovery failed to find master (5)
VifMgr: RDB auto-recovery failed to find master (5)
Bcom: RDB auto-recovery failed to find master (0)
Crs: RDB auto-recovery failed to find master (0)
 
  • Clear the flag:
system configuration recovery node mroot-state clear -recovery-state rdb
 

 

 2. Delete vol0 snapshots and old core files:

  • Go into full vol0 nodeshell:
::> node run -node <affected Node>
 
  • Delete snap list for vol0:
snap del -a vol0
 
  • Set vol0 snap schedule to "0 0 0" so that no more snapshots are taken:
snap sched 0 0 0
 
 

 3. Reboot affected node.

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.