Skip to main content
NetApp Knowledge Base

ONTAP Node fails and won't start with vldb or vifmgr error on boot: “PANIC: Process vifmgr unresponsive for xxx seconds in process nodewatchdog on release 9.x”

Views:
1,876
Visibility:
Public
Votes:
1
Category:
ontap-9
Specialty:
core
Last Updated:

Applies to

  • ONTAP 9

Issue

  • The node panicked due to vifmgr not responding before the watchdog timed out and was unable to recover the mdbs when it rebooted:

Panic String: PANIC : Process vifmgr unresponsive for 629 seconds version: 9.1P12

or

Panic String: PANIC : Process vldb unresponsive for 160 seconds in process nodewatchdog on release 9.10.1P12
 

  • The root volume filled up due to high snapshot delta and snapshot space utilization on vol0 resulting from rolling packet trace that was running for 2 weeks:

Mon Mar 30 08:28:37 CDT [nodename: rshd_0: kern.cli.cmd:debug]: Command-line input: The command is 'pktt'. The full command line is 'pktt start a0a-10 -d /etc/crash -m 9018 -b 8m -s 2g -r 12'.

  • Immediately prior to the panic the console log showed that vifmgr and vldb crashed and were unable to restart:

Apr 13 00:49:45 [nodename:spm.vldb.process.exit:EMERGENCY]: Volume Location Database(VLDB) subsystem with ID 34409 exited as a result of signal normal exit (1). The subsystem will attempt to restart.

Apr 13 00:49:47 [nodename:spm.vifmgr.process.exit:EMERGENCY]: Logical Interface Manager(VifMgr) with ID 34415 aborted as a result of signal normal exit (1). The subsystem will attempt to restart.

  • When the node reboots it is unable to recovery the mdbs due to lack of space on vol0:

Apr 13 02:54:46 [nodename:callhome.mdb.recovery.unsuccessful:EMERGENCY]: Call home for MDB RECOVERY UNSUCCESSFUL FOR THE notifyd WARNING.

ln: /var/zoneinfo/zoneinfo: No space left on device

root: Unable to ln /mroot/etc/zoneinfo to /var/zoneinfo - error code(1)

/usr/bin/plxcoeff_log: cannot create /mroot/etc/log/plxcoeff/plxcoeff.log.tmp: No space left on devicestat: /mroot/etc/log/plxcoeff/plxcoeff.log.tmp: stat: No such file or directory

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.