Node panics due to root volume being full
Applies to
ONTAP 9
Issue
- Node panics with a process (vifmgr, vldb, mgwd, etc.) being unresponsive:
PANIC: Process vifmgr unresponsive for 180 seconds in process nodewatchdog on release 9.7P9 (C) on Sun Nov 21 16:47:29 CET 2021
- During boot node may report the following:
[node-01:callhome.mdb.recovery.unsuccessful:EMERGENCY]: Call home for MDB RECOVERY UNSUCCESSFUL FOR THE vifmgr WARNING.
[node-01:boot.varfs.backup.issue:EMERGENCY]: Backup of the /var file system failed (cp: /mroot/etc/tmpvarfs.tgz: No space left on device
[node-01:callhome.mdb.recovery.unsuccessful:EMERGENCY]: Call home for MDB RECOVERY UNSUCCESSFUL FOR THE crs WARNING.
[node-01:callhome.mdb.recovery.unsuccessful:EMERGENCY]: Call home for MDB RECOVERY UNSUCCESSFUL FOR THE bcomd WARNING.
[node-01:callhome.mdb.recovery.unsuccessful:EMERGENCY]: Call home for MDB RECOVERY UNSUCCESSFUL FOR THE vifmgr WARNING.
[node-01:mgmtgwd.rootvolrec.low.space:EMERGENCY]: The root volume on node "cluster-01" is dangerously low on space. Less than 10 MB of free space remaining.
[node-01:callhome.root.vol.recovery.reqd:EMERGENCY]: Call home for ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED.
DF
may shows root volume snapshot spill large space.
Filesystem kbytes used avail capacity Mounted on
/vol/vol0/ 150707748 150201388 506360 100% /vol/vol0/
/vol/vol0/.snapshot 7931984 15559744 0 196% /vol/vol0/.snapshot