CONTAP-777838: ONTAP cluster node reboots unexpectedly
Issue
An ONTAP cluster node reboots unexpectedly:
The process name can be mgwd, secd, or bcomd, the process can be unresponsive for varying durations (in seconds) and it does not recover.
A generic node watchdog signature is not sufficient to match this issue. The disruption event must also have the FreeBSD memory-exhaustion signature:
Use the
PANIC: Process <process_name> unresponsive for <seconds> seconds ... in process nodewatchdog
The process name can be mgwd, secd, or bcomd, the process can be unresponsive for varying durations (in seconds) and it does not recover.
A generic node watchdog signature is not sufficient to match this issue. The disruption event must also have the FreeBSD memory-exhaustion signature:
- FreeBSD wired memory consumes nearly all of the BSD memory partition.
All known incidents show approximately 94% or more of wired memory. - Available BSD memory is reduced to a few hundred MB or less.
- Systemshell
top -bSoutput shows highWiredand lowfreememory usage.
Use the
systemshell command with NetApp Support guidance to determine the current mhost memory usage by running the top command. For example:cluster::> set -privilege diagnostic
cluster::*> systemshell -node <node_name> -command "top -bS | head -n 5"
last pid: 78952; load averages: 29.70, 28.44, 26.02 up 138+18:44:26
3339 processes:72 running, 282 sleeping, 2 zombie, 2983 waiting
CPU: 2.3% user, 0.0% nice, 36.9% system, 0.1% interrupt, 60.7% idle
Mem: 62M Active, 1132K Inact, 3532K Laundry, 15G Wired, 905M Buf, 253M Free
Swap: 8192M Total, 1400M Used, 6792M Free, 17% Inuse
