Switch-Health Status Shows "Unknown" After ONTAP Upgrade
Applies to
- ONTAP 9
- NVIDIA Switch
Issue
- After upgrading the ONTAP, the switch-health monitoring subsystem reports an "unknown" status and to be hung in start-discovery state
Example Log Output:
::> system health subsystem show -subsystem switch-health -instance
Subsystem: Switch-Health
Health: unknown
InitializationState: start_discovery
Number of Outstanding Alerts: 0
Number of Suppressed Alerts: 0
Node: <Node-Name>
SubsystemRefreshInterval: 5m
- However, the switches were being monitored successfully via SNMP
::> system switch ethernet show
Switch Type Address Model
--------------------------- ------------------ ---------------- ---------------
NETAPP-SW1 (9X:XX:XX:XX:XX:XX) storage-network 10.xx.xx.xx MSN2100-CB2FC
Serial Number: MT2302TXXXXX
Is Monitored: true
Reason: None
Software Version: Cumulus Linux version 5.11.0 running on Mellanox
Technologies Ltd. MSN2100
Version Source: SNMP
NETAPP-SW2 (8X:XX:XX:XX:XX:XX) storage-network 10.xx.xx.xx MSN2100-CB2FC
Serial Number: MT2308TXXXXX
Is Monitored: true
Reason: None
Software Version: Cumulus Linux version 5.11.0 running on Mellanox
Technologies Ltd. MSN2100
Version Source: SNMP
- Removing and re-adding the switch does not resolve the issue
::*> system switch ethernet delete -device <device_name>
::*> system switch ethernet create -device "<DEVICE_NAME> (<MAC_ADRESS>)" -address <IP_ADDRESS> -snmp-version <SNMP_VERSION> -community-or-username <COMUNITY_OR_USERNAME> -model <MODEL> -type <TYPE> -is-monitoring-enabled-admin <TRUE_OR_FALSE>
- Restarting "cshmd" also gets stuck at start discovery phase