Node unreachable due to cluster ports being down
Applies to
- FAS2720
- ONTAP 9.7P7
- BES-53248 cluster network switches
Issue
- Node is seen via
storage failover show
command
Cluster::> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- node-01 node-02 true Connected to node-02 node-02 node-01 true Connected to node-01 node-03 node-04 true Connected to node-04 node-04 node-03 true Connected to node-03
- Node appears unreachable via
node show
command
Cluster::> node show Node Health Eligibility Uptime Model Owner Location --------- ------ ----------- ------------- ----------- -------- --------------- node-01 true true 128 days 00:59 AFF-A220 London node-02 true true 128 days 00:59 AFF-A220 London node-03 true true 128 days 00:21 FAS2720 London node-04 true true - - - - 4 entries were displayed.
- Based on uptime the node is running
Cluster::> node run -node * -c "uptime" 4 entries were acted on. Node: node-01 2:32pm up 131 days, 2:30 14786852056 NFS ops, 3892 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops, 0 NVMF ops Node: node-02 2:32pm up 131 days, 2:30 20465055899 NFS ops, 3884 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops, 0 NVMF ops Node: node-03 2:32pm up 131 days, 1:52 981629257 NFS ops, 5809919861 CIFS ops, 0 HTTP ops, 0 FCP ops, 3209009366 iSCSI ops, 0 NVMF ops Node: node-04 2:32pm up 131 days, 1:52 7395009429 NFS ops, 4593427421 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops, 0 NVMF ops
- Cluster network ports are reported as down
slot 0: 10 Gigabit Ethernet Controller IX5-SFP+ e0a MAC Address: XX:XX:XX:XX:XX:XX (auto-unknown-fd-down) SFP Vendor: Molex Inc. SFP Part Number: 747529720 SFP Serial Number: XXXXXXXXX
- Cluster network switch connected to these ports is reported as missing
Node | node-02 |
Monitor | cluster-switch |
Alert ID | ClusterSwitchMissing_Alert |
Alerting Resource | switch-01 |
Subsystem | Switch-Health |
Indication Time | Sat Jun 26 13:42:57 2021 |
Perceived Severity | Major |
Probable Cause | Configuration_error |
Description | Redundant configuration is missing for cluster switches. |
- Visual inspection of the switch shows no LED activity on any of the cluster ports
- When connecting to the switch over serial all cluster ports are shown as having been disabled
(switch-01) #show port all Admin Physical Physical Link Link LACP Actor Intf Type Mode Mode Status Status Trap Mode Timeout --------- ------ --------- ---------- ---------- ------ ------- ------ -------- 0/1 Disable 10G Full Down Enable Enable long 0/2 Disable 10G Full Down Enable Enable long 0/3 Disable 10G Full Down Enable Enable long 0/4 Disable 10G Full Down Enable Enable long