Node unreachable due to cluster ports being down
Applies to
- FAS2720
 - ONTAP 9.7P7
 - BES-53248 cluster network switches
 
Issue
- Node is seen via 
storage failover showcommand 
Cluster::> storage failover show
                              Takeover
Node           Partner        Possible State Description
-------------- -------------- -------- -------------------------------------
node-01        node-02 true   Connected to node-02
node-02        node-01 true   Connected to node-01
node-03        node-04 true   Connected to node-04
node-04        node-03 true   Connected to node-03
- Node appears unreachable via 
node showcommand 
Cluster::> node show Node Health Eligibility Uptime Model Owner Location --------- ------ ----------- ------------- ----------- -------- --------------- node-01 true true 128 days 00:59 AFF-A220 London node-02 true true 128 days 00:59 AFF-A220 London node-03 true true 128 days 00:21 FAS2720 London node-04 true true - - - - 4 entries were displayed.
- Based on uptime the node is running
 
Cluster::> node run -node * -c "uptime" 4 entries were acted on. Node: node-01 2:32pm up 131 days, 2:30 14786852056 NFS ops, 3892 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops, 0 NVMF ops Node: node-02 2:32pm up 131 days, 2:30 20465055899 NFS ops, 3884 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops, 0 NVMF ops Node: node-03 2:32pm up 131 days, 1:52 981629257 NFS ops, 5809919861 CIFS ops, 0 HTTP ops, 0 FCP ops, 3209009366 iSCSI ops, 0 NVMF ops Node: node-04 2:32pm up 131 days, 1:52 7395009429 NFS ops, 4593427421 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops, 0 NVMF ops
- Cluster network ports are reported as down
 
slot 0: 10 Gigabit Ethernet Controller IX5-SFP+ e0a MAC Address: XX:XX:XX:XX:XX:XX (auto-unknown-fd-down) SFP Vendor: Molex Inc. SFP Part Number: 747529720 SFP Serial Number: XXXXXXXXX
- Cluster network switch connected to these ports is reported as missing
 
| Node | node-02 | 
| Monitor | cluster-switch | 
| Alert ID | ClusterSwitchMissing_Alert | 
| Alerting Resource | switch-01 | 
| Subsystem | Switch-Health | 
| Indication Time | Sat Jun 26 13:42:57 2021 | 
| Perceived Severity | Major | 
| Probable Cause | Configuration_error | 
| Description | Redundant configuration is missing for cluster switches. | 
- Visual inspection of the switch shows no LED activity on any of the cluster ports
 - When connecting to the switch over serial all cluster ports are shown as having been disabled
 
(switch-01) #show port all
                 Admin     Physical   Physical   Link   Link    LACP   Actor
Intf      Type   Mode      Mode       Status     Status Trap    Mode   Timeout
--------- ------ --------- ---------- ---------- ------ ------- ------ --------
0/1              Disable   10G Full              Down   Enable  Enable long
0/2              Disable   10G Full              Down   Enable  Enable long
0/3              Disable   10G Full              Down   Enable  Enable long
0/4              Disable   10G Full              Down   Enable  Enable long
