FC Connectivity to Node Remains Down After It Was Out of CLAM Quorum
Applies to
- ONTAP 9
- VMware ESXi
- FC
Issue
- Nodes out-of-quorum due to network issue, but this was resolved:
Node (name=a1n5, ID=1008) is out of "CLAM quorum" (reason=quorum update).
Node (name=a1n6, ID=1009) is out of "CLAM quorum" (reason=quorum update).
Cluster node (name=a1n5, ID=1008) is in "CLAM quorum".
Cluster node (name=a1n6, ID=1009) is in "CLAM quorum".
- Datastore paths to LUNs on these nodes remain down
- ONTAP does not see logins from ESXi hosts to affected nodes
- Creating new LUN mappings fails due to
No nodes are available
:
::> lun mapping create -vserver vs1 -path /vol/vol1/lun1 -igroup ig1
Error: command failed: No nodes are available to process the command. Verify that all nodes are healthy using the "cluster show" command, then try the command again.
bcomd.log
entry at time of failed LUN mappings fails due toSAN kernel agent [...] cannot communicate with other cluster nodes
:
SAN management operation failed: The SAN kernel agent on node "a1n5" cannot communicate with other cluster nodes. Verify cluster quorum status using the (privilege: advanced) "cluster kernel-service show" command and resolve all node out-of-quorum issues. Once they are addressed, retry the command.
- Cluster does not show out-of-quorum issues:
::> cluster show
Node Health Eligibility
--------------------- ------- ------------
a1n1 true true
a1n2 true true
a1n3 true true
a1n4 true true
a1n5 true true
a1n6 true true
a1n7 true true
a1n8 true true
::*> cluster kernel-service show
Master Cluster Quorum Availability Operational
Node Node Status Status Status
----------------- ----------------- ------------- ------------- -------------
a1n7 a1n1 in-quorum true operational
a1n2 in-quorum true operational
a1n3 in-quorum true operational
a1n4 in-quorum true operational
a1n5 in-quorum true operational
a1n6 in-quorum true operational
a1n7 in-quorum true operational
a1n8 in-quorum true operational
- Rebooting ESXi hosts does not resolve the issue