How to collect logs for HA IC interconnect Link Down or RDMA down issues
Applies to
- High Availability Interconnect (HA IC)
- ONTAP 9
- Onprem ONTAP systems (Fabric Attached Storage (FAS)/All Flash FAS (AFF))
- Cloud Volumes ONTAP (CVO)
- ONTAP Select
- HA IC Link down
- HA IC RDMA(Remote Direct Memory Access) down
- Unsynchronized NVRAM log
- Takeover disabled
- eMulated Virtual Interface Adapter (MVIA) protocol
- iWARP
Description
HA IC is used for the below:
- NVRAM mirroring
- Exchange heartbeats, boot status information and failover state
- To support controller failover (CFO or root aggregate) and storage failover(SFO or data aggregates) capabilities in an HA pair
HA IC Link on On-prem systems is via InfiniBand Connection or via Software iwarp on MCC IP.
On CVO AWS and GCP - HA IC Link is over software MVIA and on CVO Azure and ONTAP Select HA IC is over Software iwarp
Impacts of HA IC or RDMA down:
- Takeover Disabled
- Unsynchronized NVRAM logs
Cluster::*> system ha interconnect status show Node: Cluster-01 Link Status: up IC RDMA Connection:down Node: Cluster-02 Link Status: up IC RDMA Connection: down 2 entries were displayed.
Cluster::*> storage failover show Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- Cluster-01 Cluster-02 false Waiting for Cluster-02, Takeover is not possible: NVRAM log not synchronized Cluster-02 Cluster-01 false Waiting for Cluster-01, Takeover is not possible: NVRAM log not synchronized
EMS.log: ONTAPSelect-A ALERT callhome.hainterconnect.down: Call home for HA INTERCONNECT DOWN due to peer not connected. ONTAPSelect-A ERROR ic.HAInterconnectDown: HA interconnect: Interconnect down for 839 minutes: peer not connected ONTAPSelect-A ALERT cf.takeover.disabled: HA mode, but takeover of partner is disabled due to reason : unsynchronized log.