CONTAP-364039: Unexpected heartbeat loss causing takeover events on FAS8300/AFF A400 MetroCluster IP systems
Issue
- Frequent Heartbeat miss reported, which can lead to a takeover of a node.
- Node undergoing takeover will be unresponsive just prior to Heartbeat miss, with no node logs during that time.
- Unresponsive node which undergoes takeover will not be reporting any hardware errors around the issue time.
- Partner node, which takes over, will usually log SCSI IO transport errors, NVMM mirror sync fails, QP errors, HA errors, etc.
- Example of partner node logs:
Jan 01 01:01:01 [node_name: DR_heartbeat_thread: cf.ic.xferTimedOut:error]: HA interconnect: MCC_DRSOM transfer timed out.
Jan 01 01:01:01 [node_name: DR_heartbeat_thread: rdma.rlib.event.error:debug]: QP drsom_hb event error: client disconnect.
Jan 01 01:01:01 [node_name: iw_cxgbe: nvmm.mirror.aborting:debug]: mirror of sysid 1, partner_type HA Partner and mirror state NVMM_MIRROR_ONLINE is aborted because of reason NVMM_ERR_MIRROR_COMPLETION.
Jan 01 01:01:01 [node_name: rastrace_dump: rastrace.dump.saved:debug]: A RAS trace dump for module IC instance 0 was stored in /etc/log/rastrace/IC_0_20250101_01:01:01:123456.dmp.
Jan 01 01:01:01 [node_name: cf_firmware: cf.ic.xferTimedOut:error]: HA interconnect: OFW transfer timed out.
Jan 01 01:01:03 [node_name: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of node_name-01 by node_name-02 disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).
Jan 01 01:01:04 [node_name: cf_main: cf.fsm.partnerNotResponding:notice]: Failover monitor: partner not responding
Jan 01 01:02:01 [node_name: cf_main: cf.fsm.takeoverCountdown:info]: Failover monitor: takeover scheduled in 10 seconds
Jan 01 01:02:10 [node_name: cf_main: cf.fsm.takeover.noHeartbeat:alert]: Failover monitor: Takeover initiated after no heartbeat was detected from the partner node.