A400 HA Interconnect ports down

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 205

Visibility:: Public

Votes:: 0

Category:: fas-systems

Specialty:: hw

Last Updated:

Applies to

AFF A400
HA Interconnect connection trhough e0a and e0b ports.

Issue

HA Interconnect ports offline, in both nodes. Example:

NODE-A

[NODE-A: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

[NODE-A: kernel: netif.linkDown:info]: Ethernet e0b: Link down, check cable.

[NODE-A: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of NODE-A by NODE-B disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

[NODE-A: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of NODE-B disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

[NODE-A: monitor: monitor.globalStatus.critical:EMERGENCY]: Controller failover of NODE-B is not possible: HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support. There are not enough spare disks.

[NODE-A: statd: cf.takeover.disabled:alert]: HA mode, but takeover of partner is disabled due to reason : HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support.

[NODE-A: statd: ic.HAInterconnectDown:error]: HA interconnect: Interconnect down for 15 minutes: all links are down

[NODE-A: statd: callhome.hainterconnect.down:alert]: Call home for HA INTERCONNECT DOWN due to all links are down.

NODE-B

[NODE-B: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

[NODE-B: kernel: netif.linkDown:info]: Ethernet e0b: Link down, check cable.

[NODE-B: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of NODE-B by NODE-A disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

[NODE-B: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of NODE-A disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

[NODE-B: monitor: monitor.globalStatus.critical:EMERGENCY]: Controller failover of NODE-A is not possible: HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support. There are not enough spare disks.

[NODE-B: statd: cf.takeover.disabled:alert]: HA mode, but takeover of partner is disabled due to reason : HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support.

[NODE-B: statd: ic.HAInterconnectDown:error]: HA interconnect: Interconnect down for 15 minutes: all links are down

[NODE-B: statd: callhome.hainterconnect.down:alert]: Call home for HA INTERCONNECT DOWN due to all links are down.

SFP information for both nodes is present in ::> system node run -node * -command sysconfig -a output but, ports are down:

NODE-A

slot 0: Dual 10G/25G Ethernet Controller CX5 e0a MAC Address: d0:39:ea:2b:c3:31 (auto-unknown-fd-down) SFP Vendor: Molex SFP Part Number: 1111455002 SFP Serial Number: 93A2023020111 e0b MAC Address: d0:39:ea:2b:c3:32 (auto-unknown-fd-down) SFP Vendor: Molex SFP Part Number: 1111455002 SFP Serial Number: 93A2012020222 Device Type: CX5 PSID(NAP0000000006) Firmware Version: 16.26.4012
NODE-B

slot 0: Dual 10G/25G Ethernet Controller CX5 e0a MAC Address: d0:39:ea:2b:c2:c1 (auto-unknown-fd-down) SFP Vendor: Molex SFP Part Number: 1111455002 SFP Serial Number: 93A2023020111 e0b MAC Address: d0:39:ea:2b:c2:c2 (auto-unknown-fd-down) SFP Vendor: Molex SFP Part Number: 1111455002 SFP Serial Number: 93A2012020222 Device Type: CX5 PSID(NAP0000000006) Firmware Version: 16.26.4012

"::*> system ha interconnect status show" appears link 0 and link 1 are down status. Example:

::> set advanced

::*> system ha interconnect status show Node:node-1 Link 0 Status: down Link 1 Status: down Is Link 0 Active: false Is Link 1 Active: false IC RDMA Connection: up Node:node-2 Link 0 Status: down Link 1 Status: down Is Link 0 Active: false Is Link 1 Active: false IC RDMA Connection: up 2 entries were displayed.

The issue remains after re-seating each HA interconnect SFPs and cables connection.
Loopback test agains HA interconnect ports, in the same node, brings both ports back online.
Both nodes remain online but no takeover:

::*> storage failover show -node * Takeover Node Partner Possible State Description -------------- -------------- -------- ------------------------------------- node-1 node-2 false Waiting for node-2, Takeover is not possible: Storage failover interconnect error, NVRAM log not synchronized node-2 node-1 false Waiting for node-1, Takeover is not possible: Storage failover interconnect error, NVRAM log not synchronized 2 entries were displayed.