EMERGENCY Controller failover of node_name is not possible. Unsynchronized log

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 6,046

Visibility:: Public

Votes:: 1

Category:: aff-series

Specialty:: hw

Last Updated:

Applies to

AFF A700
FAS9000
ONTAP 9

Issue

EMS Emergency message:

Sun Apr 18 09:50:00 +0200 [node_name_1: monitor: monitor.globalStatus.critical:EMERGENCY]: Controller failover of node_name_2 is not possible: unsynchronized log.

This is a transient issue not persistent.
Storage failover is enabled.
System Health is fine and no Health Alerts present.
Event logs example:

Time Node Severity Event

----------------- --------- ------- -------------------------------

4/8/2021 08:29:18 node_name_1 ERROR    cf.fsm.takeoverByPartnerDisabled: Failover monitor: takeover of node_name_1 by node_name_2 disabled (unsynchronized log).

4/8/2021 08:29:18 node_name_1 ERROR    cf.fsm.takeoverOfPartnerDisabled: Failover monitor: takeover of node_name_2 disabled (unsynchronized log).

4/8/2021 08:29:16 node_name_1 ERROR    cf.fsm.takeoverOfPartnerDisabled: Failover monitor: takeover of node_name_2 disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

4/8/2021 08:29:15 node_name_1 ERROR    cf.fsm.takeoverByPartnerDisabled: Failover monitor: takeover of node_name_1 by node_name_2 disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

4/8/2021 07:49:11 node_name_1 ERROR    cf.fsm.takeoverByPartnerDisabled: Failover monitor: takeover of node_name_1 by node_name_2 disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

4/8/2021 07:49:11 node_name_1 ERROR    cf.fsm.takeoverOfPartnerDisabled: Failover monitor: takeover of node_name_2 disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).

Flapping interconnect link issue in both nodes:

Sun Apr 18 07:47:41 +0200 [node_name_1: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is down.

Sun Apr 18 07:47:44 +0200 [node_name_1: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is up.

Sun Apr 18 09:49:46 +0200 [node_name_1: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is down.

Sun Apr 18 09:49:49 +0200 [node_name_1: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is up.

...

Sun Apr 18 07:47:41 +0200 [node_name_2: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is down.

Sun Apr 18 07:47:44 +0200 [node_name_2: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is up.

Sun Apr 18 09:49:46 +0200 [node_name_2: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is down.

Sun Apr 18 09:49:49 +0200 [node_name_2: gop_eq_thread: ic.linkStatusChange:info]: HA interconnect: Port ic6a link is up.

....