EMS reports "cf.fsm.takeoverByPartnerDisabled"

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 225

Visibility:: Public

Votes:: 0

Category:: fas-systems

Specialty:: hw

Last Updated:

Applies to

FAS8300
AFF A400
ONTAP 9

Issue

EMS reports "cf.fsm.takeoverOfPartnerDisabled"

Example:

[?] Thu Dec 02 03:56:45 +0900 [node01: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of node02 disabled (unsynchronized log).

[?] Thu Dec 02 03:56:45 +0900 [node01: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of node01 by node02 disabled (unsynchronized log).

[?] Thu Dec 02 03:56:45 +0900 [node01: cf_firmware: cf.fm.partnerFwTransition:info]: params: {'progresscounter': '0', 'newstate': 'SF_UNKNOWN', 'prevstate': 'SF_UP'}

[?] Thu Dec 02 03:56:47 +0900 [node01: cf_firmware: cf.fm.partnerFwTransition:info]: params: {'progresscounter': '4818229', 'newstate': 'SF_UP', 'prevstate': 'SF_UNKNOWN'}

[?] Thu Dec 02 03:56:47 +0900 [node01: kernel: netif.linkUp:info]: Ethernet e0a: Link up.

[?] Thu Dec 02 03:56:47 +0900 [node01: intr: rlib.ifconfig.linkEvent:notice]: params: {'eventType': 'UP', 'ifname': 'e0a'}

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_helper: nvpm.state.changed:debug]: Node 1's NVPM state changed from "2" to "2".

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_LAYOUT_SYNCING to NVMM_MIRROR_LAYOUT_SYNCED and took 1 msecs.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_LAYOUT_SYNCED to NVMM_MIRROR_SYNCING_START and took 0 msecs.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.aborting:debug]: mirror of sysid 1, partner_type HA Partner and mirror state NVMM_MIRROR_SYNCING_START is aborted because of reason NVMM_ERR_STREAM_MAP.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_error: nvmm.mirror.aborting:debug]: mirror of sysid 1, partner_type HA Partner and mirror state NVMM_MIRROR_OFFLINE is aborted because of reason NVMM_ABORT_SYNCING_MIRROR.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_13: rdma.rlib.connected:debug]: misc:HA:P QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_1: rdma.rlib.connected:debug]: wafl:HA:P QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_14: rdma.rlib.connected:debug]: raid:HA:P QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_17: rdma.rlib.connected:debug]: misc:HA:P QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_5: rdma.rlib.connected:debug]: wafl:HA:P QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_2: rdma.rlib.connected:debug]: raid:HA:P QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_13: rdma.rlib.connected:debug]: kstat:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_11: rdma.rlib.connected:debug]: ofw:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_1: rdma.rlib.connected:debug]: kill:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_6: rdma.rlib.connected:debug]: wafl:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_14: rdma.rlib.connected:debug]: raid:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_3: rdma.rlib.connected:debug]: stream:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_9: rdma.rlib.connected:debug]: wafl:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_17: rdma.rlib.connected:debug]: misc:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_LAYOUT_SYNCING to NVMM_MIRROR_LAYOUT_SYNCED and took 3 msecs.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_5: rdma.rlib.connected:debug]: hb:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_8: rdma.rlib.connected:debug]: raid:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_2: rdma.rlib.connected:debug]: ftfs:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: ib_cm_15: rdma.rlib.connected:debug]: misc:HA:A QP is now connected.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_SYNCING_START to NVMM_MIRROR_CP1_START and took 20 msecs.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_CP1_START to NVMM_MIRROR_WAFL_INIT and took 0 msecs.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_WAFL_INIT to NVMM_MIRROR_CP2_FINISH and took 20 msecs.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_CP2_FINISH to NVMM_MIRROR_WAFL_HEADER and took 161 msecs.

[?] Thu Dec 02 03:56:48 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_WAFL_HEADER to NVMM_MIRROR_SYNCING_OTHER and took 1 msecs.

[?] Thu Dec 02 03:56:49 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.state.change:debug]: mirror of sysid 1, partner_type HA Partner, changed state from NVMM_MIRROR_SYNCING_OTHER to NVMM_MIRROR_ONLINE and took 106 msecs.

[?] Thu Dec 02 03:56:49 +0900 [node01: nvmm_mirror_sync: nvmm.mirror.onlined:debug]: params: {'mirror': 'HA_PARTNER'}

[?] Thu Dec 02 03:56:49 +0900 [node01: cf_main: cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of node02 enabled

[?] Thu Dec 02 03:56:49 +0900 [node01: cf_main: cf.fsm.takeoverByPartnerEnabled:notice]: Failover monitor: takeover of node01 by node02 enabled

Flapping interconnect link issue in node

Example:

Thu Dec 02 03:56:45 +0900 [node01: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Thu Dec 02 03:56:47 +0900 [node01: kernel: netif.linkUp:info]: Ethernet e0a: Link up.

Thu Dec 02 04:20:30 +0900 [node01: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Thu Dec 02 04:20:32 +0900 [node01: kernel: netif.linkUp:info]: Ethernet e0a: Link up.

Thu Dec 02 04:26:07 +0900 [node01: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Thu Dec 02 04:26:10 +0900 [node01: kernel: netif.linkUp:info]: Ethernet e0a: Link up.

Thu Dec 02 04:49:21 +0900 [node01: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Thu Dec 02 04:49:23 +0900 [node01: kernel: netif.linkUp:info]: Ethernet e0a: Link up.

Thu Dec 02 04:51:52 +0900 [node01: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Thu Dec 02 04:51:54 +0900 [node01: kernel: netif.linkUp:info]: Ethernet e0a: Link up.

Thu Dec 02 04:57:00 +0900 [node01: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Thu Dec 02 04:57:03 +0900 [node01: kernel: netif.linkUp:info]: Ethernet e0a: Link up.

Thu Dec 02 05:05:32 +0900 [node01: kernel: netif.linkDown:info]: Ethernet e0a: Link down, check cable.