CONTAP-400019: "NVRAM log not synchronized" SFO error for single node reporting "Send queue of QP WAFL is full"

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 90

Visibility:: Public

Votes:: 0

Category:: ontap-9

Specialty:: hw

Last Updated:

Issue

Both nodes in an HA pair sporadically reporting "NVRAM log not synchronized" messages. Example:

::> storage failover show -field reason
node        reason
----------- ----------------------------
node_name-1 "NVRAM log not synchronized"
node_name-2 "NVRAM log not synchronized"
2 entries were displayed.

The ONTAP Event Message System reports:

Fri Feb 14 12:55:32 +0100 [node_name-1: wafl_exempt03: rdma.rlib.queue.full:notice]: Send queue of QP WAFL is full.
...
Fri Feb 14 12:55:33 +0100 [node_name-1: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of node_name-2 disabled (unsynchronized log).
Fri Feb 14 12:55:33 +0100 [node_name-1: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of node_name-1 by node_name-2 disabled (unsynchronized log).
cleared after a few seconds:
Fri Feb 14 12:55:50 +0100 [node_name-1: cf_main: cf.fsm.takeoverByPartnerEnabled:notice]: Failover monitor: takeover of node_name-1 by node_name-2 enabled

And, the equivalent messages, in the partner, at the same time:

Fri Feb 14 12:55:34 +0100 [node_name-2: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of node_name-1 disabled (unsynchronized log).
Fri Feb 14 12:55:34 +0100 [node_name-2: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of node_name-2 by node_name-1 disabled (unsynchronized log).
...
Fri Feb 14 12:55:43 +0100 [node_name-2: cf_main: cf.fsm.takeoverByPartnerEnabled:notice]: Failover monitor: takeover of node_name-2 by node_name-1 enabled
Fri Feb 14 12:55:51 +0100 [ClusterA-02: cf_main: cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of node_name-1 enabled

No errors or issues noticed in the Interconnect ports and connections