CVO Azure: Controller Failover of node is not possible: repeated unsynchronized log
Applies to
- Cloud Volumes ONTAP (CVO)
- Microsoft Azure
- Unsynchronized log
- Input/Output (IO)
Issue
Mon Oct 02 04:24:50 -0400 [cvo-01: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of cvo-02 disabled (unsynchronized log).
Mon Oct 02 04:24:53 -0400 [cvo-01: cf_main: cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of cvo-02 enabled
EMS.LOG.GZ reports underlying QPs are getting POLL_TIMEOUT errors.
Tue Sep 05 17:50:07 -0400 [cvo-02: mcc_cfd_rnic: nvmm.mirror.aborting:debug]: mirror of sysid 1, partner_type HA Partner and mirror state NVMM_MIRROR_ONLINE is aborted because of reason NVMM_ERR_POLL_TIMEOUT.
Tue Sep 05 17:50:07 -0400 [cvo-02: mcc_cfd_rnic: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'RAID', 'error': 'NVMM_ERR_POLL_TIMEOUT'}
Tue Sep 05 17:50:07 -0400 [cvo-02: mcc_cfd_rnic: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'MISC', 'error': 'NVMM_ERR_STREAM'}
Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'WAFL', 'error': 'NVMM_ERR_MIRROR_COMPLETION'}
Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'WAFL', 'error': 'NVMM_ERR_STREAM'}
Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: ems.engine.suppressed:debug]: Event 'rdma.rlib.event.error' suppressed 2 times in last 2754 seconds.
Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: rdma.rlib.event.error:debug]: QP wafl event error: client disconnect.
Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: nvmm.mirror.offlined:debug]: params: {'mirror': 'HA_PARTNER'}
Tue Sep 05 17:50:07 -0400 [cvo-02: rastrace_dump: rastrace.dump.saved:debug]: A RAS trace dump for module IC instance 0 was stored in /etc/log/rastrace/IC_0_20230905_17:50:07:464435.dmp.
Tue Sep 05 17:50:07 -0400 [cvo-02: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of cvo-02 by cvo-01 disabled (unsynchronized log).
SKTRACE.GZ logs indicate RAID IO completion is taking more time.
2023-09-06T19:15:32Z 3435898944123351 [14:0] BSD_RLIB_WARN: rlib_update_comp_info: qp wafl wr_id 300000000000000000 num_of_req 2 op_type 1 op_size 256 took time 2321466 usecs pending_size:2184520
2023-09-06T19:15:32Z 3435898944131159 [14:0] BSD_RLIB_WARN: rlib_update_comp_info: qp wafl wr_id 310000000000000000 num_of_req 2 op_type 1 op_size 344 took time 2320541 usecs pending_size:2184176
2023-09-06T19:15:32Z 3435898944168065 [14:0] BSD_RLIB_WARN: rlib_update_comp_info: qp wafl wr_id 312000000000000000 num_of_req 2 op_type 1 op_size 4160 took time 2313630 usecs pending_size:2180016