High write latency due to NVLog Transfer on MetroCluster IP

• ONTAP 9
• MetroCluster IP （MCCIP）

• High write latency on MCCIP systems
• From CLI, Check using command qos statistics volume latency show

Example:

cluster::> qos statistics volume latency show -node test_node1

Workload        ID    Latency    Network    Cluster       Data       Disk      QoS      NVRAM      Cloud 

------------ ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- -

volume_1      12578  15426ms      1ms        0ms         0.5us         1ms      0ms       15423ms        0ms  

• No obvious error counters are found on MCCIP backend swtiches.
• The issue is initailed by the mediator mailbox disks lost then reconnection. 

<LR d="02Jan2023 08:46:39" n="test_node1" t="1672620399" id="1627716454/1018077" p="5" s="Ok" o="kernel" vf="" type="0" seq="1160087" >

<iscsi_session_stateChanged_1

        iscsi_session_state="Reconnecting"

        iscsi_target="iqn.2012-05.local:mailbox.target.3d77a691-cb82-11eb-9d94-00a098f45ac6:c14f72ef-cb82-11eb-9760-00a098f467e6:1"

        iscsi_target_type="mailbox"

        iscsi_target_address="x.x.x.x"

        iscsi_session_state_change_reason="Reason: no ping reply after 5 seconds."

        iscsi_timeout_value="5"/>

<LR d="02Jan2023 09:00:28" n="test_node1" t="1672621228" id="1627716454/1018143" p="7" s="Ok" o="fmmbx_diskOpsManager" vf="" type="0" seq="1160154" >

<fmmb_disk_io_replyFailed_1

        disk_name="0f.3"

        uuid="37326132:63393764:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000"

        status="disk does not exist"

        operation="MBX_DISKIO_WRITE_SPECIFIED_BLOCK"

        side="Local"

        handle="0x0"

        flag="0x44"

        io_state="0"/>

</LR>

• As the mailbox disks are rediscovered and reset, the NVMM mirror relationship is also reset, including the mirror status of the DR partner (remote) and HA partner (local).

<LR d="02Jan2023 09:02:12" n="test_node1" t="1672621332" id="1627716454/1018256" p="7" s="Ok" o="cf_worker" vf="" type="0" seq="1160267" >

<nvmm_mirror_aborting_1

        partner_sysid="2"

        partner_type="DR PARTNER"

        mirror_state="NVMM_MIRROR_ONLINE"

        error="NVMM_ERR_DRC_TRIGGER"/>

 

<LR d="02Jan2023 09:02:13" n="test_node1" t="1672621333" id="1627716454/1018266" p="7" s="Ok" o="mcc_cfd_rnic" vf="" type="0" seq="1160277" >

<mirror_stream_qp_error_1

        mirror="HA Partner"

        qp_name="MISC"

        error="NVMM_ERR_STREAM"/>