SAN increased latency with low throughput from VAAI Copy Offload

Applies to

• ONTAP 9
• VAAI Copy Offload
• vMotion

Issue

• System Manager dashboard displays a high LUN latency without a significant increase in IOPS or throughput.
• High latency is mostly with SAN Other operations. 
• QOS statistics list high latency from Data layer. 

::> qos statistics volume latency show -vserver SVM1 -volume volume1
    Workload  ID     Latency    Network   Cluster   Data       Disk     QoS Max    QoS Min      NVRAM      Cloud  FlexCache    SM Sync   VA AVSCAN
    --------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
    volume1   5935   321.31ms   143.00us  5.00us   321.15ms        0ms        0ms     0ms      10.00us       0ms    0ms         0ms        0ms     
    volume1   5935   200.02ms   132.00us  5.00us   199.88ms        0ms        0ms     0ms      9.00us        0ms    0ms         0ms        0ms     
    volume1   5935   263.63ms    76.00us 14.00us   263.52ms    18.00us        0ms     0ms      4.00us        0ms    0ms         0ms        0ms     
    volume1   5935   289.97ms   132.00us  4.00us   289.82ms        0ms        0ms     0ms      9.00us        0ms    0ms         0ms        0ms     
    volume1   5935   340.44ms   128.00us  7.00us   340.29ms        0ms        0ms     0ms      7.00us        0ms    0ms         0ms        0ms     

• HDD read/write or SSD read/write is much higher than FCP in/out or iSCSI in/out from the sysstat -x outputs. 

  • If VAAI Copy Offload source volume and destination volume are from the same node, Net in/out is as low as FCP in/out or iSCSI in/out in sysstat -x.
  • If VAAI Copy Offload source volume and destination volume are from different nodes, Net in/out is as high as HDD read/write or SSD read/write in sysstat -x.

Example 1: 

Notes:

• iSCSI + HDD
• VAAI Copy Offload Source volume and Destination volume on the same node
• Net in/out & iSCSI in/out are both much lower than HDD read/write

Example 2:

Notes: 

• FCP + SSD
• Source volume and Destination volume are from different nodes
• This is the Source node
• Net out is as high as SSD read, and they are both much higher than FCP out

Example 3: 

Notes:

• FCP + SSD
• Source volume and Destination volume are from different nodes
• This is the Destination node
• Net in is as high as SSD write, and they are both much higher than FCP in
• If SMas is configured with AFD policy, even the CPU might go high during these vMotion/Copy offload workloads