High await and util while accessing the luns on host

• ONTAP 9
• FCP
• iSCSI

• Slowness observed on the host while accessing the LUNs hosted on ONTAP.
• Long wait time and high utilization observed from the hosts.

Example: High await and %util observed from RedHat Linux

Notes:

• Other hosts willl have different ways to measure the wait time and utilization.
• For RedHat Linux, refer to [Troubleshooting] Understanding iostat -x output to understand what await and %util mean.
• High latency may or may NOT be reported from qos statistics volume latency show. 
  • No high latency reported when SAN Opsize is small, mostly < 64KB. 
  • High latency reported in Network column in qos statistics volume latency show when SAN Opsize is large, mostly > 64KB.
  • The reason is: 
    • When the Opsize is larger than 64KB, a SAN Op will be split into multiple PDUs. The round trip time between these PDUs will also be accounted in the latency reported from ONTAP, this will take the external factors into the overall latency. Such as Switch or the network in between.
    • When Opsize is smaller than 64KB, the external factors will not be accounted in the ONTAP latency. 
    • For more info, refer to Why is the reported LUN latency higher than the volume latency?
• There's a trend of degrading Tx/Rx FC optical power values in which can be verified with the command:

Cluster::> fcp adapter show -node xxxxx -adapter xx

Example:

Received Optical Power    570.7 (uWatts)
SPF Transmitted Optical Power    123.8 (uWatts)