High IO waits and latency on VMs connected to NetApp FC LUN

• NetApp FAS and AFF storage
• Fibre Channel (FC) LUNs
• Oracle DB

• Oracle DB running on Linux VM observed high IO and latency.
• Reviewing IO stat on VM, could notice high disk utilization :

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

sdc      0.00 0.00 0.00 0.67 0.00 2.67 8.00    6.00     0.00 0.00       0.00 1500.00 100.00

sda      0.00 0.00 0.00 0.33 0.00 1.33 8.00    10.00      0.00 0.00       0.00 3000.00 100.00

dm-0      0.00 0.00 0.00 0.33 0.00 1.33 8.00    12.0       0.00 0.00       0.00 3000.00 100.00

dm-3      0.00 0.00 0.00 0.67 0.00 2.67 8.00    5.00      0.00 0.00       0.00 1500.00 100.0