Poor Performance in an ONTAP Select Instance due to vm.dev.diskSlowIO

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 420

Visibility:: Public

Votes:: 0

Category:: ontap-select

Specialty:: perf

Last Updated:

Applies to

ONTAP 9
ONTAP Select
VMware

Issue

CIFS clients access slowly.

NODE-01 ERROR Nblade.CifsOperationTimedOut: Detected a timed out CIFS operation. SMB command for this operation: SMB2_COM_SESSION_SETUP, Number of times this command was suspended: 1, Number of times this command was restarted: 0, Last CSM error during this operation: CSM_OK, Remote blade UUID: 00000000000000000000000000000000, Is QoS enabled: QoS_disabled, Last SpinNp error during this operation: SPINNP_NO_FO_ERROR, Client IP address: 10.89.243.174, Local IP address: 10.84.6.38, Target Vserver ID: 2, Target disk's DSID: 0

ONTAP EMS logs show the below other errors.

NODE-01 ALERT sk.panic: Panic String: spin lock 0xffffffff81217148 (smp rendezvous) held by 0xfffff80188858000 (CCMA-Worker) too long. VM may have been paused. Check VM host logs for an error. in SK process CCMA-Worker on release 9.3P11 (C)

NODE-01 EMERGENCY vm.host.error: The VM is paused possibly due to a full datastore or connectivity to the datastore might have been lost. Check the hypervisor logs for errors, and then perform any necessary corrective actions as soon as possible.

NODE-01 ALERT vnvram.dma.long.wait: vNVRAM flush taking over 10 seconds.

NODE-01 ERROR vm.dev.diskSlowIO: The read request on disk scsi3:0:0 took 25 seconds (data size 131072 bytes, threshold 20 seconds).

NODE-01 ERROR vm.dev.diskSlowIO: The write request on disk scsi0:0:0 took 47 seconds (data size 32768 bytes, threshold 20 seconds).

The VMware Hostd logs show the below error.

2021-06-05T20:12:00.008Z warning hostd[2486798] [Originator@6876 sub=Statssvc] Calculated read I/O size 583759 for scsi0:0 is out of range -- 583759,prevBytes = 10772597760 curBytes = 10817547264 prevCommands = 27094curCommands = 27171

The VMware Vmkernel logs show the below errors.

2021-06-05T20:12:11.584Z cpu2:2098072)ScsiDeviceIO: 3469: Cmd(0x459a79a2a380) 0x1a, CmdSN 0xd3d8c from world 0 to dev "naa.64cd98f05c7681002456ea353f4d93fb" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2021-06-05T20:13:22.054Z cpu20:2098072)ScsiDeviceIO: 3469: Cmd(0x459ab3313800) 0x4d, CmdSN 0x3f11 from world 2099775 to dev "naa.64cd98f05c768100248a2fefdc4606f1" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2021-06-05T20:16:47.745Z cpu36:2098022)DVFilter: 6053: Checking disconnected filters for timeouts