vSphere hosts crash after MaxQueueDepth change on NetApp AFF-A90
Applies to
- ONTAP 9
- VMware vSphere ESXi
- NFS protocol
- Broadcom (VMware parent company)
Issue
- In the vSphere environment, the NFS.MaxQueueDepth and NFS41.MaxQueueDepth values were changed from 4294967295 to 128 based on NetApp KB recommendations.
- Approximately 42 hours after this change, all ESXi hosts running virtual machines on NetApp storage (AFF-A90 running ONTAP 9.15.1P6) experienced rolling PSODs (Purple Screen of Death), resulting in the hosts going offline and a full environment crash.
- The PSODs referenced the InfSvc41Client module with involvement of NFS41_SchedTaskSubmit, NFS41FileOp, and SwapIOClusterWriteWrapper. ESXi hosts running virtual machines on other storage devices were not affected.
- The issue occurred on the 'RangeCluster' ESXi hosts after they were rebooted to apply the MaxQueueDepth changes. The 'PRODCluster' hosts, which did not have the MaxQueueDepth changed, were unaffected.
- PSOD stack traces is:
Failed to insert scheduling element:0x432f2aaadcdO into the priQ:0x432f2a404950. Reason:Limit exceededNOT_IMPLEMENTED bora/vmkernel/sched/memsched_kmem.c:1436
