The VMware snapshot times out on a large VM located on a VVOL datastore

Applies to

• ONTAP Tools for VMware vSphere (OTV)
• VMware vSphere Virtual Machine (VM)
• ONTAP storage 

Issue

• A VM snapshot fails with error:

snapshot operation timed out

• The vvold log of ESXi shows:

2023-07-27T08:50:46.210Z error vvold[3160113] [Originator@6876 sub=Default opID=abc-781022-auto-gqn3-h5:70400761-2e-1-cec0] VasaOp::TimedOut [#1281293]: ===> prepareToSnapshotVirtualVolume failed: (TRANSPORT_FAULT /UNKNOWN FAULT) - RETRY TIMEOUT (94/90 secs)!
2023-07-27T08:50:46.210Z error vvold[3160113] [Originator@6876 sub=Default opID=abc-781022-auto-gqn3-h5:70400761-2e-1-cec0] VasaOp[#1281293] ===> FINAL FAILURE prepareToSnapshotVirtualVolume, error (TIMEOUT / Retry Timeout) VP (NetApp-VP) Container (<id>) timeElapsed=94313 msecs (#outstanding 0)

• The OTV vvolvp.log shows:

vvolvp.10:2023-07-27T08:38:16,094Z (WARN )[Thread-1251485] [] invokeZAPI(0): failed OntapConnectionImpl{ipAddress=<ip>, userName=user, port=443, ssl=true} API failed. Failed to create token for node "<node>". Reason: Reached maximum split load allowed on the node. (errno=13001) null
vvolvp.10:2023-07-27T08:39:30,272Z (ERROR)[qtp2038105753-43] [cxf-1109805:prepareToSnapshotVirtualVolume-1235] NaException com.netapp.offtap3.ontap.ApiFailedException: Failed to create token for node "<node>". Reason: Reached maximum split load allowed on the node. (errno=13001)