NFS operations failing due to high latency on backend
Applies to
- ONTAP SELECT
- NFSv4
Issue
- NFS Operations fail:
Example: Unable to tar a 6GB file.
tar: file.doc.test.tar: Cannot read: Invalid argument
tar: Too many errors, quitting
tar: Error is not recoverable: exiting
Throttling
andnfsLongRunningOp
errors observed inems.logs
:
[node01: intr: dev.driver.throttling:notice]: mpt driver unit 0 throttling I/O requests due to long latency.
[node01: intr: ems.engine.suppressed:debug]: Event 'dev.driver.throttling' suppressed 5028 times in last 601 seconds.
[node01: intr: dev.driver.throttling:notice]: mpt driver unit 2 throttling I/O requests due to long latency.
[node01: intr: ems.engine.suppressed:debug]: Event 'dev.driver.throttling' suppressed 5083 times in last 601 seconds.
[node01: kernel: Nblade.nfsLongRunningOp:debug]: Detected a long running network process operation. The client IP address:port is 10.123.123.57:719. The local IP address:port is 10.123.145.166:2049. The protocol requesting the operation is NFS4. The RPC Program Number for the operation is 100003. The RPC Procedure Number for the operation is 1. The disk process UUID is ff1c123456a123eab12345c12345ac6. The Vserver identifier is 1.
[node01: kernel: ems.engine.suppressed:debug]: Event 'Nblade.nfsLongRunningOp' suppressed 5 times in last 314 seconds.
[aspostnodp201: kernel: Nblade.nfsLongRunningOp:debug]: Detected a long running network process operation. The client IP address:port is 10.123.123.56:839. The local IP address:port is 10.123.145.166:2049. The protocol requesting the operation is NFS4. The RPC Program Number for the operation is 100003. The RPC Procedure Number for the operation is 1. The disk process UUID is ff1c123456a123eab12345c12345ac6. The Vserver identifier is 1.
[node01: kernel: ems.engine.suppressed:debug]: Event 'Nblade.nfsLongRunningOp' suppressed 3 times in last 3938 seconds.