SnapMirror operations fails due to failure to perform I/O with the capacity tier
Applies to
- ONTAP 9
- SnapMirror
- ObjectStore
Issue
- SnapMirror update failed with one of the below errors:
Transfer failed. (Filesystem error (Failure to perform I/O with the capacity tier, which has been declared unresponsive))
Transfer failed.(Filesystem error(Failure to perform I/O with the capacity tier. Retry the operation))
- EMS log: latency from cloud more than threshold (5000 msec)
Tue Feb 20 01:21:17 +0800 [NetApp-02: OscLowPriThreadPool: wafl.ca.latency.threshold:notice]: Measured latency from cloud (99988 msec) more than threshold (5000 msec) (aggregate: "test", aggregate uuid: "287a4276-4694-7583-a3e2-1e83b7ebc51c", object store: "OSC-CT-443")
- Sktrace log: Connection unavailable and Operation Timed out
2024-02-19T16:16:23Z 29244161221223132 [9:0] CLOUD_BIN_ERR: object_store_config_check_connection: HEAD failed for s3 handle 109 at try_count 1 with error 5 (Connection unavailable)
2024-02-19T16:16:34Z 29244184316598880 [10:0] CLOUD_BIN_ERR: object_store_config_check_connection: PUT failed for s3 handle 109 at try_count 2 with error 5 (Connection unavailable)
2024-02-20T01:24:16Z 29313030326095062 [10:0] OSC_ERR: logCmd:368 PUT CmdId: 1074 obj-name: 66b569f2-e91b-4a8c-b03e-0bdc9d685b95/34bcd168_000000000000444c_66b569f2-e91b-4a8c-b03e-0bdc9d685b95 failed with OSC error: Operation Timedout
2024-02-20T01:24:16Z 29313030326098162 [10:0] OSC_ERR: logCmd:378 Stats of PUT CmdId: 1074, wait 1us, start-send 20433us, 1stbyte 99975412us, 1st-lastbyte 0us
2024-02-20T01:24:16Z 29313031059319266 [10:0] OSC_ERR: logCmd:368 PUT CmdId: 951 obj-name: 66b569f2-e91b-4a8c-b03e-0bdc9d685b95/ce81223f_000000000000444d_66b569f2-e91b-4a8c-b03e-0bdc9d685b95 failed with OSC error: Operation Timedout
2024-02-20T01:24:16Z 29313031059321696 [10:0] OSC_ERR: logCmd:378 Stats of PUT CmdId: 951, wait 0us, start-send 20271us, 1stbyte 99975585us, 1st-lastbyte 0us
- The volume move uses the same replication engine in the backend, thus might encounter a similar error:
::> volume move show
Error Message:
--------------------------------
Detailed Status: Internal error
Error: Failure to perform I/O with the capacity tier, which has been declared unresponsive