Storage Replication Adapter: SyncOnce operation fails due to "read timed out" API call

Applies to

• ONTAP Storage Replication Adapter

Issue

Storage Repliation Adapter (SRA) uses the SyncOnce  operation to perform a snapmirror update of protected volumes.  The SyncOnce operation is triggered in the following SRM workflows:

• Test Failover
• Planned Migration

In some environments that have a large number of snapmirror relationships the following issue can be seen:

• The SyncOnce operation fails with a "Unknown" error.  This error has an error code of 4001
• When looking at the operation in the vvolvp.log:
  • the SnapMirror update may actually be successful:
    • 2023-07-06T00:39:30,352Z (DEBUG)[pool-3-thread-1] [] updateSnapMirror:
2023-07-06T00:39:30,352Z (DEBUG)[pool-3-thread-1] [] Sending-1861: <snapmirror-update><destination-location>volume</destination-location><source-location>volume</source-location></snapmirror-update>
2023-07-06T00:39:30,739Z (DEBUG)[queue-manager] [task-manager-queue-manager] Task 10074301 is in state Running
2023-07-06T00:39:31,135Z (DEBUG)[pool-3-thread-1] [] Received-1861: <results status='passed'><result-operation-id>12345</result-operation-id><result-status>succeeded</result-status></results>
2023-07-06T00:39:31,135Z (DEBUG)[pool-3-thread-1] [] SnapMirror update started for cluster:volume
2023-07-06T00:39:31,135Z (DEBUG)[pool-3-thread-1] [] SnapMirror update returned status succeeded
  • However there is a snapmirror-get-iter API call that follows the SnapMirror update that times out:
    • 2023-07-06T00:39:31,635Z (DEBUG)[pool-3-thread-1] [] Sending-1862:<snapmirror-get-iter><max-records>500</max-records><query><snapmirror-info><relationship-group-type>none</relationship-group-type></snapmirror-info></query></snapmirror-get-iter>
..
2023-07-06T00:39:49,094Z (WARN )[pool-3-thread-1] [] invokeZAPI(0): failed OntapConnectionImpl{ipAddress=host.domain, userName=user, port=443, ssl=true} (IOException): Read timed out null
2023-07-06T00:39:49,094Z (DEBUG)[pool-3-thread-1] [] slow-zapi-1862 redacted.domain: 17654 ms
2023-07-06T00:39:49,095Z (ERROR)[pool-3-thread-1] [] com.netapp.offtap3.ontap.SvmOperationException: Could not get snapmirror-get-iter for com.netapp.offtap3.internal.ontap.VServerConnection@12345
..
2023-07-06T00:39:49,095Z (ERROR)[pool-3-thread-1] [] Caused by: com.netapp.offtap3.ontap.ApiFailedException: Read timed out (errno=5)