Skip to main content
NetApp Response to Russia-Ukraine Cyber Threat
In response to the recent rise in cyber threat due to the Russian-Ukraine crisis, NetApp is actively monitoring the global security intelligence and updating our cybersecurity measures. We follow U.S. Federal Government guidance and remain on high alert. Customers are encouraged to monitor the Cybersecurity and Infrastructure Security (CISA) website for new information as it develops and remain on high alert.
NetApp Knowledge Base

SCV - Unable to find snapshot for SnapVault or SnapMirror update

Views:
3,562
Visibility:
Public
Votes:
3
Category:
snapcenter
Specialty:
snapx
Last Updated:

Applies to

  • SnapCenter Plug-in for VMware vSphere (SCV) 4.3 through 4.6
  • NetApp Data Broker (NDB) 1.0.1

Issue

  • SnapMirror or SnapVault operations fails with one the following messages:

The error on source side:
Unable to find Snapshot (RG|Policy_YYYY-MM-dd_HH:mm:ss.0ms) on volume (primary-volume) from storage (primary-SVM-fqdn).
Failed to attach the label to the specified Snapshot copy.(RG|Policy_YYYY-MM-dd_HH:mm:ss.0ms). Invalid response returned by Set-NcSnapshot: 0

The error on destination side:
Error: SnapVault update failed for the relationship [ source-SVM : source-volume ==> destination-SVM : destination-volume]
with error: Snapshot copy 'RG|Policy_YYYY-MM-dd_HH:mm:ss.0ms' not found on the SnapVault destination destination-SVM : destination-volume

  • SCV (and NDB) are unable to complete SnapMirror or SnapVault operations successfully and will not show as having secondary backups for these failures.
  • SCV backup completes with Warning:

Mirror failed: SnapMirror update failed for the relationship [<SOURCE_PATH> ==> <DESTINATION_PATH>] with error: Snapshot copy '<SNAPSHOT>' not found on the SnapMirror destination <DESTINATION_PATH>

Cause

  • Snapshot cannot be found on source side if:
    • Snapshot is deleted by ONTAP autodelete on the volume, in which case, the solution below will not help (adjust Volume settings instead).
    • Storage system's volume database is not updated fast enough after taking the snapshot, seen often in MCC environment or with LS Mirror on SVM root.
  • Snapshot cannot be found on destination side if:
    • SnapVault or SnapMirror update transfer failed, in which case, the solution below will not help (check ONTAP's SnapMirror logs on the destination)
    • SnapVault or SnapMirror update takes longer than (by default) two hours to complete, and the snapshot will appear later:
      After triggering the SnapVault or SnapMirror update, the plugin will check and wait repeatedly for the transfer to end and the snapshot to appear on the destination. The check is attempted 120 times with pauses of 1 minute before giving up and displaying the error message.

Solution

WARNING

Starting SCV 4.6 the below parameters should be added via Rest API as described in How to use RestAPI to edit configuration parameters in SCV

  1. From vSphere Web Interface, open the Console
  2. Login with maint user
  3. From Main Menu, press  4
  4. From Support and Diagnostic Menu, press 2
  5. Press 'y'  to enter diagnostic shell
  6. Type 'sudo su'
  7. Edit the appsettings-storage.json file by typing  vi /opt/netapp/protectionservice/publish/storage/appsettings-storage.json
  8. Add following keys inside the AppSettings section (timeouts are in milliseconds)

"SNAPSHOT_CHECK_RETRY": "120",
"SNAPSHOT_CHECK_TIMEOUT": "60000",
"SnapshotDiscoverRetryIntervalSecond": "10000",
"SnapshotDiscoverMaxRetryExist": "9",

"SnapmirrorRetry": "360",
"SnapmirrorTimeout": "300000",
"MaxRetrySnapmirrorHistory": "300",
"RetryIntervalSecondSnapmirrorHistory": "4",

  1. Shown here are the defaults that are not normally visible, so you need to adjust the values from the shown defaults to match the duration of longest normal SnapVault or SnapMirror transfer. Instead of increasing just the retries, change the timeout to 10 or 15 minutes (600000, 900000 resp.) and adjust the retries accordingly. This saves on the number of checks and logging (CPU usage).
  2. Save the changes and exit the vi editor (press Escape from editing and enter : , w , and q on the keyboard)
  3. Exit the diagnostic shell by typing 'Exit' at the command prompt
  4. Type 'b' to go back to the Main Menu
  5. From Main Menu, press 1
  6. From the Application Configuration Menu, press 3 to stop the Services
  7. From the Application Configuration Menu, press 2 to start the Services

 

Note:

  • Take a backup of the file:appsettings-storage.json before you make any changes.
  • SNAPSHOT_CHECK_RETRY defines the maximum number of retries that are performed to discover the Snapshot copies on the secondary location.
  • SNAPSHOT_CHECK_TIMEOUT defines the wait or sleep period for every retry.
  • The above default is to wait for one minute before checking again, the total is two hours by default.

Additional Information

Parent topic: Resolution guide for SnapMirror/SnapVault issues in SnapCenter

Note: The SNAPSHOT_CHECK_* variables are meant for the destination Snapshot appearance and the SnapshotDiscovery* for primary Snapshots.

When working with NDB 1.0.1(D1), the capitalized SNAPSHOT_CHECK_* variable need to be outside the AppSettings section !

 

Post upgrade to SCV 4.6, in order to use the entries add the following in json file:
"UseAppSettings": “true”

 

Scan to view the article on your device
CUSTOMER EXCLUSIVE CONTENT

Registered NetApp customers get unlimited access to our dynamic Knowledge Base.

New authoritative content is published and updated each day by our team of experts.

Current Customer or Partner?

Sign In for unlimited access

New to NetApp?

Learn more about our award-winning Support