Skip to main content
NetApp Knowledge Base

File in mounted snapshot directory cannot be opened in RMAN

Views:
507
Visibility:
Public
Votes:
1
Category:
ontap-9
Specialty:
nas
Last Updated:

Applies to

  • NFSv3
  • ONTAP 9
  • Oracle Recovery Manager (RMAN)

Issue

  • File cannot be opened in RMAN which leads to DB issue.

Example:

 RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03009: failure of catalog command on default channel at 01/05/2022 08:35:16
 ORA-19625: error identifying file /x/backup.var/y/data/.snapshot/test.2022-01-05_0749/backup/last.controlfile.backup.1
 ORA-17503: ksfdopn:11 Failed to open file /x/backup.var/y/data/.snapshot/test.2022-01-05_0749/backup/last.controlfile.backup.1
 ORA-17500: ODM err:Communication error on send
  • Tracing the communication reveals the NFSv3 error NFS3ERR_STALE at the timestamp.

Example:

  963 2022-01-05 08:35:16.474151  10.10.10.2 → 10.10.10.250 NFS 222 V3 LOOKUP Call, DH: 0xc997dd15/backup
  964 2022-01-05 08:35:16.474250 10.10.10.250 → 10.10.10.2  NFS 102 V3 LOOKUP Reply (Call In 963) Error: NFS3ERR_STALE
  • The NFSv3 exported volume is a SnapMirror destination of a relationship type vault and when checking the snapshot IDs with the following command, there is a numerical gap in the logical-snap-id field between the current (mounted) and previous snapshot and after the SnapMirror update, the id has increased:
::> set adv
::*> snapshot show -volume nfsvol_vault -fields logical-snap-id,create-time

Example:

Last login time: 1/5/2022 08:33:46

vserver          volume                      snapshot                 create-time              logical-snap-id
---------------- --------------------------- ------------------------ ------------------------ ---------------
[...]
SVMname          nfsvol_vault                test.2022-01-04_2058      Tue Jan 04 20:58:50 2022 736
SVMname          nfsvol_vault                othertest.2022-01-05_0000 Wed Jan 05 00:00:01 2022 739
SVMname          nfsvol_vault                test.2022-01-05_0749      Wed Jan 05 07:50:00 2022 741
5 entries were displayed.
 
Last login time: 1/5/2022 08:33:56

vserver          volume                      snapshot                 create-time              logical-snap-id
---------------- --------------------------- ------------------------ ------------------------ ---------------
SVMname          nfsvol_vault                test.2022-01-04_2058      Tue Jan 04 20:58:50 2022 736
SVMname          nfsvol_vault                othertest.2022-01-05_0000 Wed Jan 05 00:00:01 2022 739
SVMname          nfsvol_vault                test.2022-01-05_0749      Wed Jan 05 07:50:00 2022 742
5 entries were displayed.
 
  • To confirm, the snapmirror-audit-log on the destination states no new snapshot copies to transfer and the related snapshot being deleted.

Example:

Wed Jan  5 08:33:45 CET 2022 ManualUpdate[Jan  5 08:33:45]:uuid Operation-Uuid=uuid  Group=none Operation-Cookie=0 action=Start source=SVMsource:nfsvol_source destination=SVMname:nfsvol_vault
Wed Jan  5 08:33:45 CET 2022 ManualUpdate[Jan  5 08:33:45]:uuid Operation-Uuid=uuid  Group=none Operation-Cookie=0 action=Info No new Snapshot copy to transfer from source SVMsource:nfsvol_source.
Wed Jan  5 08:33:45 CET 2022 ManualUpdate[Jan  5 08:33:45]:uuid  Operation-Uuid=uuid  Group=none Operation-Cookie=0 action=End source=SVMsource:nfsvol_source destination=SVMname:nfsvol_vault status=Success bytes_transferred=0 network_compression_ratio=1.0:1
Wed Jan  5 08:33:56 CET 2022 StorageEfficientSnapshotRotation[Jan  5 08:33:56]:uuid Operation-Uuid=uuid Group=none Operation-Cookie=0 action=Start source=SVMsource:nfsvol_source destination=SVMname:nfsvol_vault
Wed Jan  5 08:33:56 CET 2022 StorageEfficientSnapshotRotation[Jan  5 08:33:56]:uuid Operation-Uuid=uuid Group=none Operation-Cookie=0 action=Info sm_rpm_snaplist_delete([1581261]): Snapshot snapmirrorPreSE.test.2022-01-05_0749 deleted.
Wed Jan  5 08:33:56 CET 2022 StorageEfficientSnapshotRotation[Jan  5 08:33:56]:uuid Operation-Uuid=uuid Group=none Operation-Cookie=0 action=End source=SVMsource:nfsvol_source destination=SVMname:nfsvol_vault status=Success

 
 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.