After upgrading SCV from 4.8 to 5.0 or later, backups fail

Applies to

• SnapCenter Plug-in for VMware vSphere (SCV)
• Different LIFs are being used for the SVM management LIF and the NFS data LIF
• A custom user with a custom role is used for adding storage to SCV.
• The custom user was created when ONTAP version is prior 9.11.1  

Issue

• In the Protection Service job logs, the backup fails with "SDGetStorageResponse: SDGetStorageResponse failed."

[DBG] SDGetStorageResponse : SDGetStorageResponse :
Result : SDGetStorageResult : HostResource : /(datastore)
No storage footprint available. Error : Failed to get the storage system for host resource : Name : /(datastore), Type: SDNASDir, Code : 104
Error Message : Failed to get the storage system for host resource : Name : /(datastore), Type: SDNASDir, Error Code :
[INF] End Initialize
[WRN] Failed to resolve 192.168.1.1 byHostEntry, Exception:System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
   at System.Net.Dns.GetHostEntryOrAddressesCore(IPAddress address, Boolean justAddresses, AddressFamily addressFamily, ValueStopwatch stopwatch)
   at System.Net.Dns.GetHostEntry(String hostNameOrAddress, AddressFamily family)
   at SnapDrive.Nsf.Common.Infrastructure.Storage.NetworkResourceIdCache.ResolveWithDns(String hostName)

• The above process attempted to resolve is the NFS data LIF, not SVM management LIF.
  • Since NFS data LIF is being processed, name resolution is attempted for this address. 
  • This causes the resolution failure.
• At the time of the upgrade, the REST API requests from SCV to ONTAP are resulting in 401 errors.

(SCV) (SVM) 4 (custom user) [date] "GET /api/private/cli/system/node/virtual-machine/instance/show-settings?fields=consumer HTTP/1.1" 401 65 25238 - 0 + - -
(SCV) (SVM) 4 (custom user) [date] "GET /api/cluster?fields=version HTTP/1.1" 401 65 9577 - 1 + - -

• ProtectionService-(date).log shows invalid credential error.

[INF] [Storage] [TID:6] [RID:] [CID:] [JID:-1] End InitializeONTAP
[INF] [Storage] [TID:6] [RID:] [CID:] [JID:-1] End SDSetStorageConnection
[ERR] [Storage] [TID:10] [RID:] [CID:] [JID:-1] Invalid credentials. Please check the username or password    at NetApp.DataProtection.Storage.CommandAPI.ConnectNcController.ConnectHTTP(IController filer, DataProtectionCredential credentials)
   at NetApp.DataProtection.Storage.CommandAPI.ConnectNcController.Connect(String name)
[INF] [Storage] [TID:6] [RID:] [CID:] [JID:-1] End GetRESTClientTimeout
[DBG] [Storage] [TID:6] [RID:] [CID:] [JID:-1] Trying HTTP/HTTPS
[ERR] [Storage] [TID:10] [RID:] [CID:] [JID:-1] Failed to run a command NetApp.DataProtection.Storage.CommandAPI.ConnectNcController, Exception:Invalid credentials. Please check the username or password

 Invalid credentials. Please check the username or password

[DBG] [Storage] [TID:6] [RID:] [CID:] [JID:-1] NaServer port changed to 443
[DBG] [Storage] [TID:10] [RID:] [CID:] [JID:-1] Response from GetFsxDetails(): {
  "error": {
    "code": "6691623",
    "message": "User is not authorized."
  }
}