SCV does not rotate backup Snapshots with retention due to insufficient free memory

Applies to

• SnapCenter Server 4.1.1
• SnapCenter Plug-in for VMware vSphere (SCV) 4.1.1
   

Issue

• SCV backup Snapshots are not deleted by retention.
• SMCore_<JOB_ID>.log shows RegisterBackups gets Internal Server Error (500) in the task Registering Backup and Applying Retention.

INFO  SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] Activity - Registering Backup and Applying Retention,  Starting
DEBUG SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] RegisterMetadataActivity Started
..
DEBUG SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] RemoteUrl: https://<SNAPCENTER_SERVER_URI>:8146/BackupMetadataService.svc/RegisterBackups
INFO  SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] SmWebClient : webRequest.ServicePoint.ConnectionLimit set to 1500
ERROR SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] WebException in method: Invoke.
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request)
   at System.Net.WebClient.UploadString(Uri address, String method, String data)
   at SnapMgrHelper.RestAdapter.Invoke[TReq,TRes](TReq request, String methodType, String remoteUrl, Boolean dataContractSerializer, Boolean isAuthorized, Boolean logRequest, Boolean logResponse)
Additional information:

DEBUG SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] Invoked service successfully
ERROR SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] Exception in method: Execute.
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request)
   at System.Net.WebClient.UploadString(Uri address, String method, String data)
   at SnapMgrHelper.RestAdapter.Invoke[TReq,TRes](TReq request, String methodType, String remoteUrl, Boolean dataContractSerializer, Boolean isAuthorized, Boolean logRequest, Boolean logResponse)
   at SnapMgrHelper.SmsRestHelper.Invoke[TReq,TRes](String methodType, TReq request, String serviceName, String method, Boolean xmlContentType)
   at SnapMgrCoreService.Workflows.Common.Backup.RegisterMetadataActivity.Execute(SmWorkflowContext workflowContext, SmRequest request) : Detailed Exception The remote server returned an error: (500) Internal Server Error. : BaseException System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request)
   at System.Net.WebClient.UploadString(Uri address, String method, String data)
   at SnapMgrHelper.RestAdapter.Invoke[TReq,TRes](TReq request, String methodType, String remoteUrl, Boolean dataContractSerializer, Boolean isAuthorized, Boolean logRequest, Boolean logResponse)
   at SnapMgrHelper.SmsRestHelper.Invoke[TReq,TRes](String methodType, TReq request, String serviceName, String method, Boolean xmlContentType)
   at SnapMgrCoreService.Workflows.Common.Backup.RegisterMetadataActivity.Execute(SmWorkflowContext workflowContext, SmRequest request)
ERROR SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] ErrorCode (-1), ErrorMessage (The remote server returned an error: (500) Internal Server Error.)
DEBUG SMCore_<JOB_ID> PID=[<PID>] TID=[<TID>] RegisterMetadataActivity Done
 

• SnapManagerWeb_<JOB_ID>.log shows the task Registering Backup and Applying Retention is failed.

DEBUG SnapManagerWeb_<JOB_ID> PID=[<PID>] TID=[<TID>] Updating the job (<JOB_ID>) task (Registering Backup and Applying Retention) status as Failed
 

• Application event log indicates the following error message. The service '/BackupMetadataService.svc' cannot be activated due to System.InsufficientMemoryException.

WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/10501549
 Exception: System.ServiceModel.ServiceActivationException: The service '/BackupMetadataService.svc' cannot be activated due to an exception during compilation. 
 The exception message is: Memory gates checking failed because the free memory (XXXXXXXXX bytes) is less than 5% of total memory. As a result, the service will not be available for incoming requests. 
 To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element..
 ---> System.InsufficientMemoryException: Memory gates checking failed because the free memory (XXXXXXXXX bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests. 
 To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.
   at System.ServiceModel.Activation.ServiceMemoryGates.Check(Int32 minFreeMemoryPercentage, Boolean throwOnLowMemory, UInt64& availableMemoryBytes)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CheckMemoryCloseIdleServices(EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   --- End of inner exception stack trace ---
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath, EventTraceActivity eventTraceActivity)
 Process Name: w3wp
 Process ID: <PID>