SnapCenter job failing to be triggered and long running job terminated

Applies to

SnapCenter Server 4.1.1x and above

Issue

• Scheduled default jobs, backup or clone jobs fail to be triggered for 20 minutes.
• Scheduled running default jobs, backup or clone may terminate prematurely, apparently remaining Running in the webui.
• Job launcher, when launching jobs also uses the SMCore service to start it up through the SnapCenter Web Application, which fails to start the backup between the two IIS events (IIS no longer handles incoming requests at that point).
• SMcore error message:
  DEBUG SMCore_28089 PID=[7152] TID=[49] RemoteUrl: https://<SNAPCENTER_SERVER>:8146/JobStatusService.svc/UpdateJobStatus
ERROR SMCore_28089 PID=[7152] TID=[49] WebException in method: Invoke.
    System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
• the System Event log will show an informational IIS event with ID 5074 (SnapCenter) or 5186 (DefaultAppPool)
  "A worker process with process id of '<PID>' serving application pool 'SnapCenter' has requested
 a recycle because the worker process reached its allowed processing time limit."
• after (by default) 30 minutes, the System Event log may show a warning event with ID 5013:
  "A process serving application pool 'SnapCenter' exceeded time limits during shut down. The process id was '<PID>'."
• SnapCenter may also give the following Job error, although SMCore completed the Operation (Backup, Clone, etc.):

  We noticed an IIS application pool recycle event when this job was in progress. The final state of this job is unkown.
Consider restarting the job if required.