SnapCenter job failing to be triggered and long running job terminated
Applies to
SnapCenter Server 4.1.1x and above
Issue
- Scheduled default jobs or backup jobs fail to be triggered for 20 minutes and running jobs may terminate prematurely, apparently remaining Running.
- Job launcher, when launching jobs also uses the SMCore service to start it up through the SnapCenter Web Application, which fails to start the backup between the two IIS events (IIS no longer handles incoming requests at that point).
- SMcore error message:
DEBUG SMCore_28089 PID=[7152] TID=[49] RemoteUrl: https://<SNAPCENTER_SERVER>:8146/JobStatusService.svc/UpdateJobStatus
ERROR SMCore_28089 PID=[7152] TID=[49] WebException in method: Invoke.
System.Net.WebException: The remote server returned an error: (500) Internal Server Error. - the System Event log will show an informational IIS event with ID 5186 :
"A worker process with process id of '<PID>' serving application pool 'SnapCenter' has requested
a recycle because the worker process reached its allowed processing time limit." - after (by default) 30 minutes, the System Event log will show a warning event with ID 5013:
"A process serving application pool 'SnapCenter' exceeded time limits during shut down. The process id was '<PID>'."
- SnapCenter may also give the following Job error, although SMCore completed the Operaiton (Backup, Clone, etc.):
We noticed an IIS application pool recycle event when this job was in progress. The final state of this job is unkown.
Consider restarting the job if required.