Scheduled backups not starting after upgrading SnapCenter to 6.0 or later
Applies to
- SnapCenter Server (SC) 6.0 and later
- RabbitMQ Server
Issue
- After upgrading SnapCenter to 6.0 or later, all scheduled backups in SnapCenter no longer start / run.
- Any backups started manually run and complete successfully.
- SQL backups configured to use the SQL scheduler also run and complete successfully.
- The following errors can be seen on the SnapCenter server:
SnapCenter UI > Alerts:
RabbitMQ Connection Failure
SnapCenter Server is unable to establish RabbitMQ connection
C:\Program Files\NetApp\SnapCenter WebApp\App_Data\log\SnapManagerWeb.log:ERROR SnapManagerWeb PID=[xxxx] TID=[xxxx] RabbitMQ broker is unreachable. Retrying in 30 seconds...C:\Users\<Username>\AppData\Roaming\RabbitMQ\log\rabbit@<SC_Server_Name>.log:[error] <0.798.0> Error on AMQP connection <0.798.0> ([::1]:50342 -> [::1]:5672, state: starting):[error] <0.798.0> PLAIN login refused: user 'jobmanager' - invalid credentialsC:\Program Files\NetApp\Scheduler\logs\Scheduler<Date>.log:[ ERR] System.UnauthorizedAccessException: Token not avaliable at NetApp.DataProtection.Common.Utils.Authorization.AuthorizationMiddleware.Invoke(HttpContext httpContext) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\Authorization\AuthorizationMiddleware.cs:line 62 at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at NetApp.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\ExceptionHandler\ExceptionHandlerMiddleware.cs:line 32[ ERR] at NetApp.DataProtection.Common.Utils.Authorization.AuthorizationMiddleware.Invoke(HttpContext httpContext) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\Authorization\AuthorizationMiddleware.cs:line 62 at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at NetApp.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\ExceptionHandler\ExceptionHandlerMiddleware.cs:line 32[ ERR] Job <JobID>= threw an unhandled Exception: RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune() at RabbitMQ.Client.Framing.Impl.Connection.Open(Boolean insist) at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, String clientProvidedName) at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, ArrayPool`1 memoryPool, String clientProvidedName) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IFrameHandler fh) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)- May also see the following additional error in the
C:\Users\<Username>\AppData\Roaming\RabbitMQ\log\rabbit@<SC_Server_Name>.log:
[error] <0.12866.35> Channel error on connection <0.12855.35> ([::1]:62776 -> [::1]:5672, vhost: 'jobschedulerhost', user: 'jobmanager'), channel 1:
[error] <0.12866.35> operation exchange.declare caused a channel exception access_refused: configure access to exchange 'ScheduledJobTopic' in vhost 'jobschedulerhost' refused for user 'jobmanager'