Scheduled Exchange and SQL back ups not starting after upgrading to SnapCenter 6.0
Applies to
- SnapCenter Server (SC) 6.0 and 6.0.1
- SnapCenter Plug-in for Microsoft Exchange Server (SCE) 6.0 and 6.0.1
- SnapCenter Plug-in for Microsoft SQL Server (SCSQL) 6.0 and 6.0.1
- Scheduled backups
Issue
- After upgrading SnapCenter to 6.0, Exchange and SQL back ups scheduled by SnapCenter no longer start / run.
- Any back ups started manually will run and complete successfully.
- SQL back ups configured to use the SQL scheduler also run and complete successfully.
- May also see the following alert in SC Alerts:
RabbitMQ Connection Failure
SnapCenter Server is unable to establish RabbitMQ connection
- The following errors can be seen in the below logs on the SnapCenter server:
C:\Program Files\NetApp\SnapCenter WebApp\App_Data\log\SnapManagerWeb.log:ERROR SnapManagerWeb PID=[xxxx] TID=[xxxx] RabbitMQ broker is unreachable. Retrying in 30 seconds...C:\Users\<Username>\AppData\Roaming\RabbitMQ\log\rabbit@<SC_Server_Name>.log:[error] <0.798.0> Error on AMQP connection <0.798.0> ([::1]:50342 -> [::1]:5672, state: starting):[error] <0.798.0> PLAIN login refused: user 'jobmanager' - invalid credentialsC:\Program Files\NetApp\Scheduler\logs\Scheduler<Date>.log:[ ERR] System.UnauthorizedAccessException: Token not avaliable at NetApp.DataProtection.Common.Utils.Authorization.AuthorizationMiddleware.Invoke(HttpContext httpContext) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\Authorization\AuthorizationMiddleware.cs:line 62 at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at NetApp.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\ExceptionHandler\ExceptionHandlerMiddleware.cs:line 32[ ERR] at NetApp.DataProtection.Common.Utils.Authorization.AuthorizationMiddleware.Invoke(HttpContext httpContext) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\Authorization\AuthorizationMiddleware.cs:line 62 at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at NetApp.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context) in R:\rtprlse19\snapcenter\orion\nightly\R6.0N_240716_0044\blockbuster\Scheduler\src\Common.Utils\ExceptionHandler\ExceptionHandlerMiddleware.cs:line 32[ ERR] Job <JobID>= threw an unhandled Exception: RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune() at RabbitMQ.Client.Framing.Impl.Connection.Open(Boolean insist) at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, String clientProvidedName) at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, ArrayPool`1 memoryPool, String clientProvidedName) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IFrameHandler fh) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)- May also see the following additional error in the
C:\Users\<Username>\AppData\Roaming\RabbitMQ\log\rabbit@<SC_Server_Name>.log:
[error] <0.12866.35> Channel error on connection <0.12855.35> ([::1]:62776 -> [::1]:5672, vhost: 'jobschedulerhost', user: 'jobmanager'), channel 1:
[error] <0.12866.35> operation exchange.declare caused a channel exception access_refused: configure access to exchange 'ScheduledJobTopic' in vhost 'jobschedulerhost' refused for user 'jobmanager'