SCV is down and services won't start
Applies to
SnapCenter Plug-in for VMware vSphere (SCV) 4.6 and aboveIssue
SCV services do not start and the SCV application is not usable.
The following errors appears in the respective logs:
control-plane.log:
2023-06-05 13:28:29.330 INFO scv1.domain.com --- [nio-8080-exec-4] c.n.c.r.s.SCVStatusImpl : getSCVStatus Returning with {"result":{"scvstatus":true,"status":"Failed to VMware plugin status: Connection reset. Please make sure the plugin is up and running. . Failed to get protection services status from configuration service: Connection refused (Connection refused). Please make sure the protection services are up and running"}}
2023-06-05 13:28:29.327 ERROR scv1.domain.com --- [nio-8080-exec-2] c.n.c.r.s.VcCredentialImpl : Exception occure while getting vccredentials
vmcontrol.log:
2023-06-09 18:09:26.257 DEBUG scv1.domain.com --- [ Thread-9] c.n.a.s.c.SnapManagerHostServiceRestImpl : [JobId:0] Completed Getting SFR Session from SnapCenter Server.
2023-06-09 18:10:39.787 WARN scv1.domain.com --- [ Thread-10] c.n.a.m.DatastoreStorageDetailsCache : [JobId:0] java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-06-09 18:10:39.787 DEBUG scv1.domain.com --- [ Thread-10] c.n.a.m.DatastoreStorageDetailsCache : Shutting down executorService
vmcontrol_jetty_server.log:
2023-06-09T05:57:52.0000544+02:00 INFO SCV root 2023-06-09 17:57:52.543 WARN scv1.domain.com --- [ main] o.s.w.c.s.XmlWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'handlerMapping' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8144-scbr_war-_-any-6111840861476968003/webapp/WEB-INF/lib/nvpf-subplugin-1.0.jar!/com/netapp/aegis/spring-config-rest.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-06-09T05:57:52.0000548+02:00 INFO SCV root 2023-06-09 17:57:52.546 ERROR scv1.domain.com --- [ main] o.s.w.s.DispatcherServlet : Context initialization failed
2023-06-09T05:57:52.0000548+02:00 INFO SCV root org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'handlerMapping' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8144-scbr_war-_-any-6111840861476968003/webapp/WEB-INF/lib/nvpf-subplugin-1.0.jar!/com/netapp/aegis/spring-config-rest.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
daemon.log:
Jun 5 13:57:42 scv1 scvservice[2712]: Creating startup jobs
Jun 5 13:58:37 scv1 scvservice[2712]: successjava.lang.OutOfMemoryError: Java heap space
Jun 5 14:01:43 scv1 scvservice[3351]: # java.lang.OutOfMemoryError: GC overhead limit exceeded
additionally,
wfe.log:
2023-06-14 17:57:22.412 DEBUG scv1.domain.com --- [nio-8080-exec-1] c.n.w.r.u.RestClientUtil : Connecting to scv url: https://localhost:8144/scbr/smcoreplugin/getSCVStatus
2023-06-14 17:57:22.419 ERROR scv1.domain.com --- [nio-8080-exec-1] c.n.w.r.SCVConfigurationServiceImpl : Failed to get SCV plugin status: Connection refused (Connection refused)2023-06-14 17:57:22.420 DEBUG scv1.domain.com --- [nio-8080-exec-1] c.n.w.r.SCSConfigurationService : Getting protection services status from configuration service.2023-06-14 17:57:22.444 DEBUG scv1.domain.com --- [nio-8080-exec-1] c.n.w.c.ConfigManager : Login with name: svc_snapcenter2023-06-14 17:57:22.444 DEBUG scv1.domain.com --- [nio-8080-exec-1] c.n.w.c.ConfigManager : Successfully set server configuration2023-06-14 17:57:22.445 DEBUG scv1.domain.com --- [nio-8080-exec-1] c.n.w.r.SCSRbacServiceImpl : Getting token from SnapCenter Server2023-06-14 17:57:22.467 DEBUG scv1.domain.com --- [nio-8080-exec-1] c.n.w.r.u.RestClientUtil : Connecting to server url: http://localhost:8149/api/v1/RBAC/login2023-06-14 17:57:22.469 ERROR scv1.domain.com --- [nio-8080-exec-1] c.n.w.r.SCSRbacServiceImpl : Unable to get token from SnapCenter Server: Connection refused (Connection refused)
2023-06-14 17:57:22.469 DEBUG scv1.domain.com --- [nio-8080-exec-1] c.n.w.c.ConfigManager : failed to get token form server com.netapp.wfe.model.ApiException: Unable to get token from SnapCenter Server: Connection refused (Connection refused)
control-plane.log:
023-06-09 18:44:28.258 ERROR scv1.domain.com --- [nio-8080-exec-8] c.n.c.r.s.VcCredentialImpl : Exception occure while getting vccredentials
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:389)