Skip to main content
NetApp Knowledge Base

OnCommand Insight ElasticSearch service fails to start with an 'AccessDenied' exception

Views:
1,497
Visibility:
Public
Votes:
1
Category:
oncommand-insight
Specialty:
oci
Last Updated:

Applies to

  • OnCommand Insight 7.3.x
  • Red Hat Enterprise Linux 7.x
  • CentOS 7.x

Issue

  • When starting the services, the Wildfly service is running, but says failed during the startup. The ElasticSearch service is not running.

[root@ocirhel ~]# oci-service.sh start all
starting mysql
starting elasticsearch
starting wildfly........................................................................................................... ............................................................................................................................................................................................................................................................................................................failed

# oci-service.sh
OnCommand (R) Insight 7.3.10

mysql...............running
elasticsearch.......not running
wildfly.............running
acquisition.........not running

  • The *.war files are all .undeployed for all except oci.war

[root@ocirhel ~]# ls -al /opt/netapp/oci/wildfly/standalone/deployments/
total 101996

drwxr-xr-x. 4 ocisys ocisys     4096 Apr 23 14:27 .
dr-xr-xr-x. 4 root   root         85 Jun 16  2020 ..
-rw-r--r--. 1 ocisys ocisys   214546 May 20  2020 acq_server.war
-rw-r--r--  1 ocisys ocisys       14 May 20  2020 acq_server.war.undeployed
drwxr-xr-x. 5 ocisys ocisys       70 Jun 16  2020 client-plugins.war
-rw-r--r--  1 ocisys ocisys       18 Jun 16  2020 client-plugins.war.undeployed
-rw-r--r--. 1 ocisys ocisys  2931228 May 20  2020 client.war
-rw-r--r--  1 ocisys ocisys       10 May 20  2020 client.war.undeployed
-rw-r--r--. 1 ocisys ocisys 12512703 May 20  2020 compose.war
-rw-r--r--  1 ocisys ocisys       11 May 20  2020 compose.war.undeployed
lrwxrwxrwx. 1 root   root         66 Jun 16  2020 datasources.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/datasources.war
-rw-r--r--  1 ocisys ocisys       15 Jun 16  2020 datasources.war.undeployed
lrwxrwxrwx. 1 root   root         63 Jun 16  2020 download.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/download.war
-rw-r--r--  1 ocisys ocisys       12 Jun 16  2020 download.war.undeployed
-rw-r--r--. 1 ocisys ocisys  2638149 May 20  2020 dwh-views.war
-rw-r--r--  1 ocisys ocisys       13 May 20  2020 dwh-views.war.undeployed
-rw-r--r--. 1 ocisys ocisys    34339 May 20  2020 oci.war
-rw-r--r--  1 ocisys ocisys        7 May 20  2020 oci.war.deployed
drwxr-xr-x. 4 ocisys ocisys     4096 Jun 16  2020 onlineHelp.war
-rw-r--r--  1 ocisys ocisys       14 Jun 16  2020 onlineHelp.war.undeployed
-rw-r--r--. 1 ocisys ocisys  2782161 May 20  2020 phonehome.war
-rw-r--r--  1 ocisys ocisys       13 May 20  2020 phonehome.war.undeployed
-rw-r--r--. 1 ocisys ocisys     8888 May 20  2020 README.txt
-rw-r--r--. 1 ocisys ocisys 81772190 May 20  2020 rest.war
-rw-r--r--  1 ocisys ocisys        8 May 20  2020 rest.war.undeployed
-rw-r--r--. 1 ocisys ocisys  1430051 May 20  2020 upgrade.war
-rw-r--r--  1 ocisys ocisys       11 May 20  2020 upgrade.war.undeployed
lrwxrwxrwx. 1 root   root         61 Jun 16  2020 upload.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/upload.war
-rw-r--r--  1 ocisys ocisys       10 Apr 23 14:25 upload.war.undeployed
-rw-r--r--. 1 ocisys ocisys    32500 May 20  2020 usermanager.war
-rw-r--r--  1 ocisys ocisys       15 May 20  2020 usermanager.war.undeployed

  • When the ElasticSearch service starts it fails with an "Access Denied" exception.
  • The ElasticSearch service is not running:

[root@netapp-oci-lin ~]# oci-service.sh
OnCommand (R) Insight 7.3.9

mysql...............running
elasticsearch.......not running
wildfly.............running
acquisition.........running

  • When starting the service, one of the following errors can be seen from the command line interface (CLI) and when checking the journal:

Example 1:[root@ocirhel ~]# systemctl start elasticsearch
 Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

  Sep 11 19:28:27 ocirhel systemd[6445]: Failed at step USER spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such process
  -- Subject: Process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed
  -- Defined-By: systemd
  -- Support:  http://lists.freedesktop.org/mailman.../systemd-devel
  -- The process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed and failed.
  --
  -- The error number returned by this process is 3.

 

Example 2 (when checking journalctl -xe):
 Sep 11 19:54:07 ocirhel polkitd[735]: Unregistered Authentication Agent for unix-process:5469:138302 (system bus name :1.61, object path /org/freedesktop/PolicyKit1/AuthenticationA
 Sep 11 19:54:08 ocirhel elasticsearch[5478]: 2018-09-11 19:54:08,124 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBe
 ...
 Sep 11 19:54:08 ocirhel elasticsearch[5478]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch

 
Example 3 (when checking journalctl -xe):
 Sep 11 19:57:51 ocirhel elasticsearch[5836]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /var/run/elasticsearch
 Sep 11 19:57:51 ocirhel elasticsearch[5836]: Likely root cause: java.nio.file.AccessDeniedException: /var/run/elasticsearch

Example 4:
# systemctl status elasticsearch
...
Apr 21 20:34:54 ocirhel systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 21 20:34:54 ocirhel systemd[1]: Unit elasticsearch.service entered failed state.
Apr 21 20:34:54 ocirhel systemd[1]: elasticsearch.service failed.

#journalctl -xe
...
Apr 23 14:02:14 ocirhel elasticsearch[4106]: Exception in thread "main" java.nio.file.AccessDeniedException: /etc/elasticsearch/scripts_rpmsave
...
Apr 23 14:02:14 ocirhel systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 23 14:02:14 ocirhel systemd[1]: Unit elasticsearch.service entered failed state.

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

Scan to view the article on your device