OnCommand Insight ElasticSearch service fails to start with an 'AccessDenied' exception
Applies to
- OnCommand Insight 7.3.x
- Red Hat Enterprise Linux 7.x
- CentOS 7.x
Issue
- When starting the services, the Wildfly service is running, but says failed during the startup. The ElasticSearch service is not running.
[root@ocirhel ~]# oci-service.sh start all
starting mysql
starting elasticsearch
starting wildfly........................................................................................................... ............................................................................................................................................................................................................................................................................................................failed
# oci-service.sh
OnCommand (R) Insight 7.3.10
mysql...............running
elasticsearch.......not running
wildfly.............running
acquisition.........not running
- The *.war files are all .undeployed for all except oci.war
[root@ocirhel ~]# ls -al /opt/netapp/oci/wildfly/standalone/deployments/
total 101996
drwxr-xr-x. 4 ocisys ocisys 4096 Apr 23 14:27 .
dr-xr-xr-x. 4 root root 85 Jun 16 2020 ..
-rw-r--r--. 1 ocisys ocisys 214546 May 20 2020 acq_server.war
-rw-r--r-- 1 ocisys ocisys 14 May 20 2020 acq_server.war.undeployed
drwxr-xr-x. 5 ocisys ocisys 70 Jun 16 2020 client-plugins.war
-rw-r--r-- 1 ocisys ocisys 18 Jun 16 2020 client-plugins.war.undeployed
-rw-r--r--. 1 ocisys ocisys 2931228 May 20 2020 client.war
-rw-r--r-- 1 ocisys ocisys 10 May 20 2020 client.war.undeployed
-rw-r--r--. 1 ocisys ocisys 12512703 May 20 2020 compose.war
-rw-r--r-- 1 ocisys ocisys 11 May 20 2020 compose.war.undeployed
lrwxrwxrwx. 1 root root 66 Jun 16 2020 datasources.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/datasources.war
-rw-r--r-- 1 ocisys ocisys 15 Jun 16 2020 datasources.war.undeployed
lrwxrwxrwx. 1 root root 63 Jun 16 2020 download.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/download.war
-rw-r--r-- 1 ocisys ocisys 12 Jun 16 2020 download.war.undeployed
-rw-r--r--. 1 ocisys ocisys 2638149 May 20 2020 dwh-views.war
-rw-r--r-- 1 ocisys ocisys 13 May 20 2020 dwh-views.war.undeployed
-rw-r--r--. 1 ocisys ocisys 34339 May 20 2020 oci.war
-rw-r--r-- 1 ocisys ocisys 7 May 20 2020 oci.war.deployed
drwxr-xr-x. 4 ocisys ocisys 4096 Jun 16 2020 onlineHelp.war
-rw-r--r-- 1 ocisys ocisys 14 Jun 16 2020 onlineHelp.war.undeployed
-rw-r--r--. 1 ocisys ocisys 2782161 May 20 2020 phonehome.war
-rw-r--r-- 1 ocisys ocisys 13 May 20 2020 phonehome.war.undeployed
-rw-r--r--. 1 ocisys ocisys 8888 May 20 2020 README.txt
-rw-r--r--. 1 ocisys ocisys 81772190 May 20 2020 rest.war
-rw-r--r-- 1 ocisys ocisys 8 May 20 2020 rest.war.undeployed
-rw-r--r--. 1 ocisys ocisys 1430051 May 20 2020 upgrade.war
-rw-r--r-- 1 ocisys ocisys 11 May 20 2020 upgrade.war.undeployed
lrwxrwxrwx. 1 root root 61 Jun 16 2020 upload.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/upload.war
-rw-r--r-- 1 ocisys ocisys 10 Apr 23 14:25 upload.war.undeployed
-rw-r--r--. 1 ocisys ocisys 32500 May 20 2020 usermanager.war
-rw-r--r-- 1 ocisys ocisys 15 May 20 2020 usermanager.war.undeployed
- When the ElasticSearch service starts it fails with an "Access Denied" exception.
- The ElasticSearch service is not running:
[root@netapp-oci-lin ~]# oci-service.sh
OnCommand (R) Insight 7.3.9
mysql...............running
elasticsearch.......not running
wildfly.............running
acquisition.........running
- When starting the service, one of the following errors can be seen from the command line interface (CLI) and when checking the journal:
Example 1:[root@ocirhel ~]# systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
Sep 11 19:28:27 ocirhel systemd[6445]: Failed at step USER spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such process
-- Subject: Process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman.../systemd-devel
-- The process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed and failed.
--
-- The error number returned by this process is 3.
Example 2 (when checking journalctl -xe):
Sep 11 19:54:07 ocirhel polkitd[735]: Unregistered Authentication Agent for unix-process:5469:138302 (system bus name :1.61, object path /org/freedesktop/PolicyKit1/AuthenticationA
Sep 11 19:54:08 ocirhel elasticsearch[5478]: 2018-09-11 19:54:08,124 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBe
...
Sep 11 19:54:08 ocirhel elasticsearch[5478]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch
Example 3 (when checking journalctl -xe):
Sep 11 19:57:51 ocirhel elasticsearch[5836]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /var/run/elasticsearch
Sep 11 19:57:51 ocirhel elasticsearch[5836]: Likely root cause: java.nio.file.AccessDeniedException: /var/run/elasticsearch
Example 4:
# systemctl status elasticsearch
...
Apr 21 20:34:54 ocirhel systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 21 20:34:54 ocirhel systemd[1]: Unit elasticsearch.service entered failed state.
Apr 21 20:34:54 ocirhel systemd[1]: elasticsearch.service failed.
#journalctl -xe
...
Apr 23 14:02:14 ocirhel elasticsearch[4106]: Exception in thread "main" java.nio.file.AccessDeniedException: /etc/elasticsearch/scripts_rpmsave
...
Apr 23 14:02:14 ocirhel systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 23 14:02:14 ocirhel systemd[1]: Unit elasticsearch.service entered failed state.