Skip to main content
NetApp Knowledge Base

OnCommand Insight ElasticSearch service fails to start with an 'AccessDenied' exception

Views:
1,664
Visibility:
Public
Votes:
1
Category:
oncommand-insight
Specialty:
oci
Last Updated:

Applies to

  • OnCommand Insight 7.3.x
  • Red Hat Enterprise Linux 7.x
  • CentOS 7.x

Issue

  • When starting the services, the Wildfly service is running, but says failed during the startup. The ElasticSearch service is not running.

[root@ocirhel ~]# oci-service.sh start all
starting mysql
starting elasticsearch
starting wildfly........................................................................................................... ............................................................................................................................................................................................................................................................................................................failed

# oci-service.sh
OnCommand (R) Insight 7.3.10

mysql...............running
elasticsearch.......not running
wildfly.............running
acquisition.........not running

  • The *.war files are all .undeployed for all except oci.war

[root@ocirhel ~]# ls -al /opt/netapp/oci/wildfly/standalone/deployments/
total 101996

drwxr-xr-x. 4 ocisys ocisys     4096 Apr 23 14:27 .
dr-xr-xr-x. 4 root   root         85 Jun 16  2020 ..
-rw-r--r--. 1 ocisys ocisys   214546 May 20  2020 acq_server.war
-rw-r--r--  1 ocisys ocisys       14 May 20  2020 acq_server.war.undeployed
drwxr-xr-x. 5 ocisys ocisys       70 Jun 16  2020 client-plugins.war
-rw-r--r--  1 ocisys ocisys       18 Jun 16  2020 client-plugins.war.undeployed
-rw-r--r--. 1 ocisys ocisys  2931228 May 20  2020 client.war
-rw-r--r--  1 ocisys ocisys       10 May 20  2020 client.war.undeployed
-rw-r--r--. 1 ocisys ocisys 12512703 May 20  2020 compose.war
-rw-r--r--  1 ocisys ocisys       11 May 20  2020 compose.war.undeployed
lrwxrwxrwx. 1 root   root         66 Jun 16  2020 datasources.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/datasources.war
-rw-r--r--  1 ocisys ocisys       15 Jun 16  2020 datasources.war.undeployed
lrwxrwxrwx. 1 root   root         63 Jun 16  2020 download.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/download.war
-rw-r--r--  1 ocisys ocisys       12 Jun 16  2020 download.war.undeployed
-rw-r--r--. 1 ocisys ocisys  2638149 May 20  2020 dwh-views.war
-rw-r--r--  1 ocisys ocisys       13 May 20  2020 dwh-views.war.undeployed
-rw-r--r--. 1 ocisys ocisys    34339 May 20  2020 oci.war
-rw-r--r--  1 ocisys ocisys        7 May 20  2020 oci.war.deployed
drwxr-xr-x. 4 ocisys ocisys     4096 Jun 16  2020 onlineHelp.war
-rw-r--r--  1 ocisys ocisys       14 Jun 16  2020 onlineHelp.war.undeployed
-rw-r--r--. 1 ocisys ocisys  2782161 May 20  2020 phonehome.war
-rw-r--r--  1 ocisys ocisys       13 May 20  2020 phonehome.war.undeployed
-rw-r--r--. 1 ocisys ocisys     8888 May 20  2020 README.txt
-rw-r--r--. 1 ocisys ocisys 81772190 May 20  2020 rest.war
-rw-r--r--  1 ocisys ocisys        8 May 20  2020 rest.war.undeployed
-rw-r--r--. 1 ocisys ocisys  1430051 May 20  2020 upgrade.war
-rw-r--r--  1 ocisys ocisys       11 May 20  2020 upgrade.war.undeployed
lrwxrwxrwx. 1 root   root         61 Jun 16  2020 upload.war -> /var/lib/netapp/oci/wildfly/standalone/deployments/upload.war
-rw-r--r--  1 ocisys ocisys       10 Apr 23 14:25 upload.war.undeployed
-rw-r--r--. 1 ocisys ocisys    32500 May 20  2020 usermanager.war
-rw-r--r--  1 ocisys ocisys       15 May 20  2020 usermanager.war.undeployed

  • When the ElasticSearch service starts it fails with an "Access Denied" exception.
  • The ElasticSearch service is not running:

[root@netapp-oci-lin ~]# oci-service.sh
OnCommand (R) Insight 7.3.9

mysql...............running
elasticsearch.......not running
wildfly.............running
acquisition.........running

  • When starting the service, one of the following errors can be seen from the command line interface (CLI) and when checking the journal:

Example 1:[root@ocirhel ~]# systemctl start elasticsearch
 Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

  Sep 11 19:28:27 ocirhel systemd[6445]: Failed at step USER spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such process
  -- Subject: Process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed
  -- Defined-By: systemd
  -- Support:  http://lists.freedesktop.org/mailman.../systemd-devel
  -- The process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed and failed.
  --
  -- The error number returned by this process is 3.

 

Example 2 (when checking journalctl -xe):
 Sep 11 19:54:07 ocirhel polkitd[735]: Unregistered Authentication Agent for unix-process:5469:138302 (system bus name :1.61, object path /org/freedesktop/PolicyKit1/AuthenticationA
 Sep 11 19:54:08 ocirhel elasticsearch[5478]: 2018-09-11 19:54:08,124 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBe
 ...
 Sep 11 19:54:08 ocirhel elasticsearch[5478]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch

 
Example 3 (when checking journalctl -xe):
 Sep 11 19:57:51 ocirhel elasticsearch[5836]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /var/run/elasticsearch
 Sep 11 19:57:51 ocirhel elasticsearch[5836]: Likely root cause: java.nio.file.AccessDeniedException: /var/run/elasticsearch

Example 4:
# systemctl status elasticsearch
...
Apr 21 20:34:54 ocirhel systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 21 20:34:54 ocirhel systemd[1]: Unit elasticsearch.service entered failed state.
Apr 21 20:34:54 ocirhel systemd[1]: elasticsearch.service failed.

#journalctl -xe
...
Apr 23 14:02:14 ocirhel elasticsearch[4106]: Exception in thread "main" java.nio.file.AccessDeniedException: /etc/elasticsearch/scripts_rpmsave
...
Apr 23 14:02:14 ocirhel systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 23 14:02:14 ocirhel systemd[1]: Unit elasticsearch.service entered failed state.

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.