Unable to start up OnCommand Insight service with 404 error in web browser
Applies to
- OnCommand Insight (OCI) 7.3.2
- Compatible Windows platform
Issue
One or more of the following symptoms may be seen when all of the OCI services are up:
1. When attempting to access OCI, 404 error appears in web browser.
2. ElasticSearch (ES) just keeps timing out and no node is available for use. To check, when entering https://locahost/cat/health
and https://localhost/cat/indices
(within the web browser) in the OCI server, the web page will either:
- Try to load up, then eventually time out
- Show yellow or red at 0%.
3. ElasticSearch will fail to respond to any request, as similarly seen from the following logs:
- oci-cluster.log (located in
<installed_path>\SANscreen\program_data\elasticsearch\logs
) shows all of the requests timing out with failures to delete indices.
[2020-04-28T07:25:48,321][WARN ][o.e.t.TransportService ] [VWTSOCI001] Received response for a request that has timed out, sent [30485ms] ago, timed out [15469ms] ago, action [cluster:monitor/nodes/stats[n]], node [{VWTSOCI001}{dx4j5WoDQia-22-BA7uHxA}{o4Th7W8_TuOL8lrYj8eGyQ}{127.0.0.1}{127.0.0.1:9310}], id [13438]
[2020-04-23T01:13:30,112][DEBUG][o.e.a.a.i.d.TransportDeleteIndexAction] [VWTSOCI001] failed to delete indices []
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster vent (delete-index) within 30s
- server-indexer.log (located in
<installed_path>\SANscreen\wildfly\standalone\log
) will show failures for the incremental indexer due to version conflicts during updates. This implies that multiple updates are occurring between scheduled scans. There are also errors present that denote failures to update aliases.
2020-04-23 03:04:14,350 ERROR [EE-ManagedThreadFactory-default-Thread-11] IncrementalIndexUpdater (IncrementalIndexUpdater.java:107) - ElasticsearchException while updating application fields for type HV_VirtualMachine
com.netapp.oci.es.ElasticsearchException: Error bulk indexing data. failure in bulk execution:
[0]: index [oci-inventory-hv_virtualmachine-2020-04-07-23-03-00], type [hv_virtualmachine], id [3838685751485770960], message [[oci-inventory-hv_virtualmachine-2020-04-07-23-03-00/7KzZYy9tTaK4uokvMyg8Tg][[oci-inventory-hv_virtualmachine-2020-04-07-23-03-00][1]] VersionConflictEngineException[[hv_virtualmachine][3838685751485770960]: version conflict, current version [10] is different than the one provided [9]]]
2020-04-23 00:32:52,003 ERROR [EJB default - 7] QueryIndexBuilder (QueryIndexBuilder.java:202) - Error indexing Disk
com.netapp.oci.es.ElasticsearchException: Failed to switch alias: oci-inventory-disk
- Event queue is almost near capacity, i.e. more than 1950 out of 2000, as seen in events.log (located in
<installed_path>\SANsceen\wildfly\standalone\log
). A lot of annotation and application update events can also be observed at the same time.
2020-04-23 03:28:42,052 INFO [default task-52] AsyncGuavaEventBus (EventBusLogHelper.java:28) - Posting event: AnnotationAssignmentEvent
2020-04-23 03:28:42,052 INFO [default task-52] AsyncGuavaEventBus (EventBusLogHelper.java:28) - Posting event: EventBusStatusEvent
2020-04-23 03:28:42,052 WARN [default task-52] AsyncGuavaEventBus (AsyncGuavaEventBus.java:126) - Event queue size: 1961 out of 2000