OnCommand Insight - DataWarehouse connector issue ETL is timing out
Applies to
- OnCommand Insight 7.3.8 (OCI)
- OnCommand Insight DataWareHouse 7.3.8 (DWH)
- Both Linux and Windows installation
Issue
- The DWH connector to the OCI server is failing.
- The bellow is shown in the connector's page in DWH.
43606 Inventory: connector 'Connector_name', '18-Jun-2020 00:00' FAILED 18/06/20 05:01 18/06/20 11:28
43582 Inventory: connector 'Connector_name', '17-Jun-2020 00:00' FAILED 17/06/20 06:34 17/06/20 06:45
43570 Inventory: connector 'Connector_name', '16-Jun-2020 17:35' FAILED 16/06/20 17:35 16/06/20 23:49
- Multiple erros can be found in
dwh.log
like those below:2020-06-23 11:19:15,690 ERROR [Thread-694 (ActiveMQ-client-global-threads)] JobDispatcherBean (JobDispatcherBean.java:106) - InventoryJob{name='Inventory: connector 'Connector_Name', '23-Jun-2020 00:00'', id=43727, connectorId=9, targetTime=1592866800057, current=true} failed
com.netapp.sanscreen.dwh.applications.inventory.InventoryException: Failed running inventory job
Caused by: com.netapp.sanscreen.dwh.applications.inventory.InventoryException: Failed to execute SQL script com/netapp/sanscreen/dwh/inventory/processors/performance/performance_summary.load.sql
Caused by: com.netapp.sanscreen.dwh.util.SqlShellException: Failed to execute: INSERT IGNORE INTO dwh_inventory_transient.volume_performance SELECT pt.id AS volumeId, cl.global_id AS storageId, i.sampleTime, i.readResponseTime, i.writeResponseTime, i.totalResponseTime, i.totalResponseTimeMax, i.readThroughput, i.writeThroughput, i.totalThroughput, i.totalThroughputMax, i.readIops, i.writeIops, i.totalIops, i.totalIopsMax, i.readCacheHitRatio, i.writeCacheHitRatio, i.totalCacheHitRatio, i.totalCacheHitRatioMax, i.writePending, i.readIoDensity, i.writeIoDensity, i.totalIoDensity, i.totalIoDensityMax, i.accessed FROM dwh_inventory_staging.tmp_volume_performance i JOIN dwh_inventory_staging.chassis_local cl ON cl.connector_id = 9 AND cl.local_id = i.storageId AND cl.type = 'storage' JOIN dwh_inventory.volume pt ON pt.storageId = cl.global_id AND pt.name = i.name
Caused by: java.sql.SQLException: IJ031012: Unable to obtain lock in 60 seconds: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7be691252020-06-12 11:09:19,253 ERROR [Thread-186 (ActiveMQ-client-global-threads)] SqlShell (SqlShell.java:943) - <OCI_host> Failed to close JDBC connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.2020-06-18 11:28:08,505 ERROR [Thread-450 (ActiveMQ-client-global-threads)] JobDispatcherBean (JobDispatcherBean.java:106) - InventoryJob{name='Inventory: connector 'Connector_Name', '18-Jun-2020 00:00'', id=43606, connectorId=9, targetTime=1592434800062, current=true} failed
com.netapp.sanscreen.dwh.applications.inventory.InventoryException: Failed running inventory job