Data collection fails in AIQUM for all clusters due to space issue
Applies to
- ActiveIQ Unified Manager (AIQUM) 9.6+
- OnCommand Unified Manager (OCUM) 9.5 and below
- All OS versions
Issue
AIQUM stopped data collection in Virtual Appliance for all clusters under monitoring. The symptoms are as follows,
- MYSQL service and OCIE services are in running state.
- OCIEAU service is in inactive state
- Deployments are in deployed state and GUI is accessible
- When started manually, OCIEAU service is going to stopped state after couple of minutes.
- Au.log:
2020-11-12 14:47:28,793 ERROR [main] c.o.s.a.f.Main (Main.java:162) - Main - javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not execute statement java.sql.SQLException: Connection is closed
- mysql error.log:
2020-11-12T07:47:19.273513Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.2020-11-12T07:47:24.047703Z 1 [Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed.
2020-11-12T07:47:24.048002Z 1 [ERROR] [MY-012639] [InnoDB] Write to file ./netapp_performance/sample_networklifvserver#p#p18.ibd failed at offset 98304, 16384 bytes should have been written, only 0 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.2020-11-12T07:47:24.048180Z 1 [ERROR] [MY-012640] [InnoDB] Error number 28 means 'No space left on device'