MySQL database dump dump file was not created
Applies to
- AIQUM 9.13, OVA
Issue
Daily database backup 'AIQ_UM_9.13P1_backup_unix_05-26-2025-20-00.7z' creation failed. Reason: MySQL database dump dump file was not created
Logs
ocumserver.0 log
2025-05-26 20:00:00,064 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Regular Backup Job triggered.
2025-05-26 20:00:00,065 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Regular Backup Job got permit and started.
2025-05-26 20:00:00,065 INFO [oncommand] [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] [com.netapp.ipc.jms.OCIE_Events] OCIE JMS notification message received: {UpdateTime=1748304000000, MessageType=PERIODIC_PURGE}
2025-05-26 20:00:00,084 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Verified that Mysql dumps repo path is valid and has proper permissions: /data/ocum-backup/database-dumps-repo
2025-05-26 20:00:00,092 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.dfm.impl.backup.BackupUtils] Created nested directory with effective path: /data/ocum-backup/AIQ_UM_9.13P1_backup_unix
2025-05-26 20:00:00,261 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Last backedUp binary log file by Backup: unified-manager.007538 and oldest binary log file: unified-manager.007539
2025-05-26 20:00:00,261 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Regular Backup Job: Incremental dump couldn't be taken, proceeding for full dump.
2025-05-26 20:00:00,261 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Starting to take mysql dump
2025-05-26 20:00:06,879 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Full Backup DB Dump Size: 1850179584 bytes
2025-05-26 20:00:06,879 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Space needed for Backup: 2220215501 and Free space available in backup directory: 26591322112
2025-05-26 20:00:06,879 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.dfm.impl.backup.BackupUtils] Created nested directory with effective path: /data/ocum-backup/AIQ_UM_9.13P1_backup_unix/database_dumps
2025-05-26 20:00:06,880 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Created mysql dump Reference file: /data/ocum-backup/AIQ_UM_9.13P1_backup_unix/database_dumps/ocum_mysql_full_backup_1748304006879.sql
2025-05-26 20:00:06,880 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Created mysql dump file /data/ocum-backup/database-dumps-repo/ocum_mysql_full_backup_1748304006879.sql
2025-05-26 20:00:06,880 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Created mysql dump Reference file /data/ocum-backup/AIQ_UM_9.13P1_backup_unix/database_dumps/ocum_mysql_full_backup_1748304006879.sql
2025-05-26 20:00:06,880 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] Starting mysql full dump process
2025-05-26 20:00:06,895 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] Locking the instance for backup
2025-05-26 20:00:06,897 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] Instance is locked
2025-05-26 20:00:06,897 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] Running command /usr/bin/mysqldump --login-path=jboss --ignore-table=ocum.managementstation --flush-logs --add-drop-table --routines --triggers --databases --single-transaction mysql management sanscreen acquisition ocum ocum_view ocum_report ocum_report_birt ocum_data_collector netapp_model_view netapp_model netapp_performance vmware_model_view vmware_model vmware_performance scalemonitor opm -r /data/ocum-backup/database-dumps-repo/ocum_mysql_full_backup_1748304006879.sql by user jboss
2025-05-26 20:00:49,426 ERROR [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] Error occured while taking mysql dump. Error message:
2025-05-26 20:00:49,429 ERROR [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `sample_vserver` at row: 32791
2025-05-26 20:00:49,432 INFO [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] Unlocking the instance for backup
2025-05-26 20:00:49,461 ERROR [oncommand] [task-scheduler-14] [Backup|ScheduleBackup] [c.n.d.i.backup.MysqlDumpOperation] Exception has occured while unlocking the backup instance.
java.sql.SQLNonTransientConnectionException: (conn=23975) unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
journalctl.out.txt
2025-05-25T20:00:06-0400 vndtlp472 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/ocie.service,task=java,pid=1230,uid=999
2025-05-25T20:00:06-0400 vndtlp472 kernel: Out of memory: Killed process 1230 (java) total-vm:7319184kB, anon-rss:3127624kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:7584kB oom_score_adj:0
2025-05-25T20:00:06-0400 vndtlp472 systemd[1]: ocie.service: A process of this unit has been killed by the OOM killer.
2025-05-25T20:00:06-0400 vndtlp472 systemd[1]: ocie.service: Main process exited, code=killed, status=9/KILL
2025-05-25T20:00:07-0400 vndtlp472 ocie[1470731]: Stopping Active IQ Management Server service ocie
2025-05-25T20:00:07-0400 vndtlp472 systemd[1]: ocie.service: Failed with result 'oom-kill'.
2025-05-26T20:00:48-0400 vndtlp472 kernel: ib_srv_wkr-1 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2025-05-26T20:00:49-0400 vndtlp472 kernel: CPU: 1 PID: 914137 Comm: ib_srv_wkr-1 Not tainted 5.10.0-22-amd64 #1 Debian 5.10.178-3
2025-05-26T20:00:49-0400 vndtlp472 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mysql.service,task=mysqld,pid=914081,uid=108
2025-05-26T20:00:49-0400 vndtlp472 kernel: Out of memory: Killed process 914081 (mysqld) total-vm:5070896kB, anon-rss:3431424kB, file-rss:0kB, shmem-rss:0kB, UID:108 pgtables:8096kB oom_score_adj:0
2025-05-26T20:00:48-0400 vndtlp472 systemd[1]: mysql.service: A process of this unit has been killed by the OOM killer.
2025-05-26T20:00:49-0400 vndtlp472 systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
2025-05-26T20:00:49-0400 vndtlp472 systemd[1]: mysql.service: Failed with result 'oom-kill'.
2025-05-26T20:00:49-0400 vndtlp472 systemd[1]: mysql.service: Consumed 3h 59min 51.780s CPU time.
2025-05-26T20:00:49-0400 vndtlp472 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 2.
2025-05-26T20:00:49-0400 vndtlp472 systemd[1]: Stopped MySQL Community Server.
2025-05-26T20:00:49-0400 vndtlp472 systemd[1]: mysql.service: Consumed 3h 59min 51.780s CPU time.
2025-05-26T20:00:49-0400 vndtlp472 systemd[1]: Starting MySQL Community Server...
2025-05-26T20:00:57-0400 vndtlp472 systemd[1]: Started MySQL Community Server.
Mysql ran out of memory and was killed and restarted.