Unable to create Mysql dump in Active IQ Unified Manager

Applies to

• Active IQ Unified Manager (all platforms)
• Support Bundle creation with database dump option selected
• Active IQ Unified Manager backups.

Issue

One or more of the followings are observed during the mysql dump creation.

• No database backups exists when triggered from a schedule within the Management / Database Backup view.
• journalctl reports that the the mysql process was killed due to an out of memory(oom) condition. 
  • 2025-05-25T20:00:06-0400 vndtlp472 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/ocie.service,task=java,pid=1230,uid=999
2025-05-25T20:00:06-0400 vndtlp472 kernel: Out of memory: Killed process 1230 (java) total-vm:7319184kB, anon-rss:3127624kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:7584kB oom_score_adj:0
2025-05-25T20:00:06-0400 vndtlp472 systemd[1]: ocie.service: A process of this unit has been killed by the OOM killer.
2025-05-25T20:00:06-0400 vndtlp472 systemd[1]: ocie.service: Main process exited, code=killed, status=9/KILL
2025-05-25T20:00:07-0400 vndtlp472 ocie[1470731]: Stopping Active IQ Management Server service ocie
2025-05-25T20:00:07-0400 vndtlp472 systemd[1]: ocie.service: Failed with result 'oom-kill'.
• ocumserver.log reports a connection failure to MySQL while executing a flush query.

ERROR [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Error occured while taking mysql dump. Error message:
ERROR [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] mysqldump: Couldn't execute 'FLUSH /*!40101 LOCAL */ TABLES': Lost connection to MySQL server during query (2013)
INFO  [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Deleted BackupFileInfo entity in DB
ERROR [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Error occurred in Regular Backup Job:
com.netapp.dfm.impl.backup.BackupCreationException: Failed to create database backup using mysql utility.
      at deployment.dfm-app.war//com.netapp.dfm.impl.backup.MySqlDumpBackupManager.collectMysqlDump(MySqlDumpBackupManager.java:722)
..

• ocumserver.log reports memory exhaustion with different threads around the same time as the MySQL query failure shown above.

INFO [oncommand] [Thread-248] [c.n.i.s.OpmThreadPoolExecutorMonitor$ResourceMonitor] The total amount of memory used is greater than 80%

• ocumserver.log reports memory allocation failure.

ERROR [oncommand] [taskScheduler-3] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Error while creating the backup:
com.netapp.dfm.impl.backup.BackupCreationException: Error occurred while creating backup archive <BACKUP_FILE_PATH>.
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupArchiver.createArchive(BackupArchiver.java:130)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupArchiver.createArchive(BackupArchiver.java:95)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.compressDatabaseDumpFileAndDeleteOrigional(BackupRestoreManager.java:492)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.collectMysqlDump(BackupRestoreManager.java:591)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.createBackupInTransaction(BackupRestoreManager.java:237)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.createBackup(BackupRestoreManager.java:195)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager$$FastClassBySpringCGLIB$$7bcdd65f.invoke(<generated>)
    at deployment.dfm-app.war//org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:651)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager$$EnhancerBySpringCGLIB$$e82dbfff.createBackup(<generated>)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupScheduleJob.run(BackupScheduleJob.java:38)
    at deployment.dfm-app.war//com.netapp.dfm.core.scheduling.DelegatingJob.execute(DelegatingJob.java:53)
    at deployment.dfm-app.war//org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at deployment.dfm-app.war//org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at deployment.dfm-app.war//com.netapp.dfm.common.logging.MDCClearingThreadFactory$1.run(MDCClearingThreadFactory.java:28)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.Exception: ERROR: Can't allocate required memory!
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.runWindowsCommand(BackupRestoreManager.java:1128)
    at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupArchiver.createArchive(BackupArchiver.java:126)
    ... 20 common frames omitted
 

Note: See Unified Manager : What are the notable log files and their respective locations ? to obtain location of ocumserver.log.