Cannot generate database backups in ActiveIQ Unified Manager 9.7 and 9.8
Applies to
ActiveIQ Unified Manager 9.7 and 9.8 (all platforms)
Issue
One or more of the followings are observed during the database backup failure.
- No database backups exists when triggered from a schedule within the Management / Database Backup view.
ocumserver.log
reports a connection failure to MySQL while executing a flush query.
ERROR [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Error occured while taking mysql dump. Error message:
ERROR [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] mysqldump: Couldn't execute 'FLUSH /*!40101 LOCAL */ TABLES': Lost connection to MySQL server during query (2013)
INFO [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.i.b.MySqlDumpBackupManager] Deleted BackupFileInfo entity in DB
ERROR [oncommand] [taskScheduler-1] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Error occurred in Regular Backup Job:
com.netapp.dfm.impl.backup.BackupCreationException: Failed to create database backup using mysql utility.
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.MySqlDumpBackupManager.collectMysqlDump(MySqlDumpBackupManager.java:722)
..
ocumserver.log
reports memory exhaustion with different threads around the same time as the MySQL query failure shown above.
INFO [oncommand] [Thread-248] [c.n.i.s.OpmThreadPoolExecutorMonitor$ResourceMonitor] The total amount of memory used is greater than 80%
ocumserver.log
reports memory allocation failure.
ERROR [oncommand] [taskScheduler-3] [Backup|ScheduleBackup] [c.n.d.impl.backup.BackupScheduleJob] Error while creating the backup:
com.netapp.dfm.impl.backup.BackupCreationException: Error occurred while creating backup archive <BACKUP_FILE_PATH>.
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupArchiver.createArchive(BackupArchiver.java:130)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupArchiver.createArchive(BackupArchiver.java:95)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.compressDatabaseDumpFileAndDeleteOrigional(BackupRestoreManager.java:492)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.collectMysqlDump(BackupRestoreManager.java:591)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.createBackupInTransaction(BackupRestoreManager.java:237)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.createBackup(BackupRestoreManager.java:195)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager$$FastClassBySpringCGLIB$$7bcdd65f.invoke(<generated>)
at deployment.dfm-app.war//org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:651)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager$$EnhancerBySpringCGLIB$$e82dbfff.createBackup(<generated>)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupScheduleJob.run(BackupScheduleJob.java:38)
at deployment.dfm-app.war//com.netapp.dfm.core.scheduling.DelegatingJob.execute(DelegatingJob.java:53)
at deployment.dfm-app.war//org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at deployment.dfm-app.war//org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at deployment.dfm-app.war//com.netapp.dfm.common.logging.MDCClearingThreadFactory$1.run(MDCClearingThreadFactory.java:28)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.Exception: ERROR: Can't allocate required memory!
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupRestoreManager.runWindowsCommand(BackupRestoreManager.java:1128)
at deployment.dfm-app.war//com.netapp.dfm.impl.backup.BackupArchiver.createArchive(BackupArchiver.java:126)
... 20 common frames omitted
Note: See Unified Manager : What are the notable log files and their respective locations ? to obtain location of ocumserver.log.