- OnCommand Unified Manager 7.2.x
- OnCommand Unified Manager 7.3.x
- OnCommand Unified Manager 9.4.x
- OnCommand Unified Manager 9.5
Unified Manager database backup sizes tend to increase owing to performance data tables. Due to this reason, customers with Unified Manager instances managing larger footprints, may face issues while upgrading their instances from versions 7.2.x/7.3.x --> 9.4 (or) from 7.3.x --> 9.5.
This article intends to help customers with generic guidelines on how to plan for Unified Manager upgrades from versions 7.2.x/7.3.x/9.4 to 9.5. The article also helps plan Unified Manager database backup in a more defined format which will help customers with faster restores in the long run.
Current Scenario with Upgrades (versions: 7.2.x/7.3.x/9.4.x)
The number of records in continuous_event_participant and continuous_event_participant_stats tables under “opm” schema, tend to increase if Unified Manager manages larger ONTAP footprints or if the instance is managing ONTAP nodes for a longer period. This causes upgrades to fail from Unified Manager versions 7.2.x/7.3.x --> 9.4 (or) from 7.3.x --> 9.5. This happens due to the following reason:
- The size of the tables: continuous_event_participant and continuous_event_participant_stats
Note: With Active IQ Unified Manager 9.5P1 and 9.6RC1, the timeout value is defaulted to 18000 seconds (5 hours). These issues are currently tracked, and we will have a better solution with the next Active IQ Unified Manager release.
We are proposing a set of prerequisites and best practices, which will help you with your Unified Manager upgrades from 7.2.x/7.3.x/9.4.x/9.5 versions.
Prerequisites for Active IQ Unified Manager Upgrades (from 7.2.x/7.3.x/9.4.x --> 9.5)
It will be always beneficial if you follow these guidelines before any planned Active IQ Unified Manager upgrades from versions 7.2.X, 7.3.X, 9.4.X --> 9.5
- Full backup before upgrades
Take a backup of the system before upgrade.
Note: The system will not allow you to perform manual full backup in the GUI by default. You will need to flush some logs to break the backup sequence chain. This requires you to engage NetApp Support as explained in the section “Unified Manager hosted from physical systems and/or non-VMware infrastructure”
Unified Manager instance on VMware
- If it is a vApp, take a VMware snapshot
Perform the following:
- In the vSphere Client, click Home > Inventory > VMs and Templates.
- Select the virtual machine (VM) on which the Unified Manager virtual appliance is installed.
- If the Unified Manager VM is running, navigate to Summary > Commands > Shut Down Guest.
- Create a backup copy — such as a snapshot or clone — of the Unified Manager VM to create an application-consistent backup.
- From the vSphere Client, power ON the Unified Manager VM
- If it is a RHEL/CentOS/Windows VM hosted from VMware, stop Unified Manager and MySQL services, and take a snapshot
Unified Manager hosted from physical systems and/or non-VMware infrastructure
If the instance is not hosted from VMware, you need to force a full backup. Engage NetApp Support and perform the following steps: (for more details, see KB: How to start a new Incremental Backup chain within ActiveIQ Unified Manager versions 7.2 through 9.6)
- At the prompt, set a database by typing the following:
- Purge the Logs from the MySQL backup utility by running each of the following commands that end with a semicolon; individually, in sequence:
PURGE BINARY LOGS BEFORE CURRENT_TIMESTAMP;
- Then key in:
- Trigger backup from the GUI now.
Note: When a full backup is in session, MySQL tables are locked, which means that no new "Cluster Monitoring / Polling" updates can be added to the Active IQ Unified Manager MySQL database until the full backup process has completed. This can also mean service outage for customers until the full backup is complete.
Warning: If the snapshot or backup will not be taken according to the procedures written above, they might not be consistent and likely will not be usable. hence you will have no backup.
Make sure you followed all the steps of procedure accurately.
- Tablespace sizes
To prevent upgrade failures, users need to check the number of entries in the “opm” schema tables:
Use the following method to check table sizes (in MBs)
mysqlin CLI, to gain access to the database
- Type in the following query to find the sizes of tables in MB
SELECT table_name AS `Table`,
round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB`
WHERE table_schema = "opm"
AND table_name IN ('continuous_event_participant',
The output would be similar to the following:
View table sizes from the database
If the sizes of the tables are significantly large that is if the sum of the two tables are going beyond 700MB, engage NetApp Support to prune data in the tables: continuous_event_participant and continuous_event_participant_stats
- Skewing the upgrade timeouts
If the customer is upgrading 7.2.x/7.3.x to 9.4 or 7.3.x upgrade to 9.5 (9.5 without this timeout change as mentioned in BURT 1214422), they will likely run into BURT 1214422 if they have decent amount of history data. It is therefore, recommended to engage NetApp Support to trim down the two tables: continuous_event_participant and continuous_event_participant_stats
Start the upgrade process now.
Reduce the number of records under OPM schema
If increasing the timeouts still does not help and the upgrade fails, contact Support to check if the number of records in the “opm” schema tables — continuous_event_participant and continuous_event_participant_stats — can be reduced.
Backup and Restore Best Practices
Incremental backups over time can lead to a significant number of files. On restores, this may take significant hours if customers do not use the snapshot option (instances hosted from non-VMware environments). To alleviate this problem, NetApp recommends some best practices to follow for optimum backup and restore timelines:
- Maintain backup retention count to 30 in the Unified Manager GUI.
- For large scale customers, it is recommended to follow the KB: How to start a new Incremental Backup chain
- Force a full backup every month (adhoc backup, in the GUI)
Note: To perform full backup, pick non-business hours as it may take hours for the backup to complete. During this time, the acquisition engine will not be responsive which is equivalent to monitoring outage
- If Unified Manager is hosted from a RHEL/CentOS/Windows:
- Configure backup path to a separate remote path
- As a best practice, you can have a new VM standby with the Unified Manager version installed. Restore the latest
.sqlfile from step 1 (if required)
- On Unified Manager instances hosted from RHEL or the OVA (vApp), the full backups may fail owing to “
/” filesystem getting full. This has been addressed in BURT 1213753; see KB: OnCommand Unified Manager backup failing with "Error Occured while creating backup archive /data/ocum-backup/databasedumps-repo/ocum_mysql_full_backup.sql.7z",
You can find error logs under the following location:
Typical error will be like the following:
2018-09-09 07:14:32,993 ERROR [oncommand] [taskScheduler-4] [Backup|ScheduleBackup] [c.n.dfm.impl.backup.BackupArchiver] Error Occured while creating backup archive /data/ocumbackup/database-dumps-repo/ocum_mysql_full_backup_1536427804511.sql.7z cause :
7-Zip (a)  16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21 p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz (20651),ASM,AES-NI)
Scanning the drive:
1 file, 89533818092 bytes (84 GiB)
Backup might fail when there is not enough space in the root partition for the compressed backup file.
Try to increase the space in the root partition. If you cannot increase the root partition, contact NetApp Support; see BURT 1213753. Request NetApp Support to help with steps mentioned in KB: OnCommand Unified Manager backup failing with "Error Occured while creating backup archive /data/ocum-backup/databasedumps-repo/ocum_mysql_full_backup.sql.7z"
These steps may help you with faster restores:
- If you have VMware snapshots, restores can complete in few minutes.
- If this does not work (or the Unified Manager is not hosted from VMware), restore from the backup path as mentioned in this section.