How do Active IQ Unified Manager incremental backups work?

Overview

Introduced in OnCommand Unified Manager 7.3P1, backups are no longer full backups but rather an initial full backup followed by incremental backups. As stated in our Unified Manager 7.3 Documentation Center, a backup consists of a single file in the backup directory and one or more files in the database repository directory. The file in the backup directory is very small because it contains only a pointer to the files located in the database repository directory that are required to recreate the backup.

The first time you generate a backup, a single file is created in the backup directory and a full backup file is created in the database repository directory. The next time you generate a backup, a single file is created in the backup directory and an incremental backup file is created in the database repository directory that contains the differences from the full backup file. This process continues as you create additional backups, up to the maximum retention setting, as shown in the following figure. 

Default Backup & Repository Directories

• Microsoft Windows Server
  • Backup Directory = :\ProgramData\NetApp\OnCommandAppData\ocum\backup
  • Database Repository = :\ProgramData\NetApp\OnCommandAppData\ocum\backup\database_dumps_repo
• RedHat Enterprise Linux
  • Backup Directory = /opt/netapp/data/ocum-backup
  • Database Repository = /opt/netapp/data/ocum-backup/database-dumps-repo
• VMware Open Virtual Application (OVA)
  • Backup Directory = /data/ocum-backup
  • Database Repository = /data/ocum-backup/database-dumps-repo

Logic Walkthrough

Below is example of a OnCommand Unified Manager 9.4 OVA server. Examine each backup in turn. Notice there are two different directories here: /data/ocum-backup and /data/ocum-backup/database-dumps-repo

root@OnCommand:/data/ocum-backup# ls -lh                                                                          root@OnCommand:/data/ocum-backup/database-dumps-repo# ls -lh        
total 424K                                                                                                        total 695M
-rw-r--r-- 1 jboss shadow 137K Mar 21 14:47 UM_9.4P1_backup_unix_03-21-2019-14-33.7z                              -rw-r--r-- 1 jboss shadow 694M Mar 21 14:47 ocum_mysql_full_backup_1553196783191.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 21 15:04 UM_9.4P1_backup_unix_03-21-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow  15K Mar 21 15:04 ocum_mysql_incremental_backup_1553198640039.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 25 15:04 UM_9.4P1_backup_unix_03-25-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow 1.4M Mar 25 15:04 ocum_mysql_incremental_backup_1553544241947.sql.7z
drwxrwxrwx 2 jboss shadow 4.0K Mar 26 13:30 database-dumps-rep

Highlighted below, see the first backup that was generated on March 21st @ 14:47. On the right, you can see the corresponding full backup that was taken. Notice the size of the first backup. It is expected this initial first backup to be large as it is a full backup.

root@OnCommand:/data/ocum-backup# ls -lh                                                                          root@OnCommand:/data/ocum-backup/database-dumps-repo# ls -lh        
total 424K                                                                                                        total 695M
-rw-r--r-- 1 jboss shadow 137K Mar 21 14:47 UM_9.4P1_backup_unix_03-21-2019-14-33.7z                              -rw-r--r-- 1 jboss shadow 694M Mar 21 14:47 ocum_mysql_full_backup_1553196783191.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 21 15:04 UM_9.4P1_backup_unix_03-21-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow  15K Mar 21 15:04 ocum_mysql_incremental_backup_1553198640039.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 25 15:04 UM_9.4P1_backup_unix_03-25-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow 1.4M Mar 25 15:04 ocum_mysql_incremental_backup_1553544241947.sql.7z
drwxrwxrwx 2 jboss shadow 4.0K Mar 26 13:30 database-dumps-rep

Highlighted below, see the second backup that was generated later that same day on March 21st @ 15:04. On the right side, an incremental backup was taken this time instead of a full backup. Notice the small size. The backup file on the left points to both files highlighted on the right as the backup now requires both the initial full backup and the incremental backup in order to restore the backup.

root@OnCommand:/data/ocum-backup# ls -lh                                                                          root@OnCommand:/data/ocum-backup/database-dumps-repo# ls -lh        
total 424K                                                                                                        total 695M
-rw-r--r-- 1 jboss shadow 137K Mar 21 14:47 UM_9.4P1_backup_unix_03-21-2019-14-33.7z                              -rw-r--r-- 1 jboss shadow 694M Mar 21 14:47 ocum_mysql_full_backup_1553196783191.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 21 15:04 UM_9.4P1_backup_unix_03-21-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow  15K Mar 21 15:04 ocum_mysql_incremental_backup_1553198640039.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 25 15:04 UM_9.4P1_backup_unix_03-25-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow 1.4M Mar 25 15:04 ocum_mysql_incremental_backup_1553544241947.sql.7z
drwxrwxrwx 2 jboss shadow 4.0K Mar 26 13:30 database-dumps-rep

Again, highlighted below, a third backup was taken several days later, on March 25th @ 15:04. As before, all the highlighted files on in the database-dumps-repodirectory would be required to restore the backup pointer file on the left.

root@OnCommand:/data/ocum-backup# ls -lh                                                                          root@OnCommand:/data/ocum-backup/database-dumps-repo# ls -lh        
total 424K                                                                                                        total 695M
-rw-r--r-- 1 jboss shadow 137K Mar 21 14:47 UM_9.4P1_backup_unix_03-21-2019-14-33.7z                              -rw-r--r-- 1 jboss shadow 694M Mar 21 14:47 ocum_mysql_full_backup_1553196783191.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 21 15:04 UM_9.4P1_backup_unix_03-21-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow  15K Mar 21 15:04 ocum_mysql_incremental_backup_1553198640039.sql.7z
-rw-r--r-- 1 jboss shadow 137K Mar 25 15:04 UM_9.4P1_backup_unix_03-25-2019-15-04.7z                              -rw-r--r-- 1 jboss shadow 1.4M Mar 25 15:04 ocum_mysql_incremental_backup_1553544241947.sql.7z
drwxrwxrwx 2 jboss shadow 4.0K Mar 26 13:30 database-dumps-rep

If the goal is to collect the needed backup file(s) in order to restore this Unified Manager 9.4 OVA server on a freshly deployed Unified Manager 9.4 OVA server. Additionally, if it is required to restore the latest backup there is of Unified Manager. In situations where many incremental backups are present, it is a good idea to be sure you have all the files needed in the event you need to move the backup off box. To check which files are needed, simply check which backup files are available in your backup directory, "/data/ocum-backup".

root@OnCommand:/data/ocum-backup# ls -lh
total 424K
-rw-r--r-- 1 jboss shadow 137K Mar 21 14:47 UM_9.4P1_backup_unix_03-21-2019-14-33.7z
-rw-r--r-- 1 jboss shadow 137K Mar 21 15:04 UM_9.4P1_backup_unix_03-21-2019-15-04.7z
-rw-r--r-- 1 jboss shadow 137K Mar 25 15:04 UM_9.4P1_backup_unix_03-25-2019-15-04.7z
drwxrwxrwx 2 jboss shadow 4.0K Mar 26 13:30 database-dumps-repo

In this example, use the backup file "UM_9.4P1_backup_unix_03-25-2019-15-04.7z". Use '7z' to list the files and directories contained in the backup file.

root@OnCommand:/data/ocum-backup# 7z l UM_9.4P1_backup_unix_03-25-2019-15-04.7z

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,4 CPUs Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (306A0),ASM,AES-NI)

Scanning the drive for archives:
1 file, 139790 bytes (137 KiB)

Listing archive: UM_9.4P1_backup_unix_03-25-2019-15-04.7z

--
Path = UM_9.4P1_backup_unix_03-25-2019-15-04.7z
Type = 7z
Physical Size = 139790
Headers Size = 523
Method = LZMA2:384k
Solid = +
Blocks = 1

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2019-03-25 15:04:29 D....            0            0  au_conf_cert
2019-03-25 15:04:29 D....            0            0  custom
2019-03-25 15:04:29 D....            0            0  data_dir_ocie
2019-03-25 15:04:29 D....            0            0  database_dumps
2019-03-25 15:04:29 D....            0            0  etc_security
2019-03-25 15:04:29 D....            0            0  jboss_onaro_cert
2019-03-25 15:04:29 D....            0            0  jboss_onaro_cert/originator
2019-03-21 14:33:03 ....A            0            0  database_dumps/ocum_mysql_full_backup_1553196783191.sql
2019-03-21 15:04:00 ....A            0            0  database_dumps/ocum_mysql_incremental_backup_1553198640039.sql
2019-03-25 15:04:29 ....A            0            0  database_dumps/ocum_mysql_incremental_backup_1553544241947.sql
2019-03-25 15:04:29 ....A       178103       139267  au_conf_cert/client.keystore
2019-03-25 15:04:29 ....A         2142               data_dir_ocie/server.keystore
2019-03-25 15:04:29 ....A        83581               etc_security/autosupport.truststore
2019-03-25 15:04:29 ....A         3761               etc_security/saml_sp_metadata.xml
2019-03-25 15:04:29 .R..A         2142               jboss_onaro_cert/server.keystore
2019-03-25 15:04:29 .R..A         2161               jboss_onaro_cert/server.keystore.default
2019-03-25 15:04:29 ....A          175               ocum.conf
2019-03-25 15:04:29 ....A          554               ocum_backup_metadata.properties
------------------- ----- ------------ ------------  ------------------------
2019-03-25 15:04:29             272619       139267  11 files, 7 folders

Note: Do not let get confused by the "database_dumps" directory. This is simply a directory inside of the backup archive file. If it is extract the "UM_9.4P1_backup_unix_03-25-2019-15-04.7z"  backup file, see the above directories and files instruction. Be interested in the full and incremental backups inside this "database_dumps" directory as they are the repository files that "UM_9.4P1_backup_unix_03-25-2019-15-04.7z" points to when restoring.

Looking at the output, it is only needed to focus on several lines. Below is the output, but simplified down to what is actually needed to look at. Notice the full backup, and incremental backups. These are the only items of interest.

root@OnCommand:/data/ocum-backup# 7z l UM_9.4P1_backup_unix_03-25-2019-15-04.7z

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2019-03-21 14:33:03 ....A            0            0  database_dumps/ocum_mysql_full_backup_1553196783191.sql
2019-03-21 15:04:00 ....A            0            0  database_dumps/ocum_mysql_incremental_backup_1553198640039.sql
2019-03-25 15:04:29 ....A            0            0  database_dumps/ocum_mysql_incremental_backup_1553544241947.sql
------------------- ----- ------------ ------------  ------------------------

Note: In the event, you only see a single ocum_mysql_full_backup_xxxxxxxxxxxxxx.sql file in your output, this means your backup pointer file is only pointing towards a full backup and no incremental backups. In this situation, collect the backup pointer file and the full backup.

In summary, it is required to collect the following files from the OnCommand Unified Manager server if it is needed to move the backup off box.

/data/ocum-backup/UM_9.4P1_backup_unix_03-25-2019-15-04.7z
/data/ocum-backup/database-dumps-repo/ocum_mysql_full_backup_1553196783191.sql
/data/ocum-backup/database-dumps-repo/ocum_mysql_incremental_backup_1553198640039.sql
/data/ocum-backup/database-dumps-repo/ocum_mysql_incremental_backup_1553544241947.sql