VMware Backups with SnapCenter from 3.0 until 4.1 can leave VMware Snapshots behind on the VMs when VMs fail to quiesce
Applies to
- SnapCenter 3.0, 3.0.1, 4.0, 4.1, 4.1P1
- SnapCenter Plugin for VMware
- vSphere 6.x
- ESXi 6.x
Issue
When looking at the VM after the backup, one or more VM snapshots still exists that were not there before the backup, and the backup will have finished with warnings.
If the limit of 31 snapshots has been reached, the backup fails completely.
In the Plugin's aegis logs, you may see these:
com.netapp.aegis.vsphere.VsphereService An error occurred while quiescing the virtual machine. See the virtual machine's event log f or details. com.netapp.aegis.backup.QuiesceManager VMware snapshot status :
Failure com.netapp.aegis.backup.QuiesceManager exceptionDuringQuiesce : true ErrorCode (-1), ErrorMessage (Exception occurred when quiescing virtual machines: cluster_node1, cluster_node2, standalone-vm.
Please check 'Create virtual machine snapshot' task on vSphere Web Client or SnapCenter Plug-in for VMware vSphere logs f or detailed error message.)
During the Backup you can see that some VMs SnapShots (VMware SnapShots) are created and then at the end of the job SC did not try to delete them because there was a connectivity issue during the backup. That means SC lost the communication with vCenter and it was no aware if the VMware SnapShot creation completed with or without errors.
Information can be seen in the aegis.log
Here you can see the Exception while creating the snapshot because of the connection error
2019-01-15T20:30:43.0000103Z WARN VSC com.netapp.aegis.vsphere.VsphereSnapshotServiceGot Exception while creating snapshot 'VSC__502d76c4-d544-4972-815c-3716c0fea0bc' : java.rmi.RemoteException: VI SDK invoke exception:java.net.ConnectException: Connection timed out: connect
2019-01-15T20:30:43.0000103Z DEBUG VSC com.netapp.aegis.backup.QuiesceManager VMware snapshot status : Failure
On the VMware logs (vpxd.log) you can see that the "CreateSnapshot" did complete with no error
2019-01-15T20:30:13.090Z info vpxd[7F8CB3060700] [Originator@6876 sub=vpxLro opID=6648cc37] [VpxLRO] --BEGIN task-80754 -- vm-1638 -- vim.VirtualMachine.createSnapshot -- 5207a641-9b2c-a8ab-14ce-f21e274542c9(525effb7-e333-b4a2-6f4b-673cb2de3d7b)
2019-01-15T20:31:18.637Z info vpxd[7F8CB3060700] [Originator@6876 sub=vpxLro opID=6648cc37] [VpxLRO] --FINISH task-80754