AIQ Unified Manager cannot delete snapshots triggered by CommVault

• CommVault rotates snapshots according to its own database but some snapshots are not deleted
• there is no pattern, all random, several clusters are affected
• no error in CommVault because CommVault sends the expiry flag for the snapshot successfully to CommVault
• Unified Manager deletes these snapshots
• Unified Manager has no errors in the main logs but zapioutbound log shows the failed requests

Example of failed request
2021-07-12 13:47:55,254 DEBUG [taskScheduler-3] Executing ZAPI request snapshot-delete to [cluster ip]:
<?xml version='1.0' encoding='UTF-8'?><!DOCTYPE netapp SYSTEM 'file:/etc/netapp_filer.dtd'><netapp xmlns=http://www.netapp.com/filer/admin version="1.7" vfiler="[svm]" originator_id="AIQ_UM"><snapshot-delete><snapshot>SP_2_60486_555763_1525658422</snapshot><volume>[mirrored volume]</volume></snapshot-delete></netapp>
2021-07-12 13:47:55,271 DEBUG [taskScheduler-3] Received ZAPI response for snapshot-delete from [cluster ip]:
<?xml version='1.0' encoding='UTF-8' ?>
<!DOCTYPE netapp SYSTEM 'file:/etc/netapp_gx.dtd'>
<netapp version='1.130' xmlns='http://www.netapp.com/filer/admin'>
<results reason="This Snapshot copy is currently used as a reference Snapshot copy by one or more SnapMirror relationships. Deleting the Snapshot copy can cause future SnapMirror operations to fail. " status="failed" errno="13012"/></netapp>

Deleting on cluster cli fails as follows:
[cluster]::>  snapshot delete -vserver [vserver] -volume [vol] -snapshot [snapshot name]
 
Error: command failed: This Snapshot copy is currently used as a reference Snapshot copy by one or more SnapMirror relationships. Deleting the Snapshot copy can cause future SnapMirror operations to fail.