2-Node MetroCluster upgrade paused due to overridable vetoes

Applies to

• ONTAP 9
• MetroCluster (two-node-fabric)

Issue

• Output of cluster image show-update-status displays paused-on-error

Example

cluster::> cluster image show-update-progress
                                                Estimated     Elapsed
Cluster                               Duration        Duration Status
------------------------------ --------------- --------------- ----------------
cluster-1                              01:13:00        00:40:54 completed
cluster-2                              01:13:00        00:31:21 paused-on-error

Details: Switchover paused.

[Job 144851] Job failed: Failed to validate the node and cluster components
before the switchover operation. Use the "metrocluster operation show" command
to view detailed error information. Resolve the errors, then try the command again.

•  Output of metrocluster operation show displays overridable veto

Example

cluster::> metrocluster operation show
            Operation: switchover
            State: failed
            Start Time: 3/30/2024 12:57:50
            End Time: 3/30/2024 12:57:56
            Errors: Failed to validate the node and cluster components before the switchover operation.

cluster-1 (overridable veto): The "clus_salt" value in the Onboard Key Manager database is inconsistent. Run the "security key-manager onboard sync" command on cluster cluster-2.

•  Event logs (EMS) displays mgwd: km.cluster.okm.hierarchy.bad:alert

Sat Mar 30 12:33:21 +0100 [cluster-1-01: mgwd: km.cluster.okm.hierarchy.bad:alert]: This cluster is part of a MetroCluster configuration. The Onboard Key Manager key hierarchy on one or both of the clusters is corrupted. Specifically, the 'hashed_pass' object does not match the 'hashed_pass_l' object, part of the 'cluster_kdb' table, in either cluster.