CONTAP-764421: Switchback/Switchover fails due to configuration error
Issue
- Rarely during Motherboard replacement or head upgrade+rollback scenarios, the switchback/switchover fails with the following errors:
::*> metrocluster operation show
Operation: switchback-simulate
State: failed
Start Time: X/4/2026 10:38:31
End Time: X/4/2026 10:39:07
Errors: Failed to validate the node and cluster components before the switchback operation.
CLUSTERB::CLUSTERB-01 (non-overridable veto): This node is not in a valid configuration state. Make sure the configuration state is "configured" on this node by running "metrocluster node show".
CLUSTERB::CLUSTERB-02 (non-overridable veto): This node is not in a valid configuration state. Make sure the configuration state is "configured" on this node by running "metrocluster node show".- MetroCluster node show output reports configuration error:
ClusterA::*> metrocluster node show
DR Configuration DR
Group Cluster Node State Mirroring Mode
----- ------- ------------------ -------------- --------- --------------------
1 CLUSTERA
CLUSTERA-01 configured enabled waiting for switchback recovery
CLUSTERA-02 configured enabled waiting for switchback recovery
CLUSTERB
CLUSTERB-01 configuration error - switchover completed
CLUSTERB-02 configuration error - switchover completed
4 entries were displayed.
- The following error messages will be seen in the event logs:
[CLUSTERB-01: iw_cm_wq: mcc.config.checkPartnerError:error]: params: {'partner_names': 'CLUSTERB-02,CLUSTERA-01,CLUSTERA-02,CLUSTERA-02,CLUSTERA-01,', 'error': 'Partners seen by this node exceeds the maximum allowed'}- Nodes that have configuration error will have orphan entries in an in-memory MCC kernel node list structure which in-tun can be seen in the +testMetroclusterNodeStatus+ table present in ASUP, and the stale entries are reported as unknown:
CLUSTERB::*> debug smdb table testMetroclusterNodeStatus show
node index partner-node-uuid partner-dblade-uuid partner-cluster-uuid partner-system-id role site-ordinal is-current is-available nv-mirror-enabled dr-group-id is-visible
------------ ----- ------------------------------------ ------------------------------------ ------------------------------------ ----------------- ---- ------------ ---------- ------------ ----------------- ----------- ----------
CLUSTERB-01 0 86af3ac1-b5ab-11ec-a99d-XXXXXXXXXX 86af3ac1-b5ab-11ec-a99d-XXXXXXXXXX 49919ba2-b5ad-11ec-a99d-XXXXXXXXXX XXXX30733 self -1 false false false 1 false
CLUSTERB-01 1 907b6fde-b5ac-11ec-8a8d-XXXXXXXXXX 907b6fde-b5ac-11ec-8a8d-XXXXXXXXXX 49919ba2-b5ad-11ec-a99d-XXXXXXXXXX XXXX30956 ha -1 false false false 1 true
CLUSTERB-01 2 b3652a7c-b4cf-11ec-bdbe-XXXXXXXXXX b3652a7c-b4cf-11ec-bdbe-XXXXXXXXXX c29f4148-b4d1-11ec-bdbe-XXXXXXXXXX XXXX30779 dr-primary
-1 false true false 1 true
CLUSTERB-01 3 b617e732-b4d0-11ec-8fc9-XXXXXXXXXX b617e732-b4d0-11ec-8fc9-XXXXXXXXXX c29f4148-b4d1-11ec-bdbe-XXXXXXXXXX XXXX48904 dr-auxiliary
-1 false true false 1 true
CLUSTERB-01 4 b617e732-b4d0-11ec-8fc9-XXXXXXXXXX b617e732-b4d0-11ec-8fc9-XXXXXXXXXX c29f4148-b4d1-11ec-bdbe-XXXXXXXXXX XXXX84589 unknown
0 false false false 0 false
CLUSTERB-01 5 b3652a7c-b4cf-11ec-bdbe-XXXXXXXXXX b3652a7c-b4cf-11ec-bdbe-XXXXXXXXXX c29f4148-b4d1-11ec-bdbe-XXXXXXXXXX XXXX84645 unknown
0 false false false 0 false