storage stackmon shows failed-reconcile state in a node after upgrade
Applies to
- MetroCluster
- ONTAP (9.13.x/9.14.x)
Issue
- EMS in the event log will report:
Sat Aug 26 06:02:21 +0000 [node6: stackmon_thread: callhome.stackmon.recon.failed:alert]: Call home for STACKMON RECONCILE FAILED
- This message occurs when the system fails to assign the correct stack-id to one or more disk shelves.
- The shelf and disk names might not correctly reflect the storage topology.
- Shelf and disk operations might fail if multiple disks share the same name.
- Shelf population and id conflict errors may be reported.
[node6: dsa_disc: ses.config.shelfPopError:error]: Disk shelf population on channel Switch:10 is 16. Reduce the number of disk shelves on this channel to 10 or fewer.
[node6: stackmon_thread: stackmon.stack.id.conflict:error]: Stack ID of the Shelf 1 (S/N xxxxxxxxxxxxxx) on channel Switch:10 cannot be agreed on by all the nodes in the cluster.
- This could lead to Shelf firmware updates failing momentarily
[node6: dsa_sfu: sfu.downloadError:error]: [storage download shelf]: Error downloading firmware file IOM12A.0310.SFW on 0a.shelf6.Subenclosure status error.General firmware update error.
[node6: dsa_sfu: sfu.downloadError:error]: [storage download shelf]: Error downloading firmware file IOM12A.0310.SFW on 0a.shelf5.Subenclosure status error.General firmware update error.
[node6: dsa_sfu: sfu.downloadSummaryErrors:error]: Shelf firmware updated on 3 shelves, failed to update on 6 shelves.
[node6: dsa_sfu: sfu.firmwareDownrev:error]: Disk shelf firmware needs to be updated on 6 disk shelves.
[node6: dsa_sfu: sfu.downloadSummary:info]: Shelf firmware updated on 6 shelves.
storage stackmon showshows a node in a failed-reconciled state:
cluster1::*> storage stackmon show
Node State Paused
----- ---------------- ------
node5 stable false
node6 failed-reconcile false
STORAGE-STACKMON-TOPO.XMLAutosupport section may also show Reconcile State as unknown or failed
