Takeover not possible after AWS CVO node upgrade
Applies to
- Cloud Volumes ONTAP (CVO)
- Amazon Web Services (AWS)
Issue
- CVO major release upgrade started using the CLI
- 1st node completes upgrade and partial giveback performed
Storage failover showis stating takeover is not possible and each node is waiting on the other
Cluster::*> storage failover show
Takeover
Node Partner Possible State Description
-------------- -------------- -------- -------------------------------------
Node1 Node2 false Waiting for Node2, Partial
giveback, Takeover is not possible:
The version of software running on
each node of the SFO pair is
incompatible, Storage failover
interconnect error, NVRAM log not
synchronized, Disk inventory not exchanged
Node2 Node1 false Waiting for Node1, Partial
giveback, Takeover is not possible:
The version of software running on
each node of the SFO pair is
incompatible, Storage failover
interconnect error, NVRAM log not
synchronized, Disk inventory not exchanged
- RDMA interconnect is down for both nodes
Cluster::*> storage failover show -fields interconnect-links,interconnect-up,interconnect-type
node interconnect-up interconnect-links interconnect-type
-------------- --------------- ------------------------------------- -----------------
Node1 false RDMA Interconnect is down (Link down) MVIA-RDMA
Node2 false RDMA Interconnect is down (Link down) MVIA-RDMA
2 entries were displayed.
Cluster::*> ha interconnect status show
(system ha interconnect status show)
Node: Node1
Link Status: down
IC RDMA Connection: down
Node: Node2
Link Status: down
IC RDMA Connection: down
2 entries were displayed.
Cause
- CLI is being used for the upgrade process instead of OCCM
- SRIOV and ENA support is not enabled on the instances
- Possible unsupported instance type
Solution
1. Halt up-rev node in partial giveback (halt -node <up-rev node>)
2. Enable ENA support on instance and boot node
3. Perform giveback after full plex resync (plex show)
4. Change system image on partner node (system node image show/system node image modify)
5. Perform a takeover of down-rev node using the allow-version-mismatch flag to upgrade and enable ENA support on the other instance
6. Perform giveback after full plex resync
7. Cancel the NDU (cluster image cancel-update) and perform any next step upgrade from OCCM
Additional Information
- Enable Enhanced networking for BOTH SRIOV and ENA support
- Verify instance type is supported for current license
Note: Use OCCM for any CVO upgrades
