CRS offline during Azure CVO upgrade

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 198

Visibility:: Public

Votes:: 0

Category:: ontap-9

Specialty:: ds_cvo

Last Updated:

Applies to

ONTAP 9
Cloud Volumes ONTAP for Azure

Issue

2-node Azure CVO upgrade using System Manager.
Post-upgrade of nodes, the upgrade goes in a paused-on-error status due to cluster not being in quorum.

::*> cluster image show-update-progress

                                             Estimated         Elapsed
Update Phase         Status                   Duration        Duration
-------------------- ----------------- --------------- ---------------
Pre-update checks    completed                00:10:00        00:03:04
ONTAP updates        completed                01:32:00        00:49:24
Post-update checks   paused-on-error          00:10:00        00:30:34

Details:

Post-update Check    Status            Error-Action
-------------------- ----------------- --------------------------------------
Cluster Quorum       Error             Error: Cluster is not in quorum.
Status
                                       Action: Use the (privilege: advanced)
                                       "cluster ring show" command to verify
                                       all replication unit details.
7 entries were displayed.

Cluster ring show (run in diag mode) shows CRS rings are not syncing. CRS database can be seen going from online, to offline, to 1 node showing offline in rapid succession :

Example:

cvoxxxxxxxxx::*> cluster ring show Node UnitName Epoch DB Epoch DB Trnxs Master Online --------- -------- -------- -------- -------- --------- --------- cvoxxxxxxxxx-01 mgmt 133 133 2325 cvousncdr008-02 secondary cvoxxxxxxxxx-01 vldb 89 89 760 cvousncdr008-02 secondary cvoxxxxxxxxx-01 vifmgr 89 89 41 cvousncdr008-02 secondary cvoxxxxxxxxx-01 bcomd 88 88 29 cvousncdr008-02 secondary cvoxxxxxxxxx-01 crs 0 511 1 - offline cvoxxxxxxxxx-02 mgmt 133 133 2325 cvousncdr008-02 master cvoxxxxxxxxx-02 vldb 89 89 760 cvousncdr008-02 master cvoxxxxxxxxx-02 vifmgr 89 89 41 cvousncdr008-02 master cvoxxxxxxxxx-02 bcomd 88 88 29 cvousncdr008-02 master cvoxxxxxxxxx-02 crs 511 510 1 cvousncdr008-01 secondary

cvoxxxxxxxxx::*> cluster ring show Node UnitName Epoch DB Epoch DB Trnxs Master Online --------- -------- -------- -------- -------- --------- --------- cvoxxxxxxxxx-01 mgmt 133 133 2325 cvousncdr008-02 secondary cvoxxxxxxxxx-01 vldb 89 89 761 cvousncdr008-02 secondary cvoxxxxxxxxx-01 vifmgr 89 89 41 cvousncdr008-02 secondary cvoxxxxxxxxx-01 bcomd 88 88 29 cvousncdr008-02 secondary cvoxxxxxxxxx-01 crs 513 513 1 cvousncdr008-01 master cvoxxxxxxxxx-02 mgmt 133 133 2325 cvousncdr008-02 master cvoxxxxxxxxx-02 vldb 89 89 761 cvousncdr008-02 master cvoxxxxxxxxx-02 vifmgr 89 89 41 cvousncdr008-02 master cvoxxxxxxxxx-02 bcomd 88 88 29 cvousncdr008-02 master cvoxxxxxxxxx-02 crs 513 512 1 cvousncdr008-01 secondary