ONTAP upgrade paused with giveback is in progress reporting “waiting for cluster applications to come online on the local node”

• ONTAP 9
• FAS, AFF systems
• Automatic Non-Disruptive Upgrade (ANDU)

• Automated ONTAP upgrade paused while giveback still in progress with error: 

::> cluster image show-update-progress

                                             Estimated         Elapsed
Update Phase         Status                   Duration        Duration
-------------------- ----------------- --------------- ---------------
Pre-update checks    completed                00:10:00        00:01:14
ONTAP updates        paused-on-error          01:32:00        01:56:34

Details:

Node name            Status            Status Description
-------------------- ----------------- --------------------------------------
Node-01        failed            Error: Node "Node-01" is not in
                                       "connected" state after giveback.
                                       Action: Use the "storage failover
                                       show" command to verify that node
                                       "Node-01" is in the "connected"
                                       state, and then use the "cluster
                                       image resume-update" command to
                                       resume the update.
Node-02        completed
6 entries were displayed.

Status: Paused - An error occurred in "ONTAP updates" phase. The update cannot continue until the error has been resolved. Resolve all issues, then use the "cluster image resume-update" command to resume the update.

• "storage failover show" command output reported “Waiting for cluster applications to come online on the local node. Offline applications: mgmt, vldb, vifmgr, bcomd, crs.” error message for Node-01
• Observed mgmt and vldb services are offline for Node-01 from "cluster ring show" output.
• The version command indicates the system is not updated to a higher version.
•  Node 2 SP/BMC is throwing a mixed version state error.
• From VLDB Logs:

[kern_vldb:info:6887] rdb::qm:Mon May 22 08:11:03 2023:src/rdb/quorum/quorumimpl.cc:2042 (thr_id:0x80d372700) master_remoteOfflineUpcall: Remote offline upcall remoteId: 1000

kern_vldb:info:6887] rdb::qm:Mon May 22 08:11:03 2023:src/rdb/quorum/qm_states/inq/MasterState.cc:104 (thr_id:0x80d372700) MasterState::commFailure: commFailure occured for site: 1000 at 8145s.