ANDU fails in post-apps phase due missing RBAC role for local user. Error: entry doesn't exist
Applies to
- ONTAP 9
- Automated Non-disruptive Update (ANDU)
- Role Based Access Control (RBAC)
Issue
- ANDU from 9.7 to 9.9.1P7 paused due to failed task.
- Nodes are upgraded, but the effective Cluster version remains at previous ONTAP version
::> set d::*> node run -node * -c versionNode: Node-01
NetApp Release 9.9.1P7: Fri Jan 28 20:28:26 EST 2022
Node: Node-02
NetApp Release 9.9.1P7: Fri Jan 28 20:28:26 EST 2022
::*> version
NetApp Release 9.7P8: Thu Oct 15 04:11:57 UTC 2020
Info: The output from the version command above may not be correct because
upgrade is in progress or has failed in one or more nodes in the cluster.
Use the "upgrade-revert show" command in advanced mode to view the status
of upgrade.
- ANDU is paused on error:
::*> cluster image show-update-progress
Estimated Elapsed
Update Phase Status Duration Duration
-------------------- ----------------- --------------- ---------------
Pre-update checks completed 00:10:00 00:01:37
ONTAP updates paused-on-error 01:32:00 1 days 00:59
Details:
Node name Status Status Description
-------------------- ----------------- --------------------------------------
Node-01 failed Error: Upgrade of node
"Node-01" failed to complete.
Action: Use the (privilege: advanced)
command "system node upgrade-revert
show -node <nodename>" to check the
upgrade status.
Node-02 completed
8 entries were displayed.
- Upgrade-revert shows the upgrade is aborted in the post-apps phase. Error: entry doesn't exist.
::*> upgrade-revert show -node Node-01
(system node upgrade-revert show)
Node: Node-01 Status: aborted
Status Message: The upgrade aborted. Contact support personnel for the upgrade repair procedure.
Vers Phase Status Upgrade Phase Status Message
---- ---------- -------- ------------------------------------------------------
600 pre-root applied No upgrade is required for this phase.
600 pre-apps applied Upgrade successful.
600 post-apps applied Upgrade successful.
700 pre-root applied No upgrade is required for this phase.
700 pre-apps applied Upgrade successful.
700 post-apps applied Upgrade successful.
800 pre-root applied No upgrade is required for this phase.
800 pre-apps applied Upgrade successful.
800 post-apps applied Upgrade successful.
900 pre-root applied No upgrade is required for this phase.
900 pre-apps applied Upgrade successful.
900 post-apps applied Upgrade successful.
1100 pre-root applied No upgrade is required for this phase.
1100 pre-apps applied Upgrade successful.
1100 post-apps applied Upgrade successful.
1400 pre-root applied No upgrade is required for this phase.
1400 pre-apps applied Upgrade successful.
1400 post-apps aborted entry doesn't exist
18 entries were displayed.
- There is a failed upgrade task. Task ID 1310.
::*> upgrade-revert show-task-status -node Node-01 -status aborted
(system node upgrade-revert show-task-status)
Node Vers Phase ID Proc Status Description
------ ------ ---- ---- ---- ------- -----------------------------
Node-01 1300 post-apps 1310 mgwd aborted Predefined_roles_update_upgrade
- Attempts to manually run the task fails with "entry doesn't exist".
::*> upgrade-revert upgrade-task -node Node-01 -version 1300 -startup-phase post-apps -process mgwd -task-id 1310
(system node upgrade-revert upgrade-task)
Error: command failed: entry doesn't exist
- In the mgwd logs we see it failing when looking up pre-defined roles for a user.
MGWD logs from Node_01:
Wed Mar 30 2022 11:47:07 -07:00 [kern_mgwd:info:2179] 0x837f7bb00: 0: ERR: RBAC::Users: handleModifyOperation:src/util/rbacUtil.h:3843 entry doesn't exist
Wed Mar 30 2022 11:47:07 -07:00 [kern_mgwd:info:2179] 0x837f7bb00: 0: ERR: RBAC::Users: doNotifyImp:3112 Operation 0 failed with error entry doesn't exist; UserProfileTable: user_name;service-processor;password;-
Wed Mar 30 2022 11:47:07 -07:00 [kern_mgwd:info:2179] 0x837f7bb00: 0: ERR: smf::userprofile: notify_imp: Operation 0 failed with error: entry doesn't exist [131]
Wed Mar 30 2022 11:47:07 -07:00 [kern_mgwd:info:2179] 0x837f7bb00: 0: ERR: RBAC: upgradeCommit,src/tables/vserverrbac/rbac.cc:1080: CWPredefinedRolesUpgradeTask: 9.9.1 - Service-processor user modification failed: entry doesn't exist
Wed Mar 30 2022 11:47:07 -07:00 [kern_mgwd:info:2179] 0x837f7bb00: 0: INFO: upgrade_revert: Upgrade of row [1300,post-apps,1310,mgwd] failed. Retrying. Reason: entry doesn't exist.