ONTAP 9.8 with OKM: Giveback fails intermittently due to keys missing

Applies to

Issue

• ONTAP 9.8 node with OKM fails to import onboard key hierarchy during boot

Tue Oct 05 12:06:01 -0500 [Clus-02: sysinit_thread: crypto.debug:info]: cryptomod key table initialized with room for 10 keys (0 pages).
Tue Oct 05 12:06:01 -0500 [Clus-02: sysinit_thread: crypto.ssal.failed:alert]: SSAL operation failed: SSAL Unseal operation failed.
Tue Oct 05 12:06:01 -0500 [Clus-02: sysinit_thread: crypto.debug:info]: Onboard key hierarchy import failed: failed to create NKEK: 31.
Tue Oct 05 12:06:01 -0500 [Clus-02: sysinit_thread: crypto.okmrecovery.failed:alert]: ERROR: Import of the onboard key hierarchy failed: failed to import key hierarchy. Additional information: error: ssal unseal failedWed Oct 05 12:10:01 -0500 [Cluster01-01: cf_giveback: gb.sfo.veto.kmgr.keysmissing:error]: Giveback of aggregate Aggr_1 failed due to unavailability of volume encryption keys for the encrypted volumes of the aggregate on the partner node Cluster01-02.
...
Tue Oct 05 12:07:01 -0500 [Clus-02: rc: cf.fm.waitingForGB:debug]: params: {'reason': 'WFG: partner f/w state is SF_TO'}
Tue Oct 05 12:09:40 -0500 [Clus-02: clam.node.inq:info]: Cluster node (name=CS_OTH_TR2_PRD1-01, ID=1000) is in "CLAM quorum".
Tue Oct 05 12:09:40 -0500 [Clus-02: clam.node.avail.change:debug]: The availability status of node (name=CS_OTH_TR2_PRD1-01, ID=1000) changed from Unknown to Available.
...
Tue Oct 05 12:10:01 -0500 [Clus-02: monitor: monitor.globalStatus.ok:notice]: The system's global status is normal. 
​Tue Oct 05 12:10:01 -0500 [Clus-02: monitor: license.state.v2.modified:debug]: Licensing state for local node changed from false to true.​​​​

• ONTAP 9.8 partner node vetoes SFO giveback due to keys missing 

Tue Oct 05 12:10:01 -0500 [Clus-01: cf_giveback: sfo.sendhome.subsystemAbort:alert]: The giveback operation of 'Aggr_1' was aborted by 'keymanager'
Tue Oct 05 12:10:01 -0500 [Clus-01: sfo.giveback.failed:alert]: Giveback of aggregate Aggr_1 failed due to Giveback was vetoed..
Tue Oct 05 12:10:01 -0500 [Clus-01: The giveback operation of 'Aggr_1' was aborted by 'keymanager'.
Tue Oct 05 12:10:01 -0500 [Clus-01: sfo.retry.autoGiveback:info]: Automatic giveback of SFO aggregates will be retried after 5 minutes.
​​​​...
Tue Oct 05 12:15:01 -0500 [Clus-01: cf_giveback: gb.sfo.veto.kmgr.keysmissing:error]: Giveback of aggregate Aggr_1 failed due to unavailability of volume encryption keys for the encrypted volumes of the aggregate on the partner node Cluster01-02.
Tue Oct 05 12:15:01 -0500 [Clus-01: cf_giveback: sfo.sendhome.subsystemAbort:alert]: The giveback operation of 'Aggr_1' was aborted by 'keymanager'
​​​Tue Oct 05 12:15:01 -0500 [Clus-01: The giveback operation of 'Aggr_1' was aborted by 'keymanager'.
Tue Oct 05 12:15:01 -0500 [Clus-01: sfo.giveback.attemptExceeded:alert]: Attempts for automatic giveback of SFO aggregates exceeded the maximum number (3) of allowed attempts.