CONTAP-89115: CRS recovery may fail because the encryption key database gets incorrectly and completely filled with encryption keys
Issue
- MDV creation, which runs as a job in the background, fails and is retried in a loop.
- As each job fails, the EMS event message [mgmtgwd.jobmgr.jobcomplete.failure] is generated.
- Although the EMS event message reports that volume creation failed, ONTAP still stores a key in the encryption key database.
- A [crypto_key_stored] event is generated each time ONTAP stores a key in the encryption key database.
- The [crypto_key_stored] and [mgmtgwd.jobmgr.jobcomplete.failure] events should be adjacent in the EMS log.
- This loop results in large number of encryption keys being created, filling the encryption key database to capacity.
- Possible impacts that users may see include:
- Encrypted volume creation failure.
- Automatic switchover will be disabled.
- The following error message will be generated multiple times:
[Failed to create the volume on node "STG0000000172-01". Reason: Internal error. Failed to generate volume key in kernel. Key manager returned ERROR_KEY_GEN_CRYPTO_FAILED. Crypto return code: 8.]