Volume creation fails in GCP as VEKs cannot be created for data SVM

• ONTAP 9.9.1 or later
• External Key Manager (EKM)
• Google Cloud Key Management Service (KMS)

• Multiple CreateVolume failures are observed with the following error: 

"Error when creating - Job completed with Error: Failed to create the volume on node \"Node-01\". Reason: Volume encryption keys (VEK) cannot be created or deleted for data SVM \"SVM_2\". External key management has been configured for data SVM \"SVM_2\" but VEKs of encrypted volumes belonging to this SVM are currently stored in the key manager configured for the admin SVM. Use the (privilege: advanced) \"security key-manager key migrate -from-vserver <admin svm_name> -to-vserver <data svm_name>\" command to start using this data SVM's key manager for any new encrypted volumes or unconfigure the key manager for this data SVM. "}".

• The GCP KMS status reports an error while fetching the key from the GCP KMS endpoint:

::> set advanced
::*> security key-manager external gcp check -vserver SVM1

Vserver: SVM1
Node: Node-01

Category: service_reachability
              Status: FAILED
              Details: Google Cloud Key Management Service operation "ENCRYPT" failed.
                          Cryptsoft error: AUTH_FAILED
                          Cryptsoft status: SUCCESS
                          Cryptsoft reason: SUCCESS
                          Cryptsoft message:
                          HTTP response code: 400
                          HTTP payload: {"error":"invalid_grant","error_description":"Invalid grant: account not found"}
                       Issue (privilege: diag) "security key-manager
                       external <azure|aws|gcp|ikp> invoke" for more
                       information.

Category: ekmip_server
              Status: OK

Category: kms_wrapped_key_status
              Status: UNKNOWN
              Details: The top-level internal key protection key (KEK) is
                       unavailable on node Node-01.
                       Reason: The key manager is in mixed state.

• The Key migration fails with the following error:

::*> security key-manager key migrate -from-vserver SVM_1 -to-vserver SVM_2

Starting key migration process which may take several minutes.

Error: command failed: Check that the Google Cloud Key Management Service is healthy using the (privilege: advanced) "security key-manager external gcp check -vserver SVM_2" command, and then run the (privilege: advanced) "security key-manager key migrate -from-vserver <admin vserver_name> -to-vserver SVM_2" command again to complete the migration process. Error: Failed to establish connectivity with the cloud key management service.