MissingTokenError when "Attempting to retrieve auth token"

Applies to

• NetApp Element software 11.3 and later version
• Management node (mNode)

Issue

The following output is observed in the AIQ Collector contaner (mnode-svc-aiq-collector) logs:

AIQ Collector container (mnode-svc-aiq-collector) logs:

MainThread:[sf.mon.mnode_api_client:get_mnode_settings:101]INFO:Attempting to get session from mnode API
MainThread:[sf.mon.mnode_api_client:log_it:36]DEBUG:Retrying sf.mon.mnode_api_client.MNodeApiClient.get_mnode_settings_with_retries in 60.0 seconds as it raised (Unable to decrypt stored MNode auth configuration settings. The data protection keys may have been removed or overwritten.)

INFO Attempting to retrieve auth token...
/tmp/_MEIpLgicD/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/la...l#ssl-warnings
ERROR (missing_token) Missing access token parameter.
Traceback (most recent call last):
*
*
*
  File "site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 389, in validate_token_parameters
MissingTokenError: (missing_token) Missing access token parameter

 xDBVersionMismatch errors when calling SetAuthConfiguration will be shown at the new cluster master's auth log (/var/log/sf-element-auth.info) :

element-auth[22660]: An error occurred while refreshing the key ring. Will try again in 2 minutes.
element-auth[22660]: element_auth_svc.Exceptions.ServiceUnavailableException: Unable to store configuration data to Element API: ms/MSRestAPIServer.cpp:951
6 void solidfire::MSRestAPIServer::SetAuthConfiguration(const solidfire::JsonParams&, Json::Value&): Check failed: [Throw]
element-auth[22660]: Exception: xDBVersionMismatch at void solidfire::ThrowIfFatalError(const solidfire::DBException&, const string&) (cs/DBCommon.cpp:134): DBClient operation failed due to version mismatch at [/elementauth/mNode]
element-auth[22660]:  ---> Element.Core.Exceptions.ApiServerException: ms/MSRestAPIServer.cpp:9516 void solidfire::MSRestAPIServer::SetAuthConfiguration(const solidfire::JsonParams&, Json::Value&): Check failed: [Throw]
element-auth[22660]: Exception: xDBVersionMismatch at void solidfire::ThrowIfFatalError(const solidfire::DBException&, const string&) (cs/DBCommon.cpp:134): DBClient operation failed due to version mismatch at [/elementauth/mNode]

Swagger UI cannot login or authenticate with the cluster

• Swagger UI gives Internal Server error
• Hybrid Cloud Control (HCC) UI gives 500 connection error
• HCC UI cannot authenticate with the cluster
• mnode-svc-aiq-collector service shows following errors repeatedly:

mnode_mnode-svc-aiq-collector.1.ldqo22pevihd@SF-F9D9    | Sep 12 2020 14:58:49.620 INFO Attempting to retrieve asset information...
mnode_mnode-svc-aiq-collector.1.wli9iaea4d1d@SF-F9D9    | Sep 12 2020 10:31:51.407 INFO Attempting to retrieve asset information...