Unable to connect to the object store "FabricPool_aggr5" from node 123456abcde6543. Reason: Operation Timed out.
Applies to
- ONTAP 9
- FabricPool
- StorageGRID
Issue
- Aggergate object-store becomes unavailable after StorageGRID certificate expiration:
::> storage aggregate object-store show -instance
Aggregate Name: aggr5
ONTAP Name for this Object Store Config: FabricPool_aggr5
Availability of the Object Store: unavailable
Reason why Object Store is Unavailable: Connection unavailable
Type of the Object Store Provider: SGWS
License Space Used Percent: -
Object Store Mirror Type: primary
This object store is in mirror degraded mode: -
Force Tiering with no Mirror in a MetroCluster Configuration: false
The name of the Cluster to which the bin belongs: CLUSTER
Aggregate Name: aggr6
ONTAP Name for this Object Store Config: FabricPool_aggr6
Availability of the Object Store: available
Reason why Object Store is Unavailable: -
- EMS logs shows as reason object store unavailable
Operation timeout:
::> event log show -message-name *object*
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
7/27/2021 02:49:30 node2 EMERGENCY object.store.unavailable: Unable to connect to the object store "FabricPool_aggr5" from node 123456abcde6543. Reason: Operation Timeout.
- Intercluster LIF and port status are up:
::> net int show -role intercluster
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
AUCLANAC1
node1_intercluster
up/up 10.xx.xx.10x/26 node1 a0a-109 true
node2_intercluster
up/up 10.xx.xx.10x/26 node2 a0a-109 true
2 entries were displayed.
-
MGWD shows error
Cannot verify the certificate given by the object store:
[kern_mgwd:info:2284] 06543ff00: 54321ed0000000256: ERR: MGWD::aggr_object_store_config_iterator: object_store_config_ksmf_op:src/tables/aggr_object_store_config.cc:2480 object_store_config_ksmf_iterator::create() returned error (701:30): "Cannot verify the certificate given by the object store server. It is possible that the certificate has not been installed on the cluster. Use the 'security certificate install -type server-ca' command to install it.". node=node2, config_id=0, check_connection=1, use_iam_role=0
Cause
Expired StorageGRID certificate in ONTAP
