Unable to connect to the object store "FabricPool_aggr5" from node 123456abcde6543. Reason: Operation Timed out.

Applies to

FabricPool 

StorageGRID

Issue

• The object-store becomes unavailable post StorageGRID certificate expires in ONTAP.

::> storage aggregate object-store show
Aggregate      Object Store Name Availability   Mirror Type
-------------- ----------------- -------------  -----------
aggr5    FabricPool_aggr5 unavailable  primary
aggr6    FabricPool_aggr6 available  primary
2 entries were displayed.

::> event log show -message-name *object*
Time                Node             Severity      Event
------------------- ---------------- ------------- ---------------------------
7/27/2021 02:49:30  node2     EMERGENCY     object.store.unavailable: Unable to connect to the object store "FabricPool_aggr5" from node 123456abcde6543. Reason: Operation Timedout.

::> storage aggregate object-store show -instance
                                  Aggregate Name: aggr5
         ONTAP Name for this Object Store Config: FabricPool_aggr5
                Availability of the Object Store: unavailable
          Reason why Object Store is Unavailable: Connection unavailable
               Type of the Object Store Provider: SGWS
                      License Space Used Percent: -
                        Object Store Mirror Type: primary
    This object store is in mirror degraded mode: -
Force Tiering with no Mirror in a MetroCluster Configuration: false
The name of the Cluster to which the bin belongs: CLUSTER
                                  Aggregate Name: aggr6
         ONTAP Name for this Object Store Config: FabricPool_aggr6
                Availability of the Object Store: available
          Reason why Object Store is Unavailable: -

• Perform network troubleshooting.
• Check for the Intercluster LIF and port status.

::> net int show -role intercluster
  (network interface show)
            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
AUCLANAC1
            node1_intercluster
                         up/up    10.xx.xx.10x/26   node1         a0a-109 true
           node2_intercluster
                         up/up    10.xx.xx.10x/26   node2         a0a-109 true
2 entries were displayed.

From MGWD we could see that the error "Cannot verify the certificate given by the object store"

00000011.971ff5bd 083065cf Mon Jul 26 2021 17:52:57 +08:00 [kern_mgwd:info:2284] 06543ff00: 54321ed0000000256: ERR: MGWD::aggr_object_store_config_iterator: object_store_config_ksmf_op:src/tables/aggr_object_store_config.cc:2480 object_store_config_ksmf_iterator::create() returned error (701:30): "Cannot verify the certificate given by the object store server. It is possible that the certificate has not been installed on the cluster. Use the 'security certificate install -type server-ca' command to install it.". node=node2, config_id=0, check_connection=1, use_iam_role=0

Cause

Expired StorageGRID certificate in ONTAP