Object Store is unavailable after node reboot due to SSL cert expired on F5 Network Load Balancer
Applies to
- ONTAP 9
- StorageGrid
- F5 Network Load Balancer
Issue
-
A node in an ONTAP on-prem cluster panic's due to failed hardware and was taken over.
-
The node was recovered, but giveback fails and there is an object store inaccessible on the aggregates owned by the node that rebooted.
cluster::> aggregate object-store show
(storage aggregate object-store show)
Aggregate Object Store Name Availability Mirror Type
-------------- ----------------- ------------- -----------
...
node1_aggr2 NPH_StorageGRID available primary
node3_aggr1 NPH_StorageGRID unavailable primary
node3_aggr2 NPH_StorageGRID available primary
6/8/2024 08:53:51 NODE04 ERROR Unable to connect to the object store "StorageGRID" from node 266af68c-6536-11e8-bcdd-xxxxxxxxxxxx. Reason: Connection unavailable.
6/8/2024 08:31:12 NODE04 ALERT sfo.giveback.attemptExceeded: Attempts for automatic giveback of SFO aggregates exceeded the maximum number (3) of allowed attempts.
6/8/2024 08:30:36 NODE04 ALERT sfo.giveback.failed: Giveback of aggregate node3_aggr2 failed due to destination check failed.
6/8/2024 08:30:36 NODE04 ALERT sfo.sendhome.subsystemAbort: The giveback operation of 'node3_aggr2' was aborted by 'fabric pools'.
6/8/2024 08:30:36 NODE04 ERROR gb.netra.ca.check.failed: Giveback of aggregate 'node3_aggr2' (uuid: 27f187b6-45f7-4125-b1ac-xxxxxxxxxxxx) failed due to Object store is not reachable on destination preventing object store access on the destination node.
-
The intercluster LIF is healthy on the node and can be used to ping the object store.
-
HTTPS is used and SSL validation is enabled.
cluster::> storage aggregate object-store config show -fields server,port
object-store-name server port
----------------- ------------------------------ ----
StorageGRID storagegrid.domain.com 443 -
There is a custom CA authority and certificate configuration.
-
There is an expired certificate.
cluster::*> security certificate truststore check -server storagegrid.domain.com -vserver cluster
Error: command failed: Failed to verify server's certificate chain. Reason: certificate has expired
-
The certificates are checked in ONTAP and StorageGrid no issues are found.
-
Deleting and re-importing the signed certificates does not help.