Event log showing EMERGENCY errors for FabricPool
Applies to
- ONTAP 9
- CVO
- FabricPool
Issue
- Cluster is reporting object store unavailable events.
Cluster1::> event log show -severity EMERGENCY
Time Node Severity Event
------------------- ---------------- ------------- ---------------------------
9/25/2023 00:58:19 Cluster1-01 EMERGENCY object.store.unavailable: Unable to connect to the object store "fabric-pool" from node 274f35b6-3f6d-11ed-8fdc-4564677241b4. Reason: Expired token.
- EMS shows, the object store is unavailable and becomes available within the next minute.
Sun Sep 24 02:30:33 +1000 [Cluster-01: OscHighPriThreadPool: object.store.unavailable:debug]: Unable to connect to the object store "fabric-pool" from node 274f35b6-3f6d-11ed-8fdc-4564677241b4. Reason: Expired token.
Sun Sep 24 02:31:20 +1000 [Cluster-01: OscLowPriThreadPool: object.store.available:debug]: Able to connect to the object store "fabric-pool" from node 274f35b6-3f6d-11ed-8fdc-4564677241b4.
- These events are reported every 6 hours.
- IAM Role is being referenced by the object-store and no issue with the current expiration date/time.
Cluster1::*> storage aggregate object-store ontap-cloud credentials show
Node IAM Role Credentials Expiry
-------------- ---------------------------------------- ----------------------
Cluster1-01 cvo-instance-profile-version10-73a-IamInstanceRole-C5XXXXXXXKU 2023-09-25T02:49:15Z