NFS hangs because Reason why object Store is Unavailable: Bad request
Applies to
- ONTAP 9
- NFS
- AWS S3Bucket
- FabricPool
- Cloud Volumes ONTAP (CVO)
- Cloud Tiering (Object store)
Issue
- NFS client tries read the contents of an old file from an NFS export on a CVO instance, but the read operation hangs indefinitely
- Packet trace shows that
- ONTAP sends
Error: NFS3ERR_JUKEBOX
- ONTAP sends
- NFS export is to a volume, which is on an aggregate, which is tiering cold data to an object store, which is
unavailable-
::> vol show -junction-path /Share -fields aggregate,node vserver volume aggregate node -------- ------ ----------- ------- svm1 Share aggr_data_1 node-01
-
::> object-store show (storage aggregate object-store show) Aggregate Object Store Name Availability Mirror Type -------------- ----------------- ------------- ----------- aggr_data_1 S3Bucket unavailable primary
-
- The object store is
unavailabledue toBad request-
::> object-store show -instance (storage aggregate object-store show) Aggregate Name: aggr_data_1 ONTAP Name for this Object Store Config: S3Bucket UUID of the Bin: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Bin Number: 1 Availability of the Object Store: unavailable Reason why Object Store is Unavailable: Bad request Object Store Config Id in Dblade: 1 S3 Client Handle in Cbin: 0 SMDB Error for Partially Failed get() or next(): - Bin is being Created: false Is Bin Deleted?: false UUID of the Aggregate: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy UUID of the Object Store Configuration: zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz Type of the Object Store Provider: AWS_S3 Cached Value of Used Space: 45.24TB License Space Used Percent: - Threshold for Reclaiming Unreferenced Space: 20% Aggregate Fullness Threshold Required for Tiering: 50% Name Format Version: 1 Object Store Mirror Type: primary This object store is in mirror degraded mode: - Force Tiering with no Mirror in a MetroCluster Configuration: false Allow a degraded cloud mirror to be swapped with an unavailable primary object store: - Pending Operation on the Bin: none True if this object store is the primary destination for IO: true The name of the Cluster to which the bin belongs: cluster_name Modify Count for the Row: 0
-
- Object store profiler fails due to
Invalid token-
::*> storage aggregate object-store profiler start -node node-01 -object-store-name S3Bucket Warning: This is a CPU intensive command. It is recommended to run this command when the system is under 50% CPU utilization. This command will issue 2500 PUT requests for a time period of 60 seconds, followed by maximum GET requests to the object store for a time period of 180 seconds. This may result in additional charges to your object store account. Do you want to continue? Do you want to continue? {y|n}: y Warning: Object store profiler tests the connection to the object store and is not reflective of FabricPool PUT throughput at steady state. Error: command failed: Initializing the object store profiler on node "node-01" for object store "S3Bucket" failed. Reason: Invalid token. Wait a few minutes and try the command again.
-
- Node's cloud credential is expired
-
::*> object-store ontap-cloud credentials show (storage aggregate object-store ontap-cloud credentials show) Node IAM Role Credentials Expiry -------------- ---------------------------------------- ---------------------- node-02 fabric-pool-resources-zzzzzzzz-zzz-IamInstanceRole-zzzzzzzzzzzz 2025-10-27T21:59:25Z node-01 fabric-pool-resources-zzzzzzzz-zzz-IamInstanceRole-zzzzzzzzzzzz 2025-10-20T12:52:41Z 2 entries were displayed.
-
- EMS contains
object.store.unavailable[?] Mon Oct 20 21:05:06 -0400 [node-01: OscLowPriThreadPool: object.store.unavailable:EMERGENCY]: Unable to connect to the object store "S3Bucket" from node aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa. Reason: Bad request.[?] Tue Oct 21 01:10:30 +0000 [node-01: wafl_junkd_exempt: object.store.unavailable:EMERGENCY]: Unable to connect to the object store "S3Bucket" from node aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa. Reason: Bad request.
