Skip to main content
NetApp Knowledge Base

NFS hangs because Reason why object Store is Unavailable: Bad request

Views:
107
Visibility:
Public
Votes:
0
Category:
ontap-9
Specialty:
nas
Last Updated:

Applies to

  • ONTAP 9
  • NFS
  • AWS S3Bucket
  • FabricPool
  • Cloud Volumes ONTAP (CVO)
  • Cloud Tiering (Object store)

Issue

  • NFS client tries read the contents of an old file from an NFS export on a CVO instance, but the read operation hangs indefinitely 
  • Packet trace shows that 
    • ONTAP sends Error: NFS3ERR_JUKEBOX
  • NFS export is to a volume, which is on an aggregate, which is tiering cold data to an object store, which is unavailable
    • ::> vol show -junction-path /Share -fields aggregate,node
      vserver  volume aggregate   node
      -------- ------ ----------- -------
      svm1     Share  aggr_data_1 node-01
    • ::> object-store show
        (storage aggregate object-store show)
      Aggregate      Object Store Name Availability   Mirror Type
      -------------- ----------------- -------------  -----------
      aggr_data_1    S3Bucket          unavailable    primary
  • The object store is unavailable due to  Bad request
    • ::> object-store show -instance
        (storage aggregate object-store show)
      
                                         Aggregate Name: aggr_data_1
                ONTAP Name for this Object Store Config: S3Bucket
                                        UUID of the Bin: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
                                             Bin Number: 1
                       Availability of the Object Store: unavailable
                 Reason why Object Store is Unavailable: Bad request
                       Object Store Config Id in Dblade: 1
                               S3 Client Handle in Cbin: 0
        SMDB Error for Partially Failed get() or next(): -
                                   Bin is being Created: false
                                        Is Bin Deleted?: false
                                  UUID of the Aggregate: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
                 UUID of the Object Store Configuration: zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz
                      Type of the Object Store Provider: AWS_S3
                             Cached Value of Used Space: 45.24TB
                             License Space Used Percent: -
            Threshold for Reclaiming Unreferenced Space: 20%
      Aggregate Fullness Threshold Required for Tiering: 50%
                                    Name Format Version: 1
                               Object Store Mirror Type: primary
           This object store is in mirror degraded mode: -
      Force Tiering with no Mirror in a MetroCluster Configuration: false
      Allow a degraded cloud mirror to be swapped with an unavailable primary object store: -
                           Pending Operation on the Bin: none
      True if this object store is the primary destination for IO: true
       The name of the Cluster to which the bin belongs: cluster_name
                               Modify Count for the Row: 0
  • Object store profiler fails due to Invalid token
    • ::*> storage aggregate object-store profiler start -node node-01 -object-store-name S3Bucket
      
      Warning: This is a CPU intensive command. It is recommended to run this command when the system is under 50% CPU utilization. This command will issue 2500 PUT requests for a time period of
               60 seconds, followed by maximum GET requests to the object store for a time period of 180 seconds. This may result in additional charges to your object store account. Do you want to
               continue?
      Do you want to continue? {y|n}: y
      
      Warning: Object store profiler tests the connection to the object store and is not reflective of FabricPool PUT throughput at steady state.
      Error: command failed: Initializing the object store profiler on node "node-01" for object store "S3Bucket" failed. Reason: Invalid token. Wait a few minutes and try the
             command again.
      
  • Node's cloud credential is expired
    • ::*> object-store ontap-cloud credentials show
        (storage aggregate object-store ontap-cloud credentials show)
      Node           IAM Role                                 Credentials Expiry
      -------------- ---------------------------------------- ----------------------
      node-02
                     fabric-pool-resources-zzzzzzzz-zzz-IamInstanceRole-zzzzzzzzzzzz
                                                              2025-10-27T21:59:25Z
      node-01
                     fabric-pool-resources-zzzzzzzz-zzz-IamInstanceRole-zzzzzzzzzzzz
                                                              2025-10-20T12:52:41Z
      2 entries were displayed. 
      
  • EMS contains object.store.unavailable
    • [?]  Mon Oct 20 21:05:06 -0400 [node-01: OscLowPriThreadPool: object.store.unavailable:EMERGENCY]: Unable to connect to the object store "S3Bucket" from node aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa. Reason: Bad request.
    • [?]  Tue Oct 21 01:10:30 +0000 [node-01: wafl_junkd_exempt: object.store.unavailable:EMERGENCY]: Unable to connect to the object store "S3Bucket" from node aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa. Reason: Bad request.

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.