Getting 'Aborted Exception Status: 471' on object-store tiering

Last updated

May 13, 2025
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 25

Visibility:: Public

Votes:: 0

Category:: cloud-tiering

Specialty:: core

Last Updated:: 5/13/2025, 6:41:17 AM

Applies to

Ontap 9.13.x
StorageGrid

Issue

Slow tiering to object store has been observed.
EMS Logs shows some latency from time to time.

Mon Mar 17 17:05:32 +1100 [NUCFAS04c: OscLowPriThreadPool: wafl.ca.latency.threshold:notice]: Measured latency from cloud (26706 msec) more than threshold (5000 msec) (aggregate: "aggr01_04c_nlsas", aggregate uuid: "e241c052-885c-402f-a31c-0145b392741d", object store: "os01_sgrid01_nucpobs01")
Mon Mar 17 17:15:33 +1100 [NUCFAS04c: OscLowPriThreadPool: wafl.ca.latency.threshold:notice]: Measured latency from cloud (10899 msec) more than threshold (5000 msec) (aggregate: "aggr01_04c_nlsas", aggregate uuid: "e241c052-885c-402f-a31c-0145b392741d", object store: "os01_sgrid01_nucpobs01")
Mon Mar 17 17:25:34 +1100 [NUCFAS04c: OscLowPriThreadPool: wafl.ca.latency.threshold:notice]: Measured latency from cloud (13540 msec) more than threshold (5000 msec) (aggregate: "aggr01_04c_nlsas", aggregate uuid: "e241c052-885c-402f-a31c-0145b392741d", object store: "os01_sgrid01_nucpobs01")
Mon Mar 17 18:33:50 +1100 [NUCFAS04c: OscLowPriThreadPool: wafl.ca.latency.threshold:notice]: Measured latency from cloud (5365 msec) more than threshold (5000 msec) (aggregate: "aggr01_04c_nlsas", aggregate uuid: "e241c052-885c-402f-a31c-0145b392741d", object store: "os01_sgrid01_nucpobs01")

Running below command shows "State: ready" but on "Previous Run Status: aborted-exception" with 471 status

 # volume object-store tiering show -vserver SVM1 -volume Volume1_vault -instance
 Vserver: SVM1
                                   Volume: Volume1_vault
       Node Name: NUCFAS04d
                              Volume DSID: 2836
                           Aggregate Name: aggr02_04d_nlsas
                                    State: ready
                      Previous Run Status: aborted-exception
             Aborted Exception Status: 471
               Time Scanner Last Finished: Mon Mar 17 10:45:18 2025
 Scanner Percent Complete: -
      Time Waiting Scan will be scheduled: -
                           Tiering Policy: none
 Estimated Space Needed for Promotion: -
                        Time Scan Started: -
                   Cloud Retrieval Policy: promote
                 Elapsed Time Scanner Ran: 833476s
          Object Format Revert Scan State: -
space error code encountered in last scan: -
                          Blocks Promoted: 1083928482

No latency seen on qos statistics

NUCFAS04::> qos statistics volume latency show -volume Volume1_vault -vserver SVM1
Workload            ID    Latency    Network    Cluster       Data       Disk    QoS Max    QoS Min      NVRAM      Cloud  FlexCache    SM Sync         VA     AVSCAN
--------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
-total-              -        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms
-total-              -        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms
-total-              -        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms
-total-              -        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms
-total-              -        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms        0ms

No issues seen with ICMP test

# network ping -destination <objectstore_fqdn> -vserver <svm_name> -lif <intercluster_lif> -show-detail true -verbose true
PING cluster1.domain.net.au (100.25.19.6) from 100.25.19.3: 56 data bytes
to 100.25.227.38 64 bytes from 100.25.19.6: icmp_seq=0 ttl=62 time=0.147 ms
to 100.25.227.38 64 bytes from 100.25.19.6: icmp_seq=1 ttl=62 time=0.249 ms
to 100.25.227.38 64 bytes from 100.25.19.6: icmp_seq=2 ttl=62 time=0.162 ms
to 100.25.227.38 64 bytes from 100.25.19.6: icmp_seq=3 ttl=62 time=0.153 ms
to 100.25.227.38 64 bytes from 100.25.19.6: icmp_seq=4 ttl=62 time=0.126 ms

Running "storage aggregate object-store profiler show" shows acceptable throughput but with latency

:*> storage aggregate object-store profiler show
Object store config name: os01_sgrid01_nucpobs01
Node name: NUCFAS04c
Status: Done
Start time: 2/21/2025 16:08:55
Object name prefix: f2a22fdc-f011-11ef-9ca0-d039ea941bad
Object prefix:

Op      Size       Total     Failed             Latency(ms)          Throughput
                                        min       max       avg
-------------------------------------------------------------------------------
PUT     4MB        2500      0         338       3657      832       908.8MB
GET     4KB        1102416   0         2         426       5         95.68MB
GET     8KB        1322422   0         2         434       4         229.4MB
GET     32KB       1205693   0         2         500       5         837.1MB
GET     256KB      489047    0         4         1760      66        2.63GB