ONTAP volume move fails due to object store configuration mismatch

Applies to

Issue

• A volume move in a FabricPool configuration fails in the setup state due to the object store configuration being modified:

::> volume move show -vserver svm1 -volume vol1
Vserver Name: svm1
Volume Name: vol1
Actual
Completion Time: Thu Oct 27 08:42:04 2022
Bytes Remaining: -
Destination Aggregate:
aggr1_n6
Detailed Status: Failed to move volume "vol1" in Vserver "svm1" because
object store configurations are modified on source or destination aggregate. Try the operation again.
Volume Move failed in the
Setup State
Estimated Time of Completion: -
Managing Node: node6
Percentage Complete: -
Move Phase: failed
Estimated Remaining Duration: -
Replication Throughput: -
Duration of Move: 00:00:12
Source Aggregate:
aggr1_n1
Start Time of Move: Thu Oct 27 08:41:52 2022
Move State: failed
Is Source Volume Encrypted: false
Encryption Key ID of Source Volume:
Is Destination Volume
Encrypted: false
Encryption Key ID of Destination Volume

• In the EMS logs:

[cluster03-01: mgwd: mgmtgwd.jobmgr.jobcomplete.failure:info]: Job "Volume Move" [id 294944] (Move "vol1" in Vserver "svm1" to aggregate "aggr1_n6") completed unsuccessfully: Failed to move volume "vol1" in Vserver "svm1" because object store configurations are modified on source or destination aggregate. Try the operation again. Volume Move failed in the Setup State (1).

• In the mgwd logs we see Access Denied and Cannot verify availability of the of the object store during the aggregate reaper job:

[kern_mgwd:info:2475] 0x8275e3d00: 86040000015c0edb: ERR: MGWD::aggr_object_store_config_iterator: object_store_config_ksmf_op:src/tables/aggr_object_store_config.cc:2499 object_store_config_ksmf_iterator::create() returned error (701:15): "Access denied". node=cluster-04, config_id=2, check_connection=1, use_iam_role=00000001b.00303e1a 09c27d77

[kern_mgwd:info:2475] 0x8275e3d00: 86040000015c0edb: ERR: AGGR::compositeAggrReaper: populate_object_store_config:src/compositeAggrReaperJob.cc:384 object_store_config_ksmf_op(..., create, ..., true, true) returned error (701:15): "Cannot verify availability of the object store from node cluster-04. Reason: Access denied.". bin-uuid=38f6fe1d-2b59-11ec-9cce-00a09865021f, config-id=2

• From the object store cluster we see there are still objects in the bucket:

::*> vserver object-store-server bucket show -bucket s3-bucket_name -instance
Vserver: s3_svm
  Bucket            :s3-bucket_name
  Uuid              :d4bc2c69-ff6d-11eb-87b7-d039ea2bc4a4
  Volume            :fg_oss_1629213225
  Size              :173TB
  Logical Used Size :44.21TB
  Object Count      :11476897
  Encryption        :false
  Comment           :
  Qos Policy Group  :-

• From the On-prem cluster collect:

::*> storage aggregate object-store show-freeing-status -aggregate-name <aggr_name>