StorageGRID EC rebalance failing after a few minutes
Applies to
StorageGRID 11.6.0.4
Issue
When running EC rebalance it fails after a few minuites
root@mdebglgrid-admin:~ # rebalance-data
status
==============================================================================
Job ID :
14639721269233092077
Site : mdebgl-rz3
State : Failure
Percentage
: Unknown
Start Time : 2022-12-16 12:04:43 UTC
End Time : 2022-12-23 12:06:46 UTC
Check the rebalance-data.log and the bycast.log from the EC leader
Dec 30 07:05:29 mdebglgrid-node10 ADE: |21664688 0784005611 ECJM CSRT 2022-12-30T07:05:29.685124| ERROR 1067 PROC: Exception: /build/src/modules/ErasureCoding/EC_JobManager_Module/SiteRebalanceJob.cc(346): Throw in function std::vector<VcsMoveInfo> erasurecoding::SiteRebalanceJob::getMoveRecommendations(byc::GroupID)#012Dynamic exception type: boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::runtime_error> >#012std::exception::what: ENFORCE failed: !"Exhausted retry limit or retry time for getting move recommendations"#012
7