StorageGRID multipart uploads fail intermittently with Internal server 500

• Application uploads multipart objects to the GRID and some of those multipart fails with:

2022-12-19 04:55:52,498 ERROR com.xxxx.Operation [ http-nio-8083-exec-112] Failed to upload object data/xxxx/source_files/xx457644262201 to bucket <bucketname> due to Unable to complete multi-part upload. Individual part upload failed : Unable to execute HTTP request: Timeout waiting for connection from pool

• StorageGRID bycast.log reports:

/var/local/log/bycast-err.log.1:Nov 19 11:08:01 xx-sn004 ADE: |12919189 0180200777 S3RQ POPD 2022-11-19T11:08:01.565690| ERROR 1880 6ce1c66ab514a1e4 S3RQ: S3 request error: connection=1668855670467520, Resource=/xx/source_files/xx470563752200?uploadId=6d4gicvWiWYH_0BGhau_qandmXVlP4CYLD_0hJPTbkZyICljlHzl4eYZYw&partNumber=5081, HTTP Status Code=500, ErrorMsg=InternalError, ErrorType=Client CustomErrorMessage={<none>}, Details={asyncPop error: no producer}