Restore from Glacier S3 bucket to AVA hung at 0%
Applies to
- AltaVault
- Amazon Glacier S3
Issue
- A restore was initiated from Amazon Glacier S3 for 400GB data.
- The restore hung at 0% for past 3 days.
- Prepopulating job active as identified from the logs:
Jan 19 18:00:26 localhost mgmtd[7823]: [mgmtd.INFO]: Forking then execing binary "/opt/rbt/bin/prepop" with argv "/opt/rbt/bin/prepop -p /tapenfs10/Folder_04.26.2017_07.03/CV_MAGNETIC/V_48093/* -r 1".
Jan 19 18:00:26 localhost prepop[50683]: [prepop.NOTICE] (50683) Prepopulating files matching /tapenfs10/Folder_04.26.2017_07.03/CV_MAGNETIC/V_48093/* using retrieval type: 1
Jan 19 18:00:26 localhost rfsd[50387]: [dataprepop.INFO] (61226) Generated taskid 8049 for Expedited retrieval type
Jan 19 18:00:26 localhost prepop[50683]: [prepop.INFO] (50687) Prepopulating /tapenfs10/Folder_04.26.2017_07.03/CV_MAGNETIC/V_48093/CHUNK_2755954
Jan 19 18:00:27 localhost rfsd[50387]: [dataprepop.INFO] (61205) Adding file /tapenfs10/Folder_04.26.2017_07.03/CV_MAGNETIC/V_48093/CHUNK_2755954 to task 8049
Jan 19 18:00:28 localhost rfsd[50387]: [dataprepop.INFO] (61010) prepop_tasks/8049.data has 245 user files
Jan 19 18:00:28 localhost rfsd[50387]: [dataprepop.INFO] (61010) read pathname /tapenfs10/Folder_04.26.2017_07.03/CV_MAGNETIC/V_48093/CHUNKMAP_TRAILER_2755591 from prepop_tasks/8049.data
- There were 240 files to restore with retrieval type
Expediated
. - 20 tries made to download the xx.pkg from the Glacier, however none of them were successful.
[curl.INFO] (52356) HTTP request forhttps://s3-ap-XXX.amazonaws.com:443/archival/3A/pkg/03FBFFA4F9A3.pkg?restore
[curl.INFO] (52356) (01:03:24.930279): POST /s3-ava-archival/3A/pkg/xx.pkg?restore HTTP/1.1