Long running cluster backups and MGWD restarts due to full root volume

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Views:: 257

Visibility:: Public

Votes:: 0

Category:: ontap-9

Specialty:: core

Last Updated:

Applies to

ONTAP 9

Issue

EMS logs MGWD restart emergency.

Sat Jun 19 08:17:40 -0600 [cluster01: spmd: spm.mgwd.process.exit:EMERGENCY]: Management Gateway (mgwd) subsystem with ID 2517 exited as a result of signal normal exit (0). The subsystem will attempt to restart.

MGWD logs indicate restart is due to out of space.

0000001f.008cfdbe 005ae5d0 Sat Jun 19 2021 08:17:38 -06:00 [kern_mgwd:info:2517] MDB: error: PANIC: No space left on device

LIFs on the node were non-functioning and are migrated to partner node.
On checking space usage, the root volume is nearly full with high snapshot space usage.

Filesystem kbytes used avail capacity Mounted on /vol/vol0/ 347277596 322787716 24489880 93% /vol/vol0/ /vol/vol0/.snapshot 18277768 175605520 0 961% /vol/vol0/.snapshot

A temporary snapshot is found on the root volume.

Volume vol0 working... date name ------------ -------- Jun 19 10:15 .tmp.bk.cluster01.8hour.5.1624119300 Jun 19 08:00 hourly.0 Jun 19 00:00 nightly.0 Jun 18 20:00 hourly.1 Jun 18 16:00 hourly.2 Jun 18 12:00 hourly.3

Cluster backups were long running during time and there is a large backup.

cluster01::> system configuration backup show Node Backup Name Time Size --------- ----------------------------------------- ------------------ ----- node01 cluster01.8hour.2021-06-18.18_15_00.7z 06/18 19:15:00 35.87GB node01 cluster01.8hour.2021-06-18.10_15_00.7z 06/18 11:15:00 43.37MB node01 cluster01.8hour.2021-06-19.10_15_00.7z 06/19 11:15:00 42.08MB node01 cluster01.daily.2021-06-17.00_10_00.7z 06/17 01:10:00 42.62MB node01 cluster01.daily.2021-06-18.00_10_04.7z 06/18 01:10:04 42.48MB node01 cluster01.weekly.2021-06-06.00_15_00.7z 06/06 01:15:00 77.41MB node01 cluster01.weekly.2021-06-13.00_15_00.7z 06/13 01:15:00 39.36MB node02 cluster01.8hour.2021-06-18.02_15_00.7z 06/18 03:15:00 42.74MB node02 cluster01.8hour.2021-06-18.10_15_00.7z 06/18 11:15:00 43.37MB node02 cluster01.8hour.2021-06-19.10_15_00.7z 06/19 11:15:00 42.08MB node02 cluster01.daily.2021-06-16.00_10_00.7z 06/16 01:10:00 42.50MB node02 cluster01.daily.2021-06-17.00_10_00.7z 06/17 01:10:00 42.62MB node02 cluster01.daily.2021-06-18.00_10_04.7z 06/18 01:10:04 42.48MB node02 cluster01.weekly.2021-05-30.00_15_00.7z 05/30 01:15:00 122.0MB node02 cluster01.weekly.2021-06-06.00_15_00.7z 06/06 01:15:00 77.41MB node02 cluster01.weekly.2021-06-13.00_15_00.7z 06/13 01:15:00 39.36MB 16 entries were displayed.