Backup logs from aborted and/or resumed NDMP operations can cause an ONTAP node's root volume to fill, possibly leading to node panics
Applies to
- ONTAP 9
- Network Data Management Protocol (NDMP) operations, such as
ndmpcopy
Issue
- Rapid increase in the used size of a single node's root volume. This can be seen by running the following command periodically:
cluster1::> volume show -vserver cluster1-01
Vserver Volume Aggregate State Type Size Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
cluster1-01 vol0 aggr0 online RW 442.4GB 407.6GB 7%
(Using a node name as the -vserver parameter will return that node's root volume)
- The backup log located at
/mroot/etc/log/backupis filled with messages similar to the following:
Tue Mar 27 00:11:36 EDT 2018 /svm1/vol1 Log_msg (Flush DIRNET for BKP ID=248, type=3 interrupted while waiting for min inflight. Error = Interrupted system call.
The simplest way to access the backup log is through the Service Processor Infrastructure (SPI) interface by clicking the logs link. See KB: How to manually collect logs and copy files from a clustered Data ONTAP storage system (under "Option 1") for assistance working with the SPI.
- Affected node may panic with messages similar to the following:
Process vldb unresponsive for 631 seconds in process nodewatchdog onrelease 9.2P1 (C)
Note: This panic may be caused by many other issues. This panic alone does not indicate the issue outlined here; make sure to check the node's root volume status as well as the contents of the backup log.