Skip to main content
NetApp Knowledge Base

FAQ: NDMP File history

  1. Last updated
  2. Save as PDF
Views:
4,537
Visibility:
Public
Votes:
5
Category:
ndmp
Specialty:
dp
Last Updated:

​ ​ ​  

Applies to

  • ONTAP 9
  • Data ONTAP 8 7-mode
  • NDMP

Answer

About
What is NDMP file history and how is it communicated?
  • File history is generated during an Network Data Management Protocol (NDMP) backup of a volume hosted on NetApp storage using the dump engine
  • File history enables a backup application or Data Management Application (DMA) to build an index database of all the files in a backup
  • This database enables users to locate which backup contains a particular file, when that file was modified, and other useful metadata
  • The purpose of collecting and storing file history is to:
    • Provide a human-readable user interface to backup data
    • Provide a basis for Direct Access Recovery (DAR). DAR allows a DMA to access files / directories directly on tape without having to traverse the entire backup. This allows for quicker file and directory recovery operations.
  • How file history is communicated:
  1. During a backup, ONTAP's dump process generates file history in phase 3 and 4 of a backup. For more information on dump phases, see Network Data Management Protocol (NDMP) dump phases description.
  2. dump communicates file history information to the NDMP Server running in ONTAP. This communication is internal to the storage controller running the backup.
  3. The NDMP Server in ONTAP communicates the file history to the backup application / DMA through the NDMP control connection over the network.
  4. The backup application / DMA receives the file history from the network, ingests the data, and writes it to the file history index database.

What is NDMP file history and how is it communicated?

What is NDMP file history back pressure and what impact does it have?
  • Generating, communicating, and ingesting file history will always add some overhead to a backup.
    • A backup typically always runs faster with file history disabled even if there are no other performance issues.
    • A bottleneck in one step of file history communication can trigger latencies downstream.
  • Due to the way NDMP and dump work together, latency in file history delivery or ingestion can cause a slow-down in the overall backup performance.
    • In other words, dump cannot continue writing data to the backup stream until the associated file history is completely ingested and acknowledged by the backup application / DMA.

What is NDMP file history back pressure and what impact does it have?

How is NDMP file history back pressure identified?
  • ONTAP may warn of possible file history backpressure in the backup log, with messages such as:

Warning (Total Dir to FH time spent is greater than 15 percent of phase 3 total time. Please verify the settings of backup application and the network connectivity)

Warning (Total Node to FH time spent is greater than 15 percent of phase 4 total time. Please verify the settings of backup application and the network connectivity)

  • When such warnings are received, it is possible to manually calculate file history statistics, confirming the presence of file history backpressure.

Note: Double-check the units (seconds or milliseconds) to ensure accurate calculations

  1. Check the dump backup log located in /etc/log/backup on the node which hosts the volume being backed up. The interesting logs print after a backup has completed successfully. An aborted or incomplete backup will not provide the logs sufficient to diagnose this problem.
  2. Calculate the duration of phase 3 and phase 4 of the backup in question. In the following example, phase 3 was 20 minutes long and phase 4 was 5 hours long:

dmp Thu Feb 27 12:01:36 CET 2020 /vol/NdmpBackup/(0) Phase_change (III)

dmp Thu Feb 27 12:21:36 CET 2020 /vol/NdmpBackup/(0) Phase_change (IV)

dmp Thu Feb 27 17:21:36 CET 2020 /vol/NdmpBackup/(0) Phase_change (V)

  1. Find the total Dir to FH entry time stats and Node to FH entry time stats for the backup which are presented in milliseconds in ONTAP 9.6 and earlier and seconds in ONTAP 9.7 and later. For example:

dmp Thu Feb 27 17:21:52 CET 2020 /vol/NdmpBackup/(0) Log_msg (Dir to FH entry time stats (msec) numEntries: 2000 min: 0 max: 1526 avg: 5 tot: 15236)

dmp Thu Feb 27 17:21:52 CET 2020 /vol/NdmpBackup/(0) Log_msg (Node to FH Entry time stats (msec) numEntries: 4000 min: 0 max: 1599 avg: 7 tot: 85569)

Note: Make note of the units as these values will be milliseconds "(msec)" in ONTAP 9.6 and earlier and seconds "(sec") in ONTAP 9.7 and later.

  1. If the total Dir to FH entry time is 15% or more of total phase 3 time, this is considered file history backpressure in phase 3. In the above example, 15.236 seconds is only ~1% of the 20 minute phase 3 time, so this is not considered backpressure.
  2. If the total Node to FH entry time is 15% or more of total phase 4 time, this is considered file history backpressure in phase 4. In the above example, 85.569 seconds is <1% of the 5 hour phase 4 time, so this is also not considered backpressure.

Note:

  • In releases prior to ONTAP 9.6, these warnings can be inaccurate due to BUG 1177614.
    • In ONTAP 9.6 and prior, it is recommended to use the above manual calculations to confirm file history backpressure.

How is NDMP file history back pressure identified?

What are the common root cause and solutions of NDMP file history back pressure?
  1. File history ingestion at the backup application / DMA common causes and solutions:
    • Cause #1: Resource contention on the server or VM hosting the backup application / DMA as seen in general performance statistics from the hosting OS.
      • Solution #1: Add additional resources to the server / VM hosting the DMA or reduce resource contention to allow for faster ingestion and indexing.
    • Cause #2: Poor performance writing to the file history index database.
      • Solution #2:
        • This is typically due to poor performance of the underlying storage or configuration / sizing issues with the DMA software.
        • Ensure the storage hosting the file history index database has adequate performance.
        • If required, contact your DMA vendor for sizing assistance or other tweaks to allow for faster indexing of file history.

  2. NDMP Server communication to the backup application common cause and solution:
    • Cause: The NDMP control connection is using a latent or lossy network path.
    • Solution:
      • Ensure the network path hosting the NDMP control connection has adequate speed and throughput to support the NDMP control connection and file history messages.
      • Check interface statistics for signs of packet loss or other issues.
      • On a NetApp storage controller, the node-shell ifstatcommand can be used to view interface statistics.
      • Finally, ensure the MTU configured in ONTAP and at the backup application is supported through the entire network path.

What are the common root cause and solutions of NDMP file history back pressure?

Additional Information

Add your text here.

 

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.