Skip to main content
NetApp Knowledge Base

How does Aggregate Deduplication affect SnapMirror?

  1. Last updated
  2. Save as PDF
Views:
3,261
Visibility:
Public
Votes:
1
Category:
snapmirror
Specialty:
dp
Last Updated:

Applies to

  • Snapmirror destination DP volumes
  • Aggregate-level Cross-Volume-Deduplication savings

Answer

  • Cross-Volume-Deduplication at the aggregate-level eliminates duplicate blocks across multiple volumes hosted at the same aggregate
    • These savings are achieved by an aggregate-level background efficiency-scanner that triggers automatically
  • Cross-Volume-Deduplication savings on a Snapmirror source volume (R/W) aggregate are are not retained by Snapmirror transfers on the Destination volume (DP) aggregate by default in ONTAP when the relationship was established
    • Snapmirror maintains inner-volume storage efficiency deduplication savings from the source to the destination over the wire via the LRSE replication engine.
      • This does not require the State of volume efficiency to report Enabled on the DP volume at all (vol efficiency show -fields state can report Enabled or Disabled on the DP volume and it does not matter for the topic of this KB article)
    • After each Snapmirror update with the LRSE engine, an efficiency operation triggers on the DP volume by default. This gains volume level deduplication savings that had not been processed background on source yet. 
      • No dedicated efficiency configuration is required on the destination DP-volume for this to happen
    • The LRSE replication engine is in use by Snapmirror
      • When the destination DP volume has no dedicated efficiency configuration, or
      • When the efficiency configuration of the DP volume uses identical compression settings as the source R/W volume
  • SnapMirror destination clusters can use aggregate deduplication with type-DP Snapmirror destination volumes optionally
    • This requires manual changes for efficiency configuration on the Snapmirror destination DP volumes, because cross-volume-deduplication is disabled by default on individual- or svmdr-DP volumes
    • A one-time manual volume-level deduplication full-scan (-scan-old-data true) is required on a DP volume, when Cross-Volume-Deduplication on a DP volume is enabled AFTER the snapmirror initialize has already completed or the destination DP volume is member of a svmdr relationship
    • Enabling Cross-Volume-Deduplication explicitly on Snapmirror destination DP volumes has no impact to Snapmirror LRSE engine to maintain volume-level storage efficiency savings from the source over the wire and does not require an  efficiency-schedule or -policy on the DP volume
    • To gain extra deduplication savings from Cross-Volume-Deduplication on Snapmirror DP volumes it is not require to set an auto- or scheduled- or threshold-based background operation efficiency policy on the DP volume, as long as the DP volume does not become R/W (snapmirror break).

Additional Information

Parent topic: SnapMirror storage efficiency configurations and behavior

To achieve cross-volume savings on a DP (Snapmirror destination) volume:

  1. Enable cross-volume-efficiency on the DP volume prior snapmirror initialize
  • Enable efficiency on the DP volume:
    volume efficiency on -vserver svm-destination -volume volume_dp
  • Enable cross-volume-background-dedupe on the DP volume:
    volume efficiency modify -vserver svm-destination -volume volume_dp -cross-volume-background-dedupe true
  • Initialize the Snapmirror relationship:
    snapmirror initialize -destination-vserver svm-destination -destination-volume volume_dp
  1. If Snapmirror re-initialize is not possible, a full-scan is required on the DP volume to initialize its block-fingerprint database:
  • This will take longer, because a full-scan takes a long time and can only consume limited CPU, so if possible prefer option 1
  • Enable efficiency on the DP volume:
    volume efficiency on -vserver svm-destination -volume volume_dp
  • Enable cross-volume-background-dedupe on the DP volume:
    volume efficiency modify -vserver svm-destination -volume volume_dp -cross-volume-background-dedupe true
  • Run a full-scan on the DP volume:
    volume efficiency start -vserver svm-destination -volume volume_dp -scan-old-data true
  • Wait for the full-scan to complete:
    volume efficiency show -vserver svm-destination -volume volume_dp -scan-old-data true -fields op-status
  1.  For SVM-DR, option 1 is not possible, because the volumes are only created during snapmirror initialize.
  • Use the same steps as in option 2, but make use of advanced command vserver config override for the non-show command:

set advanced

vserver config override -command "volume efficiency on -vserver svm-destination -volume volume_dp"

vserver config override -command "volume efficiency modify -vserver svm-destination -volume volume_dp -cross-volume-background-dedupe true"

vserver config override -command "volume efficiency start -vserver svm-destination -volume volume_dp -scan-old-data true"

WARNING for Options 2. and 3.

When XDP is in use and there is no difference in the efficiency compression algorithm configuration between source RW and destiantion DP volumes, the Snapmirror relationship is based on the Logical Replication Engine with Storage Efficiency (LRSE). LRSE is capable to maintain source efficiency over-the-wire to the destination. The LRSE capabilities require specific metadata to be transferred between source and destination, the so called datawarehouse file (DWH file). For this to be possible, every Snapmirror update will pause the efficiency full scan operation shortly. When the Snapmirror schedule is frequent and the data transferred by Snapmirror is large - the efficiency fullscan might not be able to gather all the data that is appended to the DWH file unless the next Snapmirror update starts. Such situation repeating over a longer period of time can cause logical capacity usage on the DP volume to raise uncontrolled, unless the snapmirror relationship is not quiesced while the efficiency full scan operation can run its gathering stage to completion. The logical capacity, for example:
 

MyCluster::> volume show-footprint MyVolume
      Vserver : MySVM
      Volume  : MyVolume
      Feature                                         Used       Used%
      --------------------------------             ----------    -----
      Volume Data Footprint                           27.07TB      46%
      Volume Guarantee                                     0B       0%
      Flexible Volume Metadata                        204.8GB       0%
      Deduplication Metadata                          798.5PB          
           Temporary Deduplication                    798.5PB
      Delayed Frees                                   378.4GB       1%
      File Operation Metadata                             4KB       0%
      Total Metadata Footprint                         1013GB       2%
      Total Footprint                                 28.06TB      47%
      Footprint Data Reduction                        12.03TB      20%
           Data Compaction                            12.03TB      20%
      Effective Total Footprint                       16.03TB      27%

 

 

With either option:

  • Cross-volume-savings are achieved after next aggregate efficiency background operation had triggered.
    • Check output from aggregate efficiency cross-volume-dedupe show to determine last operation end or current operational state
    • An aggregate efficiency background operation is triggered automatically upon a certain percentage of block-changes within the aggregate
    • An aggreate efficiency background operation can get trigegred manually using: aggregate efficiency cross-volume-dedupe start
  • An efficiency operation schedule for the DP volume efficiency operation is not required when LRSE (logical replication with storage efficiency) is in use.
  • With LRSE in use, every snapmirror update will trigger an efficiency background operation (only visible in sis.log) to achive cross volume savings.
  • The per-volume savings from the source-volume are maintained over from primary to secondary with XDP relatioonships using storage efficiency when there is no secondary compression enabled on the DP volume. Additional details:

 

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.