Slow file moves across directories in FlexGroups

Applies to

Issue

• Poor performance is seen on a FlexGroup.
• A file moves take many minutes or hours to occur.
  • Example: A file rename takes from 5:46 PM to 12:13 AM the next morning.

    • [2023-10-10 17:46:29,751] {bash_operator.py:126} INFO - 2023.10.10D22:46:29.751918000 [info][0] Moving :/mnt/home/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8$/file1_lastModified
      -> :/mnt/home/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir/lastModified
      [2023-10-11 00:13:48,255] {bash_operator.py:126} INFO - 2023.10.11D05:13:48.255446000 [info][0] Executed successfully
      

  • High latency is seen in qos statistics volume latency show

    • Cluster::> qos statistics volume latency show -vol vol1 -vserver svm1
      Workload            ID    Latency    Network    Cluster       Data       Disk    QoS Max    QoS Min      NVRAM      Cloud  FlexCache    SM Sync         VA     AVSCAN
      --------------- ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
      -total-              -     8.58ms     2.99ms   247.00us     5.32ms     2.00us        0ms        0ms    28.00us        0ms        0ms        0ms        0ms        0ms
      fg1-wid50693     50693     35.39s     16.59s   628.87ms     18.18s        0ms        0ms        0ms     1.80ms        0ms        0ms        0ms        0ms        0ms
      

  • High rewind count (millions in a few minutes) are seen for NFS Replay Cache which contributes to the high Network Processing latency seen above:

    • Cluster::> set -privilege diagnostic; statistics start -object rw_ctx -counter nfs_rewinds
      (wait 5 minutes)
      Cluster::*> statistics show
      Object: rw_ctx
      Instance: rw_ctx
      Start-time: 10/25/2023 15:18:01
      End-time: 10/25/2023 15:22:44 
      Elapsed-time: 283s
      Scope: node2
      
          Counter                                                     Value
          -------------------------------- --------------------------------
          nfs_rewinds                                                     -
                               Credentials                                0
      ...
                              Replay Cache                         12697850 <---
      ...
      Cluster::*> statistics stop