Swift PUT API to StorageGRID sometimes takes much longer time than usual

• StorageGRID
• Swift PUT API (PutObject)

• Swift PUT API which completes within 1 minute sometimes take over 45 minutes
• While performing PUT API, running two parallel executions took twice as long as running a single execution:
  • Single execution
    • 14:09:25 (JST) – 14:10:10 (JST) → 45 seconds
      PUT AUTH_60ed935cfbb048e79b1e7xxxxxxx/test/test.zip
      x-ntap-sg-trace-id: f6d5d5f0axxxxxx
      File size: 1.6GB
  • Two parallel executions
    • 14:26:07 (JST) – 14:28:08 (JST) → 2 minutes 1 second
      PUT AUTH_60ed935cfbb048e79b1e7xxxxxxx/test/test.zip
      x-ntap-sg-trace-id: 2336cebf8xxxxxx
      File size: 1.6GB
    • 14:26:07 (JST) – 14:27:46 (JST) → 1 minute 39 seconds
      PUT AUTH_60ed935cfbb048e79b1e7xxxxxxx/test2/test.zip
      x-ntap-sg-trace-id: 10f9e58xxxxxx
      File size: 1.6GB
• Not all Swift PUT API s are affected
• No error returned to client and StorageGRID
• StorageGRID does not indicate any errors and alerts
• audit.log on admin node indicates WPUT operation takes long time

[AUDT:[RSLT(FC32):SUCS][CNID(UI64):<ID>][TIME(UI64):2615772577][SAIP(IPAD):"<IP>"] .. [ATYP(FC32):WPUT][ANID(UI32):12603831][AMID(FC32):SWRQ][ATID(UI64):7580014692295726309]]