CONTAP-311454: Latency and slowness due to high traffic from clients

Issue

• The following EMS message is seen and high utilization is seen on a network context:

Mon Sep 09 10:23:29 +0000 [node1: kernel: net_cg_utilThresholdExceed_1:notice]: params: {'connection_group': '20', 'current_utilization': '98', 'configured_threshold': '95'}
Cluster::> set -privilege diagnostic; statistics show -object nwk_ctx -counter utilization -raw -tab -filter "utilization>30" -node node1; set -privilege admin
node1: nwk_ctx: 10/22/2024 10:06:06
instance
name utilization
-------- -----------
ctx_09 98

• Latency is seen which is high from users, as shown in the Network column of the below output:
  Cluster::> qos statistics volume latency show -vserver svm1 -volume vol1
  Workload ID Latency Network Cluster Data Disk QoS Max QoS Min NVRAM Cloud FlexCache SM Sync VA AVSCAN
  
• ------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
  -total- - 744.00us 328.00us 0ms 392.00us 4.00us 0ms 0ms 20.00us 0ms 0ms 0ms 0ms 0ms
  vol1-wid12503 12503 93.25ms 93.18ms 0ms 69.00us 0ms 0ms 0ms 1.00us 0ms 0ms 0ms 0ms 0ms
  

- ifstat -a -v from the node shell shows the RSS utilization (9.14.1 or higher) on one Connection Group by a higher value of packets:
Cluster::> run -node node1 ifstat -a -v
...
Queue 7
CG ID: 8 | Rx packets: 361m | Rx bytes: 1065g
Tx packets: 472m | Tx bytes: 4039g
...
Queue 8
CG ID: 9 | Rx packets: 35967m | Rx bytes: 50080g <---higher amount of packets
Tx packets: 3206m | Tx bytes: 21002g
...
Queue 9
CG ID: 10 | Rx packets: 1063m | Rx bytes: 3071g
Tx packets: 426m | Tx bytes: 5950g