CONTAP-329751: Running API commands such as the nfs connected-clients API excessively causes user or ONTAP slowness
Issue
- ESXi APDs may be seen or other client latency issues.
- The qos statistics volume latency show command shows high Network latency.
- High CPU in Exempt domain per sysstat -M 1 (see https://kb.netapp.com/on-prem/ontap/...PU_utilization)
- Looking at the process using the most CPU, it is svc_queue_thread with the most CPU total.
Cluster::> set diag Cluster::*> statistics start -object process -counter pct_cpu -node node1 Warning: The existing sample will be overwritten for sample-id: :sample_310. Do you want to continue? {y|n}: y Statistics sample deleted, for sample-id: sample_310 Statistics collection is being started for sample-id: sample_310 Cluster::*> statistics show -filter "pct_cpu>5" -fields value object instance counter value ------- ------------------ ------- ----- .. process "svc_queue_thread" pct_cpu 22% process "svc_queue_thread" pct_cpu 21% ... Cluster::*> statistics stop
- API calls are being driven for the REST API equivilent of nfs connected-clients show at a high rate (20/minute).
10.1.2.3 [Mon Oct 07 03:38:57.884998 2024 +0000] "GET /api/protocols/nfs/connected-clients?fields=server_ip%2Cclient_ip%2Cidle_duration&svm.name=svm1&volume.name=vol1 HTTP/1.1" 200 458 510788 - 0 + 127.0.0.1 -
- The sktrace logs show this output:
2024-10-07T03:53:23Z 26130792276755676 [13:0] KSMF_SMF_SVC_NORM: process_request: Table: ksmf_nfs_show_clients is occupying at least half of kSMF threads. Active thread count:25