High CPU utilization in HostOS with Docker Swarm
Applies to
- ONTAP 8.2 or newer
- Docker Swarm
Issue
- High CPU is visible on one or more nodes
- Running
node -run <node_name> -command top
against the node showsvldb
with highWCPU%
Example:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
6685 root 101 47 0 290M 109M uwait 1 ??? 410.99% vldb <-----------
1709 root 231 66 0 1092M 449M RUN 7 ??? 141.06% mgwd
1929 root 1 8 - 0K 64K WAIT 3 1141.4 8.79% NwkThd_04
3308 root 1 12 - 0K 64K RUN 0 366.6H 8.69% wafl_exempt00
1927 root 1 8 - 0K 64K WAIT 3 1142.8 8.64% NwkThd_02
3312 root 1 12 - 0K 64K RUN 0 366.6H 8.54% wafl_exempt04
3310 root 1 12 - 0K 64K RUN 1 366.7H 8.50% wafl_exempt02
system node run -node <node_name> rdfile /etc/log/mlog/mgwd.log
contains entries similar to the following, whereaaa.bbb.ccc.ddd
is an IP address belonging to one or more Docker Swarm servers
Example:
Tue Jul 14 2020 13:28:30 +02:00 [kern_mgwd:info:1709] ontapi :: aaa.bbb.ccc.ddd :: docker :: volume-get-iter :: This request exceeds the configured session limit (10) for your Vserver ("<VSERVER_NAME>"). :: Throttled Tue Jul 14 2020 13:28:31 +02:00 [kern_mgwd:info:1709] ontapi :: aaa.bbb.ccc.ddd :: docker :: volume-get-iter :: This request exceeds the configured session limit (10) for your Vserver ("<VSERVER_NAME>"). :: Throttled Tue Jul 14 2020 13:28:31 +02:00 [kern_mgwd:info:1709] ontapi :: aaa.bbb.ccc.ddd :: docker :: volume-size :: This request exceeds the configured session limit (10) for your Vserver ("<VSERVER_NAME>"). :: Throttled Tue Jul 14 2020 13:28:31 +02:00 [kern_mgwd:info:1709] ontapi :: aaa.bbb.ccc.ddd :: docker :: volume-get-iter :: This request exceeds the configured session limit (10) for your Vserver ("<VSERVER_NAME>"). :: Throttled Tue Jul 14 2020 13:28:31 +02:00 [kern_mgwd:info:1709] ontapi :: aaa.bbb.ccc.ddd :: docker :: volume-size :: This request exceeds the configured session limit (10) for your Vserver ("<VSERVER_NAME>"). :: Throttled Tue Jul 14 2020 13:28:32 +02:00 [kern_mgwd:info:1709] ontapi :: aaa.bbb.ccc.ddd :: docker :: volume-get-iter :: This request exceeds the configured session limit (10) for your Vserver ("<VSERVER_NAME>"). :: Throttled Tue Jul 14 2020 13:28:32 +02:00 [kern_mgwd:info:1709] ontapi :: aaa.bbb.ccc.ddd :: docker :: volume-size :: This request exceeds the configured session limit (10) for your Vserver ("<VSERVER_NAME>"). :: Throttled