- Data ONTAP operating in 7-mode
Perfstat8 can have issues collecting against a cluster with one or more nodes under CPU pressure. This can result in failed Perfstats or Perfstats with partial data, primarily missing the stats * outputs. Generally, an error similar to the following is reported in the command window.
[03-24-2016 23:03:00] ERROR 192.168.1.1 Error in sending command to the controller
[03-24-2016 23:09:45] ERROR 192.168.1.1 No activity for more than 5 minutes. Giving up...
This can be reported against multiple commands, but is primarily a concern against the
stats stopcommands that contain all stats related counters. By default, Perfstat is set to collect 560+ commands, not including the different subsections of
stats start. Perfstat can use preset files to reduce the amount of data it collects and reduces the time spent on output for the
stats stopcommand, in turn lowering the delta per iteration and success of collecting valid data.
There can also be errors related to SSH communications for multiple reasons, including network issues, CPU contention, etc. These errors generally appear as follows:
[06-23-2016 10:32:23] ERROR 192.168.1.1 Error in sending command to the controller
[06-23-2016 10:32:23] ERROR 192.168.1.1 Lost SSH connection. Reconnecting...
Before continuing with these steps, ensure the following:
- The most current version of perfstat is being used, perfstat 8
- The host running Perfstat isn't a VM hosted on the affected controller
- Not connected over VPN, and ensure
- The host can SSH to the cluster management IP
- Collection host is on the same network as the cluster, minimize hops
Also, run a trace route from the host to the cluster management IP and ensure there are not too many hops or routes getting stuck.
Tracing route to 192.168.1.1 over a maximum of 30 hops 192.168.1.3
1 4 ms 3 ms 3 ms 192.168.1.2
2 1 ms 1 ms 1 ms
3 1 ms 1 ms 1 ms 192.168.1.4
4 7 ms 3 ms 3 ms 192.168.1.5
5 93 ms 96 ms 94 ms 192.168.1.6
6 92 ms 92 ms 92 ms 192.168.1.7
7 92 ms 92 ms 93 ms 192.168.1.8
8 115 ms 92 ms 92 ms 192.168.1.9
9 92 ms 92 ms 92 ms 192.168.1.10
10 92 ms 92 ms 92 ms 192.168.1.11