Skip to main content

NetApp_Insight_2020.png 

NetApp Knowledgebase

How to collect performance data when Perfstat8 fails by using preset files and manual data collection

Views:
911
Visibility:
Public
Votes:
0
Category:
data-ontap-8
Specialty:
perf
Last Updated:

Applies to

  • ONTAP
  • Data ONTAP operating in 7-mode

Description

This article is intended to be used only if Perfstat8 is failing to collect performance data. Please use the standard Perfstat collection methods if possible.

Perfstat8 can have issues collecting against a cluster with one or more nodes under CPU pressure. This can result in failed Perfstats or Perfstats with partial data, primarily missing the stats * outputs. Generally, an error similar to the following is reported in the command window.

[03-24-2016 23:03:00]   ERROR 192.168.1.1   Error in sending command to the controller
[03-24-2016 23:09:45]   ERROR 192.168.1.1   No activity for more than 5 minutes. Giving up...


This can be reported against multiple commands, but is primarily a concern against the stats start and stats stop commands that contain all stats related counters. By default, Perfstat is set to collect 560+ commands, not including the different subsections of stats start. Perfstat can use preset files to reduce the amount of data it collects and reduces the time spent on output for the stats stop command, in turn lowering the delta per iteration and success of collecting valid data.

There can also be errors related to SSH communications for multiple reasons, including network issues, CPU contention, etc. These errors generally appear as follows:

[06-23-2016 10:32:23]   ERROR   192.168.1.1   Error in sending command to the controller
[06-23-2016 10:32:23]   ERROR   192.168.1.1   Lost SSH connection. Reconnecting...


Before continuing with these steps, ensure the following:
  • The most current version of perfstat is being used, perfstat 8
  • The host running Perfstat isn't a VM hosted on the affected controller
  • Not connected over VPN, and ensure
  • The host can SSH to the cluster management IP
  • Collection host is on the same network as the cluster, minimize hops


Also, run a trace route from the host to the cluster management IP and ensure there are not too many hops or routes getting stuck.

Tracing route to 192.168.1.1 over a maximum of 30 hops

  1     4 ms     3 ms     3 ms  192.168.1.2
  2     1 ms     1 ms     1 ms  
192.168.1.3
  3     1 ms     1 ms     1 ms   192.168.1.4
  4     7 ms     3 ms     3 ms   192.168.1.5
  5    93 ms    96 ms    94 ms   192.168.1.6
  6    92 ms    92 ms    92 ms   192.168.1.7
  7    92 ms    92 ms    93 ms   192.168.1.8
  8   115 ms    92 ms    92 ms   192.168.1.9
  9    92 ms    92 ms    92 ms   192.168.1.10
 10    92 ms    92 ms    92 ms   192.168.1.11

Trace complete.

 

CUSTOMER EXCLUSIVE CONTENT

Registered NetApp customers get unlimited access to our dynamic Knowledge Base.

New authoritative content is published and updated each day by our team of experts.

Current Customer or Partner?

Sign In for unlimited access

New to NetApp?

Learn more about our award-winning Support