Skip to main content

NetApp_Insight_2020.png 

NetApp Knowledgebase

ONTAP/Data ONTAP Log Overview

Views:
2,781
Visibility:
Public
Votes:
1
Category:
data-ontap-8
Specialty:
core
Last Updated:

Applies to

  • ONTAP EMS 
  • Data ONTAP

Answer

What are the logging features of ONTAP/Data ONTAP?

Logging in ONTAP/Data ONTAP

Logs are event-triggered messages ranging in severity that are generated by the clustered Data ONTAP operating system and recorded in flat text files on the cluster. Logs are the primary resource for administrators, NetApp Support, and AutoSupport™ systems to determine and isolate root causes for a wide range of issues.

Logs can be collected, viewed, and forwarded using several different methods.  All logs are stored in /mroot/etc/log and /mroot/etc/log/mlog, including EMS, audit logs, and user space application logs.  Logs in /mroot/etc/log rotate once per week, with a maximum of five rotations before the oldest log is deleted.  Logs in /mroot/etc/log/mlog rotate once per day, with a maximum of 35 rotations before the oldest log is deleted.

Event Management System

The event management system (EMS) is the clustered Data ONTAP messaging facility built on the syslog standard. EMS simplifies the management of cluster wide events and how the administrator chooses to be notified. EMS provides a cataloged logging mechanism, and every event has a formal definition. This allows EMS to provide services such as automatic spam management (such as message suppression), configurable notifications, assistance with translating low-level data into understandable text, NVRAM backing of messages, and automatic tagging of messages.

EMS contains thousands of predefined messages that are triggered on the corresponding event. The dot-separated, tree-style naming scheme of the messages provides significant accuracy pertaining to the messages’ origin and meaning. The formal event definition describes the meaning of the event in the context of the cluster. Each event contains a corrective action description, which can assist and accelerate the decisions the administrator needs to make in response to the event. This standardization and accuracy also carry over to NetApp’s manageability tools, which utilize EMS data.
Note: EMS does not contain command history or administrative auditing.

EMS events are viewed at the command line with:


cluster::> event log show
Time                Node             Severity       Event
------------------- ---------------- ------------- ---------------------------
3/18/2014 13:00:04  cluster-01       INFORMATIONAL kern.uptime.filer: 1:00pm up 20:17

Audit Logs

Audit logging is essential for the administrative security of the clustered Data ONTAP system. The audit log records the commands sent to the cluster, the user who is sending them, and the success or failure of the command. This applies to command line interface (CLI), Data ONTAP API (ONTAPI®) calls (such as commands from NetApp manageability tools), and HTTP requests.

In Data ONTAP 8.3 and earlier, the audit log is stored in /mroot/etc/log/mlog/command-history.log. Command history can also be viewed in the MGWD log, located in /mroot/etc/log/mlog/mgwd.log.  Beginning in ONTAP 9, the  command-history.log file is replaced by  audit.log, and the  mgwd.log file no longer contains audit information.

How ONTAP implements audit logging
Management activities recorded in the audit log are included in standard AutoSupport reports, and certain logging activities are included in EMS messages. You can also forward the audit log to destinations that you specify, and you can display audit log files by using the CLI or a web browser.

ONTAP logs management activities that are performed on the cluster, for example, what request was issued, the user who triggered the request, the user's access method, and the time of the request.
The management activities can be one of the following types:
  • Set requests, which typically apply to non-display commands or operations
    • These requests are issued when you run a create, modify, or delete command, for instance.
    • Set requests are logged by default.
  • Get requests, which retrieve information and display it in the management interface
    • These requests are issued when you run a show command, for instance.
    • Get requests are not logged by default, but you can use the security audit modify command to control whether get requests sent from the ONTAP CLI (-cliget) or from the ONTAP APIs (-ontapiget) are logged in the file.

 

ONTAP records management activities in the  /mroot/etc/log/mlog/audit.log file of a node. Commands from the three shells for CLI commands—the clustershell, the nodeshell, and the non-interactive systemshell (interactive systemshell commands are not logged)—as well as API commands are logged here. Audit logs include timestamps to show whether all nodes in a cluster are time synchronized.

The  audit.log file is sent by the AutoSupport tool to the specified recipients. You can also forward the content securely to external destinations that you specify; for example, a Splunk or a syslog server.
The  audit.log file is rotated daily. The rotation also occurs when it reaches 100 MB in size, and the previous 34 copies are preserved (with a maximum total of 35 files). When the audit file performs its daily rotation, no EMS message is generated. If the audit file rotates because its file size limit is exceeded, an EMS message is generated.

You can use the  security audit log show command to display audit entries for individual nodes or merged from multiple nodes in the cluster. You can also display the content of the  /mroot/etc/log/mlog directory on a single node by using a web browser.

AutoSupport

The AutoSupport (ASUP™) system is the clustered Data ONTAP automated health-monitoring facility that enables error reporting and, in some instances, can generate a NetApp Support case. Reporting might be triggered by an error condition using an EMS event or by schedule. ASUP alerts can be sent to the administrator’s internal IT organization using e-mail and/or to NetApp Support for automated analysis. The ASUP message contains important log data from EMS and other user space applications. Exactly which logs ASUP collects is discussed in the next section.

Other Logs

EMS events follow the syslog standard because they have the ability to be forwarded to a syslog server for real-time monitoring and because EMS events are the most relevant events to an administrator. The rest of the logs generated by the clustered Data ONTAP operating system are generated from user space applications that are constantly logging their activity. These logs are lower level and not targeted for administrators, but are mostly utilized by NetApp Support, development, and QA.

Table 1) Logs in /mroot/etc/log/

Log or directory

Description

acp/

  • Sent to ASUP
  • Shelf Alternate Control Path Management (ACP) logs

auditlog.log

  • Logs node shell commands (i.e., node run commands)
  • Equivalent is command-history.log in 8.3 and earlier.
  • Sent in Autosupport starting in clustered Data ONTAP 8.2.2

autosupport/

  • Directory for the compressed archives containing the log files to be sent to ASUP

backup.log

  • Log for NDMP backup procedures such as SMTape

bcomka/

  • Debug-level logs for the SAN kernel module

clone.log

  • Logs LUN cloning

ems, ems.log

  • EMS events
  • Sent in Autosupport

ems_persist

  • Binary formatted file, used by NetApp Support in certain circumstances

leak_data, leak_data_filtered

  • Memory information, pertinent mostly for debugging purposes

messages, messages.log

  • Node level logs
  • Logs are links to /mroot/etc/log/mlog/messages.log

mlog/

  • Contents of this directory are sent to ASUP
  • Contains management component application logs

named.log

  • Name service logs

nbu_snapvault.log

  • SnapVault® logs

playlist_diag

  • Logs absent FileIDs from the WAFL® playlist

plxcoeff/

  • Contains PLX PCI-E switch logs
  • Sent to ASUP starting in clustered Data ONTAP 8.2.1

rastrace/

  • Debug SAN trace logs

servprocd/

  • Service processor logs

shelflog/

  • Shelf logs

sis, sis.log

  •  Deduplication logs

ssram/

  • System scratchpad RAM log

stats/

  • Performance-related logs

snapmirror.log, snapmirror_audit.log, snapmirror_error.log

  • SnapMirror® logs

treecompare.log

  • Logs for the treecompare process that compare data integrities in volumes and/or qtrees using Snapshot® copies

volread.log

  • Logs for the volread engine used by SnapMirror

 

Table 2) Logs in /mroot/etc/log/mlog/

Log or directory

Description

last_rotate.log

  • Records history of log rotations

apache_access.log

  • Logs history of access to the apache server
  • Contains history of GET requests for log files over HTTP(S)

apache_error.log

  • Logs apache errors

audit.log

  • Audit log for ONTAP 9.0 and later
  • Records commands from CLI, ONTAPI, HTTP
  • Always records set requests, but can toggle recording of get requests
bcomd.log
  • Logs for the BCOM daemon, which handles SAN interaction between the management component and SCSI blade
command-history.log
  • Audit log for clustered Data ONTAP 8.3 and earlier
  • Records commands from CLI, ONTAPI, HTTP
  • Always records set requests, but can toggle recording of get requests

debug.log

  • Logs at the DEBUG severity level

fpolicy.log

  • Logs for FPolicy®

hashd.log

  • Logs for the BranchCache hash daemon

jm-restart.log

  • Contains list of jobs that the job manager has restarted

memsnap-*.log (asterisk is a wildcard, because there are several types of memsnap logs)

  • Contains memory information

messages.log

  • The messages log in clustered Data ONTAP
  • Contains important logs throughout cluster
  • Some overlap with EMS, but no EMS features such as suppression

mgwd.log

  • Contains logs from the management component
  • Records set requests by default, but can be toggled

ndmpd.log

  • Contains logs for the NDMP daemon

notifyd.log

  • Contains logs for the NOTIFY daemon, which handles ASUP

php.log

  • Contains logs for the PHP process
  • Contains history of syncing logs across nodes in the cluster

secd.log

  • Contains logs for the SecD daemon, which handles various authentication tasks, such as NAS authentication

servprocd.log

  • Contains logs on the service processor daemon

sktlog/

  • Debug-level logs for the main kernel
sktlogd.log
  •  Debug-level log for the main kernel

spdebug.log

  • Contains logs related to abnormal events from the service processor

spmd.log

  • Contains logs on the service process manager daemon, which monitors user space applications to make sure they are healthy and running

vifmgr.log

  • Contains logs related to interfaces and networking

vldb.log

  • Contains logs on the volume location database application

 

Additional Information

additionalInformation_text