Skip to main content

NetApp_Insight_2020.png 

NetApp Knowledgebase

FAQ: Overview of Event Management System for ONTAP 9

Views:
526
Visibility:
Public
Votes:
1
Category:
data-ontap-8
Specialty:
core
Last Updated:

Applies to

ONTAP 9                

Answer

What is an EMS event message?

EMS events are a record of occurences in ONTAP 9, and are by default logged in the Event Management System Log. EMS event messages have several components, which can be reviewed in the ONTAP event catalogue.

Example of how to look up details on an EMS event message:
::> event catalog show -message-name monitor.volume.nearlyFull
     Message Name: monitor.volume.nearlyFull
         Severity: ALERT

Description: This message occurs when one or more file systems are nearly full, typically indicating at least 95% full. This event is accompanied by global health monitoring messages for the customer. The space usage is computed based on the active file system size and is computed by subtracting the value of the ' Snapshot Reserve" field from the value of the 'Used' field of the ' volume show-space' command.

Corrective Action: Create space by increasing the volume or aggregate sizes, or by deleting data or deleting Snapshot (R) copies. To increase a volume's size, run the ' volume size' command. To delete a volume's Snapshot (R) copies, run the ' volume snapshot delete' command. To increase an aggregate's size, add disks by running the ' storage aggregate add-disks' command. Aggregate Snapshot (R) copies are deleted automatically when the aggregate is full.
SNMP Trap Type: Built-in
Is Deprecated: false

 

The unique characteristic of any given event is the message name. In this example, the message name is monitor.volume.nearlyFull. Additionally, the severity is 'ALERT' and the SNMP Trap Type associated with the event is 'Built-in'.
The Severity of any given event message is an indication of the expected impact of the event. Here is a list of the severities and an explanation of their meanings:

::> event catalog show -severity ?
  EMERGENCY                   Disruption
  ALERT                       Single point of failure
  ERROR                       Degradation
  NOTICE                      Information
  INFORMATIONAL               Information
  DEBUG                       Debug information
The SNMP Trap Type is discussed in the ONTAP 9 Documentation Center
Historically, prior to ONTAP 9, EMS Event Messages were individually configured to destinations on a per-message basis:
 
cm3270c-rtp::*> even route show -message-name monitor.volume.nearlyFull -destinations ?
  (event route show)
  allevents
  asup
  criticals
  pager
  traphost

 

Eventually, the catalog of EMS Event Messages grew and became difficult to manage on a per-message basis, and so a newer filter-based system of routing event messages was implemented in ONTAP 9.  The new system allows for rules-based event filters to collect events for delivery to event destinations by using event notifications to correlate event filters to event destinations.  Upon installation or upgrade to ONTAP 9, a basic configuration of event filters, event destinations, and event notifications is implemented by default. The default configuration can be disabled by removing event notifications, but the built-in event filters and event destinations cannot be modified or deleted (but they can be copied into new user customizable filters and destinations for further customization).


cm3270c-rtp::*> system snmp traphost show
snmp-traphost   snmp        - (from "system snmp traphost")
--------------  ----------  ---------------------
Name            Type        Destination
cm3270c-rtp::*> event notification destination show
1    default-trap-events             snmp-traphost
---- ------------------------------  -----------------
ID   Filter Name                     Destinations
cm3270c-rtp::*> event notification show
9 entries were displayed.
            2        exclude   *                      *               *
            1        include   *                      *               EMERGENCY, ALERT, ERROR, NOTICE
no-info-debug-events
            3        exclude   *                      *               *
            2        include   callhome.*             *               ERROR
            1        include   *                      *               EMERGENCY, ALERT
important-events
            4        exclude   *                      *               *
                                                                      *
            3        include   *                      Standard, Built-in
            2        include   callhome.*             *               ERROR
            1        include   *                      *               EMERGENCY, ALERT
default-trap-events
----------- -------- --------- ---------------------- --------------- --------
            Position Type
Filter Name Rule     Rule      Message Name           SNMP Trap Type  Severity
cm3270c-rtp::*> event filter show
        -
The built-in destination 'snmp-traphost' is linked to the default traphost configured for a storage system by running the command ' system snmp traphost add' or when configured through OnCommand System Manager at a URL similar to the following example:
https:// /sysmgr/SysMgr.html#snmp


 
1073980.png

How do ONTAP 9 EMS Event Filters work?

Any time an EMS Event Message is generated, it will be compared against all configured EMS Event Filters. An EMS Event Filter is a list of rules that include or exclude any given EMS Event Message. Each message is compared to the rules in an EMS Event Filter in sequential order, seeking to match the rule, and upon any match, further rule processing ceases. The last rule in every EMS Event Filter will match every event message and exclude them. Therefore, if an EMS Event Message does not match an earlier rule, it will be excluded from the filter.  So, a newly created EMS Event Filter with only the default rule will not match ANY EMS Event Messages.
You can create a custom EMS Event Filter similar to the following example:
cm3270c-rtp::> event filter create -filter-name Custom_Filter
cm3270c-rtp::> event filter show -filter-name Custom_Filter
Filter Name Rule     Rule      Message Name           SNMP Trap Type  Severity
            Position Type
----------- -------- --------- ---------------------- --------------- --------
Custom_Filter
            1        exclude   *                      *               *

 

Notice, this newly created Event Filter automatically includes the default rule in position 1 that excludes Event Messages that match any criteria (Message Name, SNMP Trap Type, and Severity). This ensures the filter will not collect any EMS Event Messages that are not desired. For the example EMS Event Message, monitor.volume.nearlyFull, create a rule that will collect it in the new filter.

cm3270c-rtp::> event filter rule add -filter-name Custom_Filter -type include -message-name monitor.volume.nearlyFull
 
cm3270c-rtp::> event filter show -filter-name Custom_Filter                                                   Filter Name Rule     Rule      Message Name           SNMP Trap Type  Severity
            Position Type
----------- -------- --------- ---------------------- --------------- --------
Custom_Filter
            1        include   monitor.volume.nearlyFull
                                                      *               *
            2        exclude   *                      *               *
2 entries were displayed.

 

This rule will collect any events that match the message name monitor.volume.nearlyFull, however, assuming that it is required to collect all of the EMS Event Messages that match the query "monitor.volume.*", it is possible to




            2        exclude   *                      *               *
            1        include   monitor.volume.*       *               *
Custom_Filter
----------- -------- --------- ---------------------- --------------- --------
            Position Type
Filter Name Rule     Rule      Message Name           SNMP Trap Type  Severity
cm3270c-rtp::> event filter show -filter-name Custom_Filter
cm3270c-rtp::> event filter rule add -filter-name Custom_Filter -type include -message-name monitor.volume.* 
cm3270c-rtp::> event filter rule delete -filter-name Custom_Filter -position 1
Now, our rule will collect all of these EMS Event Messages:
cm3270c-rtp::> event catalog show -message-name monitor.volume.*
Message                          Severity         SNMP Trap Type
-------------------------------- ---------------- -----------------
monitor.volume.full              DEBUG            Built-in
monitor.volume.nearlyFull        ALERT            Built-in
monitor.volume.ok                DEBUG            Built-in
3 entries were displayed.

 

However in testing, it was decided that it is not desirable to collect monitor.volume.ok. Therefore, insert a rule to be processed earlier in the event filter to exclude specifically that event message. This is how it is done:

cm3270c-rtp::> event filter rule add -filter-name Custom_Filter -type exclude -message-name monitor.volume.ok -position 1
 
cm3270c-rtp::> event filter show -filter-name Custom_Filter                                                   Filter Name Rule     Rule      Message Name           SNMP Trap Type  Severity
            Position Type
----------- -------- --------- ---------------------- --------------- --------
Custom_Filter
            1        exclude   monitor.volume.ok      *               *
            2        include   monitor.volume.*       *               *
            3        exclude   *                      *               *
3 entries were displayed.

 

These examples focus on the EMS Event Message Name, however, it is possible to also filter on SNMP Trape Type or Severity as well. Therefore, for example, if you wanted to also have the filter collect all events of Severity level ALERT, you could add a rule using that criteria. 

cm3270c-rtp::> event filter rule add -filter-name Custom_Filter -type include -severity ALERT
 
cm3270c-rtp::> event filter show -filter-name Custom_Filter
Filter Name Rule     Rule      Message Name           SNMP Trap Type  Severity
            Position Type
----------- -------- --------- ---------------------- --------------- --------
Custom_Filter
            1        exclude   monitor.volume.ok      *               *
            2        include   monitor.volume.*       *               *
            3        include   *                      *               ALERT
            4        exclude   *                      *               *
4 entries were displayed.


What does Error: command failed: This rule does not match any event. Enter a valid rule. mean?

Ex.
cm3270c-rtp::>event filter rule add -filter-name Inodes_Events -type include -message-name wafl.vol.runningOutOfInodes -severity ALERT

Error: command failed: This rule does not match any event. Enter a valid rule.
This means the severity type is incorrect for the Ems message.

To verfy:
cm3270c-rtp::*> event catalog show -message-name wafl.vol.runningOutOfInodes
Message Name: wafl.vol.runningOutOfInodes
Severity: ERROR

The severity should be ERROR, the correct command would be
cm3270c-rtp::>event filter rule add -filter-name Inodes_Events -type include -message-name wafl.vol.runningOutOfInodes -severity ERROR


For more information on the commands to manage EMS Event Filters, follow this link:
ONTAP 9 Documentation Center 
 

How do ONTAP 9 EMS Event Notification Destinations work?

ONTAP 9 Event Notification Destinations control the delivery of EMS Event Messages that have been collected by an EMS Event Filter. The destinations can be email addresses, syslog servers, SNMP traphosts, or REST .API Servers. By default, the only EMS Event Notification Destination is the built-in ' snmp-traphost', which is not removable, and maps to the SNMP traphost configuration found in 'system snmp traphost', which can be configured separately (or not at all). 
cm3270c-rtp::> event notification destination show
Name            Type        Destination
--------------  ----------  ---------------------
snmp-traphost   snmp        - (from "system snmp traphost")
cm3270c-rtp::> system snmp traphost show
        -

 

You can create additional custom event destinations by running the following command:

cm3270c-rtp::> event notification destination create
Usage:
   [-name]              Destination Name
   { [-email]   Email Destination
   | [-syslog]          Syslog Destination
   | [-rest-api-url]    REST API Server URL
    [[-certificate-authority] ]
                              Client Certificate Issuing CA
    [ -certificate-serial ] }
                              Client Certificate Serial Number
 
cm3270c-rtp::> event notification destination create Custom_Destination_syslog -syslog 1.2.3.4
 
cm3270c-rtp::> event notification destination create Custom_Destination_email -email user@domain.com
cm3270c-rtp::> event notification destination show
Name            Type        Destination
--------------  ----------  ---------------------
Custom_Destination_email
                email       user@domain.com (via "localhost" from "admin@localhost", configured in "event config")
Custom_Destination_syslog
                syslog      1.2.3.4
snmp-traphost   snmp        - (from "system snmp traphost")
3 entries were displayed.
 
The Custom EMS Event Notification Destination for syslog has the IP address 1.2.3.4. Notice, the event notification destination type 'email' has a note in parenthesis that shows the mail server and the source email address as configured in event config:
cm3270c-rtp::> event config show
                      Mail From:  admin@localhost
                    Mail Server:  localhost
                      Proxy URL:  -
                     Proxy User:  -

How do EMS Event Notifications work?

EMS Event Notifications define maps between the payload collected in an EMS Event Filter and delivery targets defined in EMS Event Notification Destinations. By default, there is one EMS Event Notification preconfigured to map the built-in default-trap-events EMS Event Filter to the built-in snmp-traphost EMS Event Notification Destination. If desired, this default EMS Event Notification can be removed.
cm3270c-rtp::> event notification show
ID   Filter Name                     Destinations
---- ------------------------------  -----------------
1    default-trap-events             snmp-traphost
cm3270c-rtp::> event notification delete 1
cm3270c-rtp::> event notification show
This table is currently empty.

 

When creating an EMS Event Notification, specify exactly one EMS Event Filter and one or more EMS Event Notification Destinations. The EMS Event Message will automatically be translated into the appropriate format for each EMS Event Notification Destination depending on the type, either email, SNMP, syslog event message, etc. 

cm3270c-rtp::vserver> event notification create -filter-name Custom_Filter -destination Custom_Destination_email
 
cm3270c-rtp::vserver> event notification create -filter-name Custom_Filter -destination Custom_Destination_syslog
cm3270c-rtp::vserver> event notification show
ID   Filter Name                     Destinations
---- ------------------------------  -----------------
1    Custom_Filter                   Custom_Destination_email
2    Custom_Filter                   Custom_Destination_syslog
2 entries were displayed.
An EMS Event Filter can be referenced in multiple EMS Event Notifications, and there can be redundancies introduced if care is not taken:
cm3270c-rtp::vserver> event notification create -filter-name Custom_Filter -destination Custom_Destination_syslog,Custom_Destination_email
 
cm3270c-rtp::vserver> event notification show                                                                 ID   Filter Name                     Destinations
---- ------------------------------  -----------------
1    Custom_Filter                   Custom_Destination_email
2    Custom_Filter                   Custom_Destination_syslog
3    Custom_Filter                   Custom_Destination_syslog, Custom_Destination_email
3 entries were displayed.

 

If you delete an EMS Event Filter, any corresponding EMS Event Notifications will be deleted as well. If you delete an EMS Event Notification Destination, it will automatically be removed from any EMS Event Notifications (and if it is the last defined EMS Event Destination, the EMS Event Notification will also be deleted):

cm3270c-rtp::> event notification destination delete -name Custom_Destination_syslog
 
Warning: The destination will be deleted from all notifications, if present. If
         this was the only destination in the notification, it will be deleted
         too.
Do you want to continue? {y|n}: y
 
cm3270c-rtp::> event filter delete -filter-name Custom_Filter
 
Warning: Deleting this filter will delete the notification as well.
Do you want to continue? {y|n}: y
 
cm3270c-rtp::> event filter delete -filter-name Custom_Filter
cm3270c-rtp::> event notification show
This table is currently empty.
 
  

Additional Information

N/A