Events not sent to syslog server due to network routes

• ONTAP 9
• Syslog
• Event Forwarding

• After configuring event forwarding to a syslog server the events are not sent to the syslog server.  

cluster::> event filter show -filter-name syslog-NetApp
FFilter Name Rule     Rule      Message Name           SNMP Trap Type  Severity
            Position Type
----------- -------- --------- ---------------------- --------------- --------
syslog-NetApp
            1        include   callhome.*             *               ERROR
            2        include   *                      *               EMERGENCY, ALERT, ERROR, NOTICE, INFORMATIONAL
            3        exclude   *                      *               *
3 entries were displayed.

cluster::> event notification destination show
Name            Type        Destination
--------------  ----------  ---------------------
syslog-NetApp   syslog      1.2.3.4

cluster::> event notification show
ID   Filter Name                     Destinations
---- ------------------------------  -----------------
...
4    syslog-NetApp                   syslog-NetApp
4 entries were displayed.

• Syslog server does not see any communication from any of the mgmt LIFs.  
• Packet traces do not show any communication to the syslog server.

• Can ping syslog server by IP and hostname using the mgmt LIFs(node and cluster mgmt):

Cluster::*> net int show -role *mgmt*

  (network interface show)
            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
Cluster
            node1_mgmt1
                         up/up    x.x.x.x/24   node1
                                                                   e0M     true
            node2_mgmt1
                         up/up    x.x.x.x/24   node2
                                                                   e0M     true
            cluster_mgmt_lif_1
                         up/up    x.x.x.x/24   node1
                                                                   e0M     true

Example

cluster::> network ping -vserver Cluster -lif node2_mgmt1 -destination 1.2.3.4
  (network ping)
1.2.3.4 is alive 

Cluster::> network ping -lif node2_mgmt1 -destination syslog.mycompany.com -vserver Cluster
  (network ping)
syslog.mycompany.com is alive

cluster::> net traceroute -vserver Cluster -lif node2_mgmt1 -destination 1.2.3.4
  (network traceroute)
traceroute to 1.2.3.4 (1.2.3.4) from 10.130.252.31, 64 hops max, 40 byte packets
 1  1.2.3.4 (1.2.3.4)  0.194 ms  0.144 ms  0.113 ms

• Test events are sent to the syslog destination.  

cluster::> event generate -message-name monitor.volume.nearlyFull -values TEST,TEST,TEST,TEST,TEST,TEST    

Cluster::*> event notification history show -destination syslog-NetApp
Time                Node             Severity      Event
------------------- ---------------- ------------- ---------------------------
5/6/2022 13:48:42   node1 
                                     ERROR         monitor.volume.nearlyFull: TEST TESTTESTTEST is nearly full (using or reserving TEST% of space and TEST% of inodes).

• Admin vserver has one default route

Cluster::*> network route show -vserver Cluster                                     
                                     cluster::> network route show -vserver Cluster
Vserver             Destination     Gateway         Metric
------------------- --------------- --------------- ------
Cluster
                    0.0.0.0/0       x.x.252.1    20