Skip to main content
NetApp Knowledge Base

Client automount occasionally fails due to DNS query failure

  1. Last updated
    Dec 18, 2024
  2. Save as PDF
Views:
648
Visibility:
Public
Votes:
0
Category:
fas-systems
Specialty:
nas
Last Updated:
12/18/2024, 4:41:33 PM

Applies to

On-box DNS load balancing
 

Issue

  • In a mount storm environment, it is possible that the external linux DNS server with Bind version equo or greater than bind-9.9.4-69.el7 which start to support 0 ttl forwards DNS query request to non-listener On-box DNS server LIFs
  • If external DNS server forwards DNS query request to non-listener On-box DNS server LIFs, ONTAP refuses initial connection request
  • It finally results in client DNS query failure
 
Example:
 
/// IP info
 
NFS client:
192.168.0.61
External DNS server IP:
192.168.0.188
Storage IPs:
192.168.0.134
192.168.0.135 (only this IP is allowed to process DNS query request. listen-for-dns-query:true)
192.168.0.136
192.168.0.137
192.168.0.139
 
/// OTNAP config
 
cluster1::*> net int show -vserver svm2_cluster1 -fields listen-for-dns-query,dns-zone,address
(network interface show)
vserver   lif    address   dns-zone   listen-for-dns-query
------------- --------------------- ------------- ----------------------- --------------------
svm2_cluster1 lif_svm2_cluster1_433 192.168.0.135 storage_hostname.test.com true
svm2_cluster1 lif_svm2_cluster1_453 192.168.0.137 storage_hostname.test.com false
svm2_cluster1 lif_svm2_cluster1_583 192.168.0.134 storage_hostname.test.com false
svm2_cluster1 lif_svm2_cluster1_858 192.168.0.136 storage_hostname.test.com false
svm2_cluster1 lif_svm2_cluster1_996 192.168.0.139 storage_hostname.test.com false
 
/// Zone file config
 
/var/named/test.zone
------------------------------------------
;TR guide setting
$ORIGIN storage_hostname.com.
@    IN    NS    storage_hostname.com.
IN    NS    ansible.seccae.com.
storage_hostname.test.com.   IN    A   192.168.0.135
 

/// Log analysis

NFS client returns SERVFAIL error and automount fail message 

Sun Jun 26 15:14:06 UTC 2022
storage_hostname.test.com has address 192.168.0.134
Host storage_hostname.test.com not found: 2(SERVFAIL)
Host storage_hostname.test.com not found: 2(SERVFAIL)
Sun Jun 26 15:14:07 UTC 2022
Host storage_hostname.test.com not found: 2(SERVFAIL)

May 9 07:55:12 client_hostname automount[1364] : add_host_addrs: hostname lookup for storage_hostname failed: Name or service not known

External DNS server returns connection refused info level message from named.run log

6-Jun-2022 15:10:02.301 resolver: debug 1: fetch: storage_hostname.test.com/AAAA
26-Jun-2022 15:10:02.335 resolver: debug 1: fetch: storage_hostname.test.com/MX
26-Jun-2022 15:14:06.990 resolver: debug 1: fetch: storage_hostname.test.com/A
26-Jun-2022 15:14:07.004 resolver: debug 1: fetch: storage_hostname.test.com/AAAA
26-Jun-2022 15:14:07.005 lame-servers: info: connection refused resolving 'storage_hostname.test.com/AAAA/IN': 192.168.0.134#5326-Jun-2022 15:14:07.005 resolver: debug 1: fetch: storage_hostname.test.com/AAAA
26-Jun-2022 15:14:07.005 query-errors: debug 1: client @0x7f5362edac20 192.168.0.61#60563 (storage_hostname.test.com): query failed (SERVFAIL) for storage_hostname.test.com/IN/AAAA at ../../../bin/named/query.c:8580
26-Jun-2022 15:14:07.005 lame-servers: debug 1: lame server resolving 'storage_hostname.test.com' (in 'storage_hostname.test.com'?): 192.168.0.188#53
26-Jun-2022 15:14:07.005 resolver: debug 1: fetch: storage_hostname.test.com/MX
26-Jun-2022 15:14:07.006 lame-servers: info: connection refused resolving 'storage_hostname.test.com/AAAA/IN': 192.168.0.134#53
26-Jun-2022 15:14:07.006 lame-servers: info: connection refused resolving 'storage_hostname.test.com/MX/IN': 192.168.0.134#53
26-Jun-2022 15:14:07.006 query-errors: debug 1: client @0x7f53600aa070 192.168.0.61#57471 (storage_hostname.test.com): query failed (SERVFAIL) for storage_hostname.test.com/IN/MX at ../../../bin/named/query.c:8580
26-Jun-2022 15:14:07.017 resolver: debug 1: fetch: storage_hostname.test.com/A
26-Jun-2022 15:14:07.017 lame-servers: info: connection refused resolving 'storage_hostname.test.com/A/IN': 192.168.0.134#53
26-Jun-2022 15:14:07.017 query-errors: debug 1: client @0x7f5362edac20 192.168.0.61#48698 (storage_hostname.test.com): query failed (SERVFAIL) for storage_hostname.test.com/IN/A at ../../../bin/named/query.c:8580

Storage side pakcet trace clearly recorded this behavior
Here, only 192.168.0.135 is On-box DNS listner IP

72   2022-06-26 23:14:19.572835   6.154698   192.168.0.188   38078    192.168.0.135   53   DNS   Standard query 0x1d9c A storage_hostname.test.com OPT
73   2022-06-26 23:14:19.585266   0.012431   192.168.0.135   53   192.168.0.188   38078   DNS   Standard query response 0x1d9c A storage_hostname.test.com A 192.168.0.134 NS storage_hostname.test.com OPT
74   2022-06-26 23:14:19.586980   0.001714   192.168.0.188   33860    192.168.0.134   53   DNS   Standard query 0x88a9 AAAA storage_hostname.test.com OPT
75   2022-06-26 23:14:19.587027   0.000047   192.168.0.134   33860    192.168.0.188   53   ICMP   Destination unreachable (Port unreachable)

Complete behavior can be checked from the packet trace which is captured from external DNS server

  • A query succed
    • Client send A query request to external DNS server
    • External DNS server forward this A query request to On-box DNS IP 192.168.0.135
    • On-box DNS reply with A query response to external DNS server (at this point, it seems external DNS erver remember the IP 192.168.0.134)
    • External DNS server respond client with A query response (use 192.168.0.134 to mount storage system)
  • AAAA query failed
    • Client send AAAA query request to external DNS server
    • External DNS server forward this AAAA query request to On-box DNS IP 192.168.0.134
    • On-box DNS reply with Destination unreachable to external DNS server
    • External DNS server respond Client with query failed (SERVFAIL)
  • MX query failed
    • Client send MX query request to external DNS server
    • External DNS server forward this MX query request to On-box DNS IP 192.168.0.134
    • On-box DNS reply with Destination unreachable to external DNS server
    • External DNS server respond Client with query failed (SERVFAIL)
  • A query failed
    • Client send A query request to external DNS server
    • External DNS server forward this A query request to On-box DNS IP 192.168.0.134
    • On-box DNS reply with Destination unreachable to external DNS server
    • External DNS server respond Client with query failed (SERVFAIL)

967   2022-06-26 23:14:06.989631   0.000501   192.168.0.61   36365    192.168.0.188   53   DNS   Standard query 0x6fea A storage_hostname.test.com
969   2022-06-26 23:14:06.990708   0.000557   192.168.0.188   38078    192.168.0.135   53   DNS   Standard query 0x1d9c A storage_hostname.test.com OPT
970   2022-06-26 23:14:07.003287   0.012579   192.168.0.135   53   192.168.0.188   38078    DNS   Standard query response 0x1d9c A storage_hostname.test.com A 192.168.0.134 NS storage_hostname.test.com OPT
971   2022-06-26 23:14:07.003852   0.000565   192.168.0.188   53   192.168.0.61   36365    DNS   Standard query response 0x6fea A storage_hostname.test.com A 192.168.0.134 NS storage_hostname.test.com
972   2022-06-26 23:14:07.004491   0.000639   192.168.0.61   60563    192.168.0.188   53   DNS   Standard query 0x47bc AAAA storage_hostname.test.com
973   2022-06-26 23:14:07.004892   0.000401   192.168.0.188   33860    192.168.0.134   53   DNS   Standard query 0x88a9 AAAA storage_hostname.test.com OPT
974   2022-06-26 23:14:07.005017   0.000125   192.168.0.134   33860    192.168.0.188   53   ICMP   Destination unreachable (Port unreachable)
975   2022-06-26 23:14:07.005451   0.000434   192.168.0.188   53   192.168.0.61   60563    DNS   Standard query response 0x47bc Server failure AAAA storage_hostname.test.com
976   2022-06-26 23:14:07.005710   0.000259   192.168.0.61   57471    192.168.0.188   53   DNS   Standard query 0xe348 MX storage_hostname.test.com
977   2022-06-26 23:14:07.005742   0.000032   192.168.0.188   59080    192.168.0.134   53   DNS   Standard query 0xde2d AAAA storage_hostname.test.com OPT
978   2022-06-26 23:14:07.005809   0.000067   192.168.0.134   59080    192.168.0.188   53   ICMP   Destination unreachable (Port unreachable)
979   2022-06-26 23:14:07.005901   0.000092   192.168.0.188   51515    192.168.0.134   53   DNS   Standard query 0x0a65 MX storage_hostname.test.com OPT
980   2022-06-26 23:14:07.005999   0.000098   192.168.0.134   51515    192.168.0.188   53   ICMP   Destination unreachable (Port unreachable)
981   2022-06-26 23:14:07.006199   0.000200   192.168.0.188   53   192.168.0.61   57471    DNS   Standard query response 0xe348 Server failure MX storage_hostname.test.com
982   2022-06-26 23:14:07.016776   0.010577   192.168.0.61   48698    192.168.0.188   53   DNS   Standard query 0x573a A storage_hostname.test.com
983   2022-06-26 23:14:07.017217   0.000441   192.168.0.188   46874    192.168.0.134   53   DNS   Standard query 0xf57a A storage_hostname.test.com OPT
984   2022-06-26 23:14:07.017343   0.000126   192.168.0.134   46874    192.168.0.188   53   ICMP   Destination unreachable (Port unreachable)
985   2022-06-26 23:14:07.017536   0.000193   192.168.0.188   53   192.168.0.61   48698    DNS   Standard query response 0x573a Server failure A storage_hostname.test.com
987   2022-06-26 23:14:07.027914   0.010016   192.168.0.61   53770    192.168.0.188   53   DNS   Standard query 0x4a57 A storage_hostname.test.com
989   2022-06-26 23:14:07.028215   0.000217   192.168.0.188   53   192.168.0.61   53770    DNS   Standard query response 0x4a57 Server failure A storage_hostname.test.com
995   2022-06-26 23:14:07.038529   0.009511   192.168.0.61   41057    192.168.0.188   53   DNS   Standard query 0x498a A storage_hostname.test.com
996   2022-06-26 23:14:07.038756   0.000227   192.168.0.188   53   192.168.0.61   41057    DNS   Standard query response 0x498a Server failure A storage_hostname.test.com

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.