Skip to main content
NetApp Knowledge Base

Configuring a HA group with a single network interface can cause keepalived to crash

Views:
139
Visibility:
Public
Votes:
2
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

StorageGRID 11.7

Issue

Adding an HA group containing a single network interface can cause keepalived to crash.
Resulting in the virtual IP address being unavailable/inaccessible on the node:
  • The S3-Endpoint is unable to receiving (or transmit) any data.
  • The load-balancer reports that the keepalived is down.
  • And the endpoint/keepalived will not come back up even after a node reboot.
keepalived.log will show:

Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: pid 33299 exited due to segmentation fault (SIGSEGV).
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Please report a bug at https://github.com/acassen/keepalived/issues
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: and include this log from when keepalived started, a description
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: of what happened before the crash, your configuration file and the details below.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Also provide the output of keepalived -v, what Linux distro and version
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: you are running on, and whether keepalived is being run in a container or VM.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: A failure to provide all this information may mean the crash cannot be investigated.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: If you are able to provide a stack backtrace with gdb that would really help.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Source version 2.1.5
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Built with kernel headers for Linux 5.10.70
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Running on Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1+ntap7 (2023-02-22)
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Command line: '/usr/sbin/keepalived' '--vrrp' '--log-detail' '--dump-conf' '--log-facility=6'
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: configure options: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --localstatedir=/var --disable-option-checking --disable-silent-rules
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --disable-maintainer-mode --disable-dependency-tracking --with-kernel-dir=debian/
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --enable-snmp --enable-sha1 --enable-snmp-rfcv2 --enable-snmp-rfcv3 --enable-dbus
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --enable-json --enable-bfd --enable-regex build_alias=x86_64-linux-gnu CFLAGS=-g
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: -O2 -ffile-prefix-map=/build/keepalived-jCUlld/keepalived-2.1.5=.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: -fstack-protector-strong -Wformat -Werror=format-security LDFLAGS=-Wl,-z,relro
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Config options: NFTABLES LVS REGEX VRRP VRRP_AUTH JSON BFD OLD_CHKSUM_COMPAT FIB_ROUTING
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 DBUS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: System options: PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV4_DEVCONF IPV6_ADVANCED_API
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_OIFNAME FRA_PROTOCOL
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: IP_MULTICAST_ALL LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_VMAC VRRP_IPVLAN IFLA_LINK_NETNSID
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE VRF SO_MARK
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: SCHED_RESET_ON_FORK
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: VRRP child process(33299) died: Respawning
 

keepalived crash dumps (/var/local/core) will show:

[New LWP 8316]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/keepalived --vrrp --log-detail --dump-conf --log-facility=6'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055cef0222380 in inet_sockaddrcmp ()
#0  0x000055cef0222380 in inet_sockaddrcmp ()
#1  0x000055cef01fa770 in vrrp_dispatcher_init ()
#2  0x000055cef0227b9e in process_threads ()
#3  0x000055cef01e91f1 in start_vrrp_child ()
#4  0x000055cef01bd779 in ?? ()
#5  0x000055cef0227b9e in process_threads ()
#6  0x000055cef01bf046 in keepalived_main ()
#7  0x00007f20617ced0a in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x000055cef01bc90a in _start ()
 
Thread 1 (Thread 0x7f2060e81cc0 (LWP 8316)):
#0  0x000055cef0222380 in inet_sockaddrcmp ()
#1  0x000055cef01fa770 in vrrp_dispatcher_init ()
#2  0x000055cef0227b9e in process_threads ()
#3  0x000055cef01e91f1 in start_vrrp_child ()
#4  0x000055cef01bd779 in ?? ()
#5  0x000055cef0227b9e in process_threads ()
#6  0x000055cef01bf046 in keepalived_main ()
#7  0x00007f20617ced0a in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x000055cef01bc90a in _start ()

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.