Configuring a HA group with a single network interface can cause keepalived to crash
Applies to
StorageGRID 11.7
Issue
Adding an HA group containing a single network interface can cause keepalived to crash.
Resulting in the virtual IP address being unavailable/inaccessible on the node:
Resulting in the virtual IP address being unavailable/inaccessible on the node:
- The S3-Endpoint is unable to receiving (or transmit) any data.
- The load-balancer reports that the keepalived is down.
- And the endpoint/keepalived will not come back up even after a node reboot.
keepalived.log
will show:Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: pid 33299 exited due to segmentation fault (SIGSEGV).
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Please report a bug at https://github.com/acassen/keepalived/issues
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: and include this log from when keepalived started, a description
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: of what happened before the crash, your configuration file and the details below.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Also provide the output of keepalived -v, what Linux distro and version
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: you are running on, and whether keepalived is being run in a container or VM.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: A failure to provide all this information may mean the crash cannot be investigated.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: If you are able to provide a stack backtrace with gdb that would really help.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Source version 2.1.5
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Built with kernel headers for Linux 5.10.70
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Running on Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1+ntap7 (2023-02-22)
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Command line: '/usr/sbin/keepalived' '--vrrp' '--log-detail' '--dump-conf' '--log-facility=6'
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: configure options: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --localstatedir=/var --disable-option-checking --disable-silent-rules
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --disable-maintainer-mode --disable-dependency-tracking --with-kernel-dir=debian/
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --enable-snmp --enable-sha1 --enable-snmp-rfcv2 --enable-snmp-rfcv3 --enable-dbus
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: --enable-json --enable-bfd --enable-regex build_alias=x86_64-linux-gnu CFLAGS=-g
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: -O2 -ffile-prefix-map=/build/keepalived-jCUlld/keepalived-2.1.5=.
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: -fstack-protector-strong -Wformat -Werror=format-security LDFLAGS=-Wl,-z,relro
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: Config options: NFTABLES LVS REGEX VRRP VRRP_AUTH JSON BFD OLD_CHKSUM_COMPAT FIB_ROUTING
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 DBUS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: System options: PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV4_DEVCONF IPV6_ADVANCED_API
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_OIFNAME FRA_PROTOCOL
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: IP_MULTICAST_ALL LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_VMAC VRRP_IPVLAN IFLA_LINK_NETNSID
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE VRF SO_MARK
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: SCHED_RESET_ON_FORK
Jul 6 10:32:24 R06SG100LB05 Keepalived[8315]: VRRP child process(33299) died: Respawning
keepalived
crash dumps (/var/local/core
) will show:
[New LWP 8316]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/keepalived --vrrp --log-detail --dump-conf --log-facility=6'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055cef0222380 in inet_sockaddrcmp ()
#0 0x000055cef0222380 in inet_sockaddrcmp ()
#1 0x000055cef01fa770 in vrrp_dispatcher_init ()
#2 0x000055cef0227b9e in process_threads ()
#3 0x000055cef01e91f1 in start_vrrp_child ()
#4 0x000055cef01bd779 in ?? ()
#5 0x000055cef0227b9e in process_threads ()
#6 0x000055cef01bf046 in keepalived_main ()
#7 0x00007f20617ced0a in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x000055cef01bc90a in _start ()
Thread 1 (Thread 0x7f2060e81cc0 (LWP 8316)):
#0 0x000055cef0222380 in inet_sockaddrcmp ()
#1 0x000055cef01fa770 in vrrp_dispatcher_init ()
#2 0x000055cef0227b9e in process_threads ()
#3 0x000055cef01e91f1 in start_vrrp_child ()
#4 0x000055cef01bd779 in ?? ()
#5 0x000055cef0227b9e in process_threads ()
#6 0x000055cef01bf046 in keepalived_main ()
#7 0x00007f20617ced0a in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x000055cef01bc90a in _start ()