Bug 1298695 - Backport: vrrp: set router flag in neighbour advertisements
Backport: vrrp: set router flag in neighbour advertisements
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: keepalived (Show other bugs)
7.2
Unspecified Linux
urgent Severity urgent
: rc
: 7.2
Assigned To: Ryan O'Hara
Brandon Perkins
: OtherQA, ZStream
Depends On: 1419049
Blocks: 1420851 1433318
  Show dependency treegraph
 
Reported: 2016-01-14 13:26 EST by Nir Magnezi
Modified: 2017-08-01 15:36 EDT (History)
9 users (show)

See Also:
Fixed In Version: keepalived-1.3.4-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1433318 (view as bug list)
Environment:
Last Closed: 2017-08-01 15:36:38 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2169 normal SHIPPED_LIVE keepalived bug fix and enhancement update 2017-08-01 14:18:21 EDT

  None (edit)
Description Nir Magnezi 2016-01-14 13:26:28 EST
Description of problem:
=======================
Currently in openstack, Neutron HA-Routers are broken[1] for IPv6 due to a fix[2] for routers failover issue that uses garp_master_* flags.

The suggested backport[3] to keepalived will resolve the IPv6 issue[2] by proparly setting the router flag properly for neighbour advertisement packets.
By doing so, we can ensure both [1] and [2] are resolved alltoghether.

Note that this backport is needed regardless if we decide to change or revert the first fix[2].

[1] https://bugs.launchpad.net/neutron/+bug/1520517
[2] https://bugs.launchpad.net/neutron/+bug/1453855
[3] https://github.com/acassen/keepalived/pull/200/files

Version-Release number of selected component (if applicable):
=============================================================
keepalived-1.2.13-7.el7

How reproducible:
=================
Every time.

Steps to Reproduce:
===================
1. See https://bugs.launchpad.net/neutron/+bug/1520517
2.
3.

Actual results:
===============
Neighbour advertisements causes the instance to loose its default route, which stays gone until the next router advertisement packet arrives.

Expected results:
=================
IPv6 networking should work proparly both with IPv6 and IPv4.
Comment 4 Assaf Muller 2016-01-15 10:49:35 EST
This is the keepalived.conf we generate for a Neutron router that is connected to a single IPv6 (SLAAC) network:

cat keepalived.conf 
vrrp_instance VR_1 {
    state BACKUP
    interface ha-fbb34de1-85
    virtual_router_id 1
    priority 50
    garp_master_repeat 5
    garp_master_refresh 10
    nopreempt
    advert_int 2
    track_interface {
        ha-fbb34de1-85
    }
    virtual_ipaddress {
        169.254.0.1/24 dev ha-fbb34de1-85
    }
    virtual_ipaddress_excluded {
        2001:cdba:9abc:5678::1/64 dev qr-863163ef-24
        fe80::f816:3eff:fee7:6aeb/64 dev qr-863163ef-24 scope link
    }
}

'ip address' output for that router:

sudo ip netns exec qrouter-84383bbe-64e0-461f-bd30-76b37d714c4d ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    ...
51: ha-fbb34de1-85: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:93:07:e6 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-fbb34de1-85
       valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-fbb34de1-85
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe93:7e6/64 scope link 
       valid_lft forever preferred_lft forever
56: qr-863163ef-24: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:e7:6a:eb brd ff:ff:ff:ff:ff:ff
    inet6 2001:cdba:9abc:5678::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fee7:6aeb/64 scope link 
       valid_lft forever preferred_lft forever

Then what I did was to spawn a VM on the network. It boots up, gets an IPv6 address in the 2001:cdba:9abc:5678::/64 network, and a default route pointing to the router's LLADDR (fe80...) (From ip -6 route). After roughly 10 seconds the default route just vanishes. If I remove 'garp_master_refresh' from keepalived.conf, SIGHUP keepalived and spawn another VM, it won't experience the same issue.
Comment 5 Ryan O'Hara 2016-01-19 15:10:05 EST
I am reluctant to backport this for the following reasons:

1. We have an alternative solution where we get rid of garp_master_repeat and garp_master_refresh and only use garp_master delay. I do not understand why garp_master_repeat and garp_master_delay cause the default IPv6 route to be deleted.

2. This upstream patch is not fully tested. I am concerned that always setting the router flag in an IPv6 neighbor advertisement will not work with forwarding is disabled.

3. Following on #2, I think setting the router flag should, at best, be conditional. This was even mentioned by the author of the patch in question.

4. I am unable to reproduce this bug in a basic, IPv6 keepalived configuration. This may be due to some differences in sysctl setting that control how router/neighbor advertisements and handled by the network stack. I've asked for  information about these settings in an environment where the problem exists.
Comment 6 mailinglists 2016-05-15 11:39:31 EDT
Any update on solving this issue?
We are are also hitting it and it's a blocker for IPv6.
If possible please push a fix.
It would be nice to have IPv6 HA working again, without building from sources manually from upstream (https://github.com/acassen/keepalived).

More info follows. Maybe you can compare sysctl options to your own now.
As soon as the keepalived sends out unsolicited neighbour advertisement, default route vanishes from clients routing table.
It doesn't matter if it's sent upon garp_master_refresh period or when the resource is taken over. It can not be fixed via config file.
If you are concerned, that always setting the router flag in an IPv6 neighbor advertisement will not work with forwarding is disabled, it can probably be worked around by looking at current system settings or by providing configuration option to disable router flag in the packet.

In the demo case below, our client is OSX 10.9.5 and Server is EL7 (CentOS) with keepalived (x86_64, 1.2.13, 7.el7).

Here is our config:
r1:
keepalived conf:
#112 vlan
vrrp_instance IPs {
	state BACKUP
	interface bond1.112
	native_ipv6
	unicast_peer {
		2001:XXXX:XXXX::112:0:2
	}
	unicast_src_ip 2001:XXXX:XXXX::112:0:3
	#garp_master_delay 4
	#garp_master_refresh 66
	virtual_router_id 112
	priority 99
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass XXXXXXXX
	}
	virtual_ipaddress {
		2001:XXXX:XXXX::112:0:1/96
		2001:XXXX:XXXX:dead:beef::1/96
	}
	smtp_alert
}
[root@r1 keepalived]# ip a l bond1.112
34: bond1.112@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:0f:53:0e:6c:e1 brd ff:ff:ff:ff:ff:ff
    inet6 2001:XXXX:XXXX:dead:beef::3/96 scope global 
       valid_lft forever preferred_lft forever
    inet6 2001:XXXX:XXXX::112:0:3/96 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::20f:53ff:fe0e:6ce1/64 scope link 
       valid_lft forever preferred_lft forever


r2:
keepalived conf:
#112 vlan
vrrp_instance IPs {
	state MASTER
	interface bond1.112
	native_ipv6
	unicast_peer {
		2001:XXXX:XXXX::112:0:3
	}
	unicast_src_ip 2001:XXXX:XXXX::112:0:2
	#garp_master_delay 4
	#garp_master_refresh 66
	virtual_router_id 112
	priority 100
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass XXXXXXXX
	}
	virtual_ipaddress {
		2001:XXXX:XXXX::112:0:1/96
		2001:XXXX:XXXX:dead:beef::1/96
	}
	smtp_alert
}
[root@r2 keepalived]# ip a l bond1.112
36: bond1.112@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:0f:53:0e:6c:dd brd ff:ff:ff:ff:ff:ff
    inet6 2001:XXXX:XXXX:dead:beef::1/96 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 2001:XXXX:XXXX::112:0:1/96 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 2001:XXXX:XXXX:dead:beef::2/96 scope global 
       valid_lft forever preferred_lft forever
    inet6 2001:XXXX:XXXX::112:0:2/96 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::20f:53ff:fe0e:6cdd/64 scope link 
       valid_lft forever preferred_lft forever

Client:
bash-3.2# netstat -rn -f inet6 | grep 2001:XXXX:XXXX:
default                                 2001:XXXX:XXXX:dead:beef::1      UGc           vlan2
2001:XXXX:XXXX:dead:beef::/96            link#8                          UC            vlan2
2001:XXXX:XXXX:dead:beef::1              0.f.53.e.6c.dd                  UHLWIir       vlan2
2001:XXXX:XXXX:dead:beef::2              link#8                          UHLWIr        vlan2
2001:XXXX:XXXX:dead:beef::3              0.f.53.e.6c.e1                  UHLWIr        vlan2
2001:XXXX:XXXX:dead:beef::666            0.23.32.ca.aa.b2                UHL             lo0

IPv6 sysctl settings from r1/r2 are on the bottom of bug report.



Here is how it looks upon shutting down keepalived on r2 which is master at the time.

r1 takes over (obviously):
May 15 17:01:52 r1 Keepalived_vrrp[16559]: VRRP_Instance(IPs) Entering MASTER STATE
May 15 17:01:52 r1 Keepalived_vrrp[16559]: VRRP_Instance(IPs) setting protocol VIPs.
May 15 17:01:52 r1 Keepalived_vrrp[16559]: VRRP_Instance(IPs) Sending Unsolicited Neighbour Adverts on bond1.112 for 2001:XXXX:XXXX::112:0:1
May 15 17:01:52 r1 Keepalived_vrrp[16559]: VRRP_Instance(IPs) Sending Unsolicited Neighbour Adverts on bond1.112 for 2001:XXXX:XXXX:dead:beef::1
May 15 17:01:52 r1 Keepalived_vrrp[16559]: Netlink reflector reports IP 2001:XXXX:XXXX::112:0:1 added
May 15 17:01:52 r1 Keepalived_vrrp[16559]: Netlink reflector reports IP 2001:XXXX:XXXX:dead:beef::1 added
May 15 17:01:57 r1 Keepalived_vrrp[16559]: VRRP_Instance(IPs) Sending Unsolicited Neighbour Adverts on bond1.112 for 2001:XXXX:XXXX::112:0:1
May 15 17:01:57 r1 Keepalived_vrrp[16559]: VRRP_Instance(IPs) Sending Unsolicited Neighbour Adverts on bond1.112 for 2001:XXXX:XXXX:dead:beef::1

Here we can see the advertisements sent out by keepalived on r1. I have left other possibly relevant stuff in:
17:01:49.906909 IP6 2001:XXXX:XXXX::112:0:2 > 2001:XXXX:XXXX::112:0:3: ip-proto-112 8
17:01:50.426375 IP6 2001:XXXX:XXXX::112:0:2 > 2001:XXXX:XXXX::112:0:3: ip-proto-112 8
17:01:50.440748 IP6 fe80::20f:53ff:fe0e:6cdd > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:01:50.467244 IP6 fe80::20f:53ff:fe0e:6cdd > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:01:51.039889 IP6 2001:XXXX:XXXX::112:0:3 > 2001:XXXX:XXXX::112:0:2: ip-proto-112 8
17:01:51.039991 IP6 2001:XXXX:XXXX::112:0:2 > 2001:XXXX:XXXX::112:0:3: ICMP6, parameter problem, next header - octet 6, length 56
17:01:52.064536 IP6 2001:XXXX:XXXX::112:0:1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX::112:0:1, length 32
17:01:52.064561 IP6 2001:XXXX:XXXX:dead:beef::1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:dead:beef::1, length 32
17:01:52.064572 IP6 2001:XXXX:XXXX::112:0:1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX::112:0:1, length 32
17:01:52.064577 IP6 2001:XXXX:XXXX:dead:beef::1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:dead:beef::1, length 32
17:01:52.064581 IP6 2001:XXXX:XXXX::112:0:1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX::112:0:1, length 32
17:01:52.064586 IP6 2001:XXXX:XXXX:dead:beef::1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:dead:beef::1, length 32
17:01:52.064590 IP6 2001:XXXX:XXXX::112:0:1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX::112:0:1, length 32
17:01:52.064595 IP6 2001:XXXX:XXXX:dead:beef::1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:dead:beef::1, length 32
17:01:52.064599 IP6 2001:XXXX:XXXX::112:0:1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX::112:0:1, length 32
17:01:52.064604 IP6 2001:XXXX:XXXX:dead:beef::1 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:XXXX:XXXX:dead:beef::1, length 32
17:01:52.064632 IP6 2001:XXXX:XXXX::112:0:3 > 2001:XXXX:XXXX::112:0:2: ip-proto-112 8
17:01:52.064680 IP6 2001:XXXX:XXXX::112:0:2 > 2001:XXXX:XXXX::112:0:3: ICMP6, parameter problem, next header - octet 6, length 56
17:01:52.066843 IP6 fe80::20f:53ff:fe0e:6ce1 > ff02::16: HBH ICMP6, multicast listener report v2, 5 group record(s), length 108
17:01:52.535871 IP6 fe80::20f:53ff:fe0e:6ce1 > ff02::16: HBH ICMP6, multicast listener report v2, 5 group record(s), length 108
17:01:53.064863 IP6 2001:XXXX:XXXX::112:0:3 > 2001:XXXX:XXXX::112:0:2: ip-proto-112 8
17:01:53.064983 IP6 2001:XXXX:XXXX::112:0:2 > 2001:XXXX:XXXX::112:0:3: ICMP6, parameter problem, next header - octet 6, length 56
17:01:54.065042 IP6 2001:XXXX:XXXX::112:0:3 > 2001:XXXX:XXXX::112:0:2: ip-proto-112 8
17:01:54.065156 IP6 2001:XXXX:XXXX::112:0:2 > 2001:XXXX:XXXX::112:0:3: ICMP6, parameter problem, next header - octet 6, length 56
17:01:55.065202 IP6 2001:XXXX:XXXX::112:0:3 > 2001:XXXX:XXXX::112:0:2: ip-proto-112 8

And voila our client lost IPv6 network connectivity (default route went missing):
bash-3.2# netstat -rn -f inet6 | grep 2001:XXXX:XXXX:
2001:XXXX:XXXX:dead:beef::/96            link#8                          UC            vlan2
2001:XXXX:XXXX:dead:beef::1              0.f.53.e.6c.e1                  UHLWIir       vlan2
2001:XXXX:XXXX:dead:beef::2              link#8                          UHLWIr        vlan2
2001:XXXX:XXXX:dead:beef::3              0.f.53.e.6c.e1                  UHLWIr        vlan2
2001:XXXX:XXXX:dead:beef::666            0.23.32.ca.aa.b2                UHL             lo0
bash-3.2# ping6 2a00:1450:400c:c04::64
ping6: UDP connect: No route to host



 
 
sysctl settings as promised:
 
 [root@r1 keepalived]# sysctl -a | grep ipv6
net.ipv6.bindv6only = 0
net.ipv6.conf.all.accept_dad = 1
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.all.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.all.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.optimistic_dad = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_probe_interval = 60
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitations = 3
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.bond0.accept_dad = 1
net.ipv6.conf.bond0.accept_ra = 0
net.ipv6.conf.bond0.accept_ra_defrtr = 0
net.ipv6.conf.bond0.accept_ra_pinfo = 0
net.ipv6.conf.bond0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.bond0.accept_ra_rtr_pref = 0
net.ipv6.conf.bond0.accept_redirects = 0
net.ipv6.conf.bond0.accept_source_route = 0
net.ipv6.conf.bond0.autoconf = 0
net.ipv6.conf.bond0.dad_transmits = 1
net.ipv6.conf.bond0.disable_ipv6 = 0
net.ipv6.conf.bond0.force_mld_version = 0
net.ipv6.conf.bond0.force_tllao = 0
net.ipv6.conf.bond0.forwarding = 1
net.ipv6.conf.bond0.hop_limit = 64
net.ipv6.conf.bond0.max_addresses = 16
net.ipv6.conf.bond0.max_desync_factor = 600
net.ipv6.conf.bond0.mc_forwarding = 0
net.ipv6.conf.bond0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.bond0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.bond0.mtu = 1500
net.ipv6.conf.bond0.ndisc_notify = 0
net.ipv6.conf.bond0.optimistic_dad = 0
net.ipv6.conf.bond0.proxy_ndp = 0
net.ipv6.conf.bond0.regen_max_retry = 3
net.ipv6.conf.bond0.router_probe_interval = 60
net.ipv6.conf.bond0.router_solicitation_delay = 1
net.ipv6.conf.bond0.router_solicitation_interval = 4
net.ipv6.conf.bond0.router_solicitations = 0
net.ipv6.conf.bond0.temp_prefered_lft = 86400
net.ipv6.conf.bond0.temp_valid_lft = 604800
net.ipv6.conf.bond0.use_tempaddr = 0
net.ipv6.conf.bond1.accept_dad = 1
net.ipv6.conf.bond1.accept_ra = 0
net.ipv6.conf.bond1.accept_ra_defrtr = 0
net.ipv6.conf.bond1.accept_ra_pinfo = 0
net.ipv6.conf.bond1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.bond1.accept_ra_rtr_pref = 0
net.ipv6.conf.bond1.accept_redirects = 0
net.ipv6.conf.bond1.accept_source_route = 0
net.ipv6.conf.bond1.autoconf = 0
net.ipv6.conf.bond1.dad_transmits = 1
net.ipv6.conf.bond1.disable_ipv6 = 0
net.ipv6.conf.bond1.force_mld_version = 0
net.ipv6.conf.bond1.force_tllao = 0
net.ipv6.conf.bond1.forwarding = 1
net.ipv6.conf.bond1.hop_limit = 64
net.ipv6.conf.bond1.max_addresses = 16
net.ipv6.conf.bond1.max_desync_factor = 600
net.ipv6.conf.bond1.mc_forwarding = 0
net.ipv6.conf.bond1.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.bond1.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.bond1.mtu = 1500
net.ipv6.conf.bond1.ndisc_notify = 0
net.ipv6.conf.bond1.optimistic_dad = 0
net.ipv6.conf.bond1.proxy_ndp = 0
net.ipv6.conf.bond1.regen_max_retry = 3
net.ipv6.conf.bond1.router_probe_interval = 60
net.ipv6.conf.bond1.router_solicitation_delay = 1
net.ipv6.conf.bond1.router_solicitation_interval = 4
net.ipv6.conf.bond1.router_solicitations = 0
net.ipv6.conf.bond1.temp_prefered_lft = 86400
net.ipv6.conf.bond1.temp_valid_lft = 604800
net.ipv6.conf.bond1.use_tempaddr = 0
net.ipv6.conf.bond1/112.accept_dad = 1
net.ipv6.conf.bond1/112.accept_ra = 0
net.ipv6.conf.bond1/112.accept_ra_defrtr = 0
net.ipv6.conf.bond1/112.accept_ra_pinfo = 0
net.ipv6.conf.bond1/112.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.bond1/112.accept_ra_rtr_pref = 0
net.ipv6.conf.bond1/112.accept_redirects = 0
net.ipv6.conf.bond1/112.accept_source_route = 0
net.ipv6.conf.bond1/112.autoconf = 0
net.ipv6.conf.bond1/112.dad_transmits = 1
net.ipv6.conf.bond1/112.disable_ipv6 = 0
net.ipv6.conf.bond1/112.force_mld_version = 0
net.ipv6.conf.bond1/112.force_tllao = 0
net.ipv6.conf.bond1/112.forwarding = 1
net.ipv6.conf.bond1/112.hop_limit = 64
net.ipv6.conf.bond1/112.max_addresses = 16
net.ipv6.conf.bond1/112.max_desync_factor = 600
net.ipv6.conf.bond1/112.mc_forwarding = 0
net.ipv6.conf.bond1/112.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.bond1/112.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.bond1/112.mtu = 1500
net.ipv6.conf.bond1/112.ndisc_notify = 0
net.ipv6.conf.bond1/112.optimistic_dad = 0
net.ipv6.conf.bond1/112.proxy_ndp = 0
net.ipv6.conf.bond1/112.regen_max_retry = 3
net.ipv6.conf.bond1/112.router_probe_interval = 60
net.ipv6.conf.bond1/112.router_solicitation_delay = 1
net.ipv6.conf.bond1/112.router_solicitation_interval = 4
net.ipv6.conf.bond1/112.router_solicitations = 0
net.ipv6.conf.bond1/112.temp_prefered_lft = 86400
net.ipv6.conf.bond1/112.temp_valid_lft = 604800
net.ipv6.conf.bond1/112.use_tempaddr = 0
net.ipv6.conf.bond1/117.accept_dad = 1
net.ipv6.conf.bond1/117.accept_ra = 0
net.ipv6.conf.bond1/117.accept_ra_defrtr = 0
net.ipv6.conf.bond1/117.accept_ra_pinfo = 0
net.ipv6.conf.bond1/117.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.bond1/117.accept_ra_rtr_pref = 0
net.ipv6.conf.bond1/117.accept_redirects = 0
net.ipv6.conf.bond1/117.accept_source_route = 0
net.ipv6.conf.bond1/117.autoconf = 0
net.ipv6.conf.bond1/117.dad_transmits = 1
net.ipv6.conf.bond1/117.disable_ipv6 = 0
net.ipv6.conf.bond1/117.force_mld_version = 0
net.ipv6.conf.bond1/117.force_tllao = 0
net.ipv6.conf.bond1/117.forwarding = 1
net.ipv6.conf.bond1/117.hop_limit = 64
net.ipv6.conf.bond1/117.max_addresses = 16
net.ipv6.conf.bond1/117.max_desync_factor = 600
net.ipv6.conf.bond1/117.mc_forwarding = 0
net.ipv6.conf.bond1/117.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.bond1/117.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.bond1/117.mtu = 1500
net.ipv6.conf.bond1/117.ndisc_notify = 0
net.ipv6.conf.bond1/117.optimistic_dad = 0
net.ipv6.conf.bond1/117.proxy_ndp = 0
net.ipv6.conf.bond1/117.regen_max_retry = 3
net.ipv6.conf.bond1/117.router_probe_interval = 60
net.ipv6.conf.bond1/117.router_solicitation_delay = 1
net.ipv6.conf.bond1/117.router_solicitation_interval = 4
net.ipv6.conf.bond1/117.router_solicitations = 0
net.ipv6.conf.bond1/117.temp_prefered_lft = 86400
net.ipv6.conf.bond1/117.temp_valid_lft = 604800
net.ipv6.conf.bond1/117.use_tempaddr = 0
net.ipv6.conf.bond1/118.accept_dad = 1
net.ipv6.conf.bond1/118.accept_ra = 0
net.ipv6.conf.bond1/118.accept_ra_defrtr = 0
net.ipv6.conf.bond1/118.accept_ra_pinfo = 0
net.ipv6.conf.bond1/118.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.bond1/118.accept_ra_rtr_pref = 0
net.ipv6.conf.bond1/118.accept_redirects = 0
net.ipv6.conf.bond1/118.accept_source_route = 0
net.ipv6.conf.bond1/118.autoconf = 0
net.ipv6.conf.bond1/118.dad_transmits = 1
net.ipv6.conf.bond1/118.disable_ipv6 = 0
net.ipv6.conf.bond1/118.force_mld_version = 0
net.ipv6.conf.bond1/118.force_tllao = 0
net.ipv6.conf.bond1/118.forwarding = 1
net.ipv6.conf.bond1/118.hop_limit = 64
net.ipv6.conf.bond1/118.max_addresses = 16
net.ipv6.conf.bond1/118.max_desync_factor = 600
net.ipv6.conf.bond1/118.mc_forwarding = 0
net.ipv6.conf.bond1/118.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.bond1/118.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.bond1/118.mtu = 1500
net.ipv6.conf.bond1/118.ndisc_notify = 0
net.ipv6.conf.bond1/118.optimistic_dad = 0
net.ipv6.conf.bond1/118.proxy_ndp = 0
net.ipv6.conf.bond1/118.regen_max_retry = 3
net.ipv6.conf.bond1/118.router_probe_interval = 60
net.ipv6.conf.bond1/118.router_solicitation_delay = 1
net.ipv6.conf.bond1/118.router_solicitation_interval = 4
net.ipv6.conf.bond1/118.router_solicitations = 0
net.ipv6.conf.bond1/118.temp_prefered_lft = 86400
net.ipv6.conf.bond1/118.temp_valid_lft = 604800
net.ipv6.conf.bond1/118.use_tempaddr = 0
net.ipv6.conf.bond1/96.accept_dad = 1
net.ipv6.conf.bond1/96.accept_ra = 0
net.ipv6.conf.bond1/96.accept_ra_defrtr = 0
net.ipv6.conf.bond1/96.accept_ra_pinfo = 0
net.ipv6.conf.bond1/96.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.bond1/96.accept_ra_rtr_pref = 0
net.ipv6.conf.bond1/96.accept_redirects = 0
net.ipv6.conf.bond1/96.accept_source_route = 0
net.ipv6.conf.bond1/96.autoconf = 0
net.ipv6.conf.bond1/96.dad_transmits = 1
net.ipv6.conf.bond1/96.disable_ipv6 = 0
net.ipv6.conf.bond1/96.force_mld_version = 0
net.ipv6.conf.bond1/96.force_tllao = 0
net.ipv6.conf.bond1/96.forwarding = 1
net.ipv6.conf.bond1/96.hop_limit = 64
net.ipv6.conf.bond1/96.max_addresses = 16
net.ipv6.conf.bond1/96.max_desync_factor = 600
net.ipv6.conf.bond1/96.mc_forwarding = 0
net.ipv6.conf.bond1/96.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.bond1/96.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.bond1/96.mtu = 1500
net.ipv6.conf.bond1/96.ndisc_notify = 0
net.ipv6.conf.bond1/96.optimistic_dad = 0
net.ipv6.conf.bond1/96.proxy_ndp = 0
net.ipv6.conf.bond1/96.regen_max_retry = 3
net.ipv6.conf.bond1/96.router_probe_interval = 60
net.ipv6.conf.bond1/96.router_solicitation_delay = 1
net.ipv6.conf.bond1/96.router_solicitation_interval = 4
net.ipv6.conf.bond1/96.router_solicitations = 0
net.ipv6.conf.bond1/96.temp_prefered_lft = 86400
net.ipv6.conf.bond1/96.temp_valid_lft = 604800
net.ipv6.conf.bond1/96.use_tempaddr = 0
net.ipv6.conf.bond1/97.accept_dad = 1
net.ipv6.conf.bond1/97.accept_ra = 0
net.ipv6.conf.bond1/97.accept_ra_defrtr = 0
net.ipv6.conf.bond1/97.accept_ra_pinfo = 0
net.ipv6.conf.bond1/97.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.bond1/97.accept_ra_rtr_pref = 0
net.ipv6.conf.bond1/97.accept_redirects = 0
net.ipv6.conf.bond1/97.accept_source_route = 0
net.ipv6.conf.bond1/97.autoconf = 0
net.ipv6.conf.bond1/97.dad_transmits = 1
net.ipv6.conf.bond1/97.disable_ipv6 = 0
net.ipv6.conf.bond1/97.force_mld_version = 0
net.ipv6.conf.bond1/97.force_tllao = 0
net.ipv6.conf.bond1/97.forwarding = 1
net.ipv6.conf.bond1/97.hop_limit = 64
net.ipv6.conf.bond1/97.max_addresses = 16
net.ipv6.conf.bond1/97.max_desync_factor = 600
net.ipv6.conf.bond1/97.mc_forwarding = 0
net.ipv6.conf.bond1/97.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.bond1/97.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.bond1/97.mtu = 1500
net.ipv6.conf.bond1/97.ndisc_notify = 0
net.ipv6.conf.bond1/97.optimistic_dad = 0
net.ipv6.conf.bond1/97.proxy_ndp = 0
net.ipv6.conf.bond1/97.regen_max_retry = 3
net.ipv6.conf.bond1/97.router_probe_interval = 60
net.ipv6.conf.bond1/97.router_solicitation_delay = 1
net.ipv6.conf.bond1/97.router_solicitation_interval = 4
net.ipv6.conf.bond1/97.router_solicitations = 0
net.ipv6.conf.bond1/97.temp_prefered_lft = 86400
net.ipv6.conf.bond1/97.temp_valid_lft = 604800
net.ipv6.conf.bond1/97.use_tempaddr = 0
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.default.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.default.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.optimistic_dad = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_probe_interval = 60
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitations = 0
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_tempaddr = 0
net.ipv6.conf.eno1.accept_dad = 1
net.ipv6.conf.eno1.accept_ra = 0
net.ipv6.conf.eno1.accept_ra_defrtr = 0
net.ipv6.conf.eno1.accept_ra_pinfo = 0
net.ipv6.conf.eno1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.eno1.accept_ra_rtr_pref = 0
net.ipv6.conf.eno1.accept_redirects = 0
net.ipv6.conf.eno1.accept_source_route = 0
net.ipv6.conf.eno1.autoconf = 0
net.ipv6.conf.eno1.dad_transmits = 1
net.ipv6.conf.eno1.disable_ipv6 = 0
net.ipv6.conf.eno1.force_mld_version = 0
net.ipv6.conf.eno1.force_tllao = 0
net.ipv6.conf.eno1.forwarding = 1
net.ipv6.conf.eno1.hop_limit = 64
net.ipv6.conf.eno1.max_addresses = 16
net.ipv6.conf.eno1.max_desync_factor = 600
net.ipv6.conf.eno1.mc_forwarding = 0
net.ipv6.conf.eno1.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.eno1.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.eno1.mtu = 1500
net.ipv6.conf.eno1.ndisc_notify = 0
net.ipv6.conf.eno1.optimistic_dad = 0
net.ipv6.conf.eno1.proxy_ndp = 0
net.ipv6.conf.eno1.regen_max_retry = 3
net.ipv6.conf.eno1.router_probe_interval = 60
net.ipv6.conf.eno1.router_solicitation_delay = 1
net.ipv6.conf.eno1.router_solicitation_interval = 4
net.ipv6.conf.eno1.router_solicitations = 0
net.ipv6.conf.eno1.temp_prefered_lft = 86400
net.ipv6.conf.eno1.temp_valid_lft = 604800
net.ipv6.conf.eno1.use_tempaddr = 0
net.ipv6.conf.eno2.accept_dad = 1
net.ipv6.conf.eno2.accept_ra = 0
net.ipv6.conf.eno2.accept_ra_defrtr = 0
net.ipv6.conf.eno2.accept_ra_pinfo = 0
net.ipv6.conf.eno2.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.eno2.accept_ra_rtr_pref = 0
net.ipv6.conf.eno2.accept_redirects = 0
net.ipv6.conf.eno2.accept_source_route = 0
net.ipv6.conf.eno2.autoconf = 0
net.ipv6.conf.eno2.dad_transmits = 1
net.ipv6.conf.eno2.disable_ipv6 = 0
net.ipv6.conf.eno2.force_mld_version = 0
net.ipv6.conf.eno2.force_tllao = 0
net.ipv6.conf.eno2.forwarding = 1
net.ipv6.conf.eno2.hop_limit = 64
net.ipv6.conf.eno2.max_addresses = 16
net.ipv6.conf.eno2.max_desync_factor = 600
net.ipv6.conf.eno2.mc_forwarding = 0
net.ipv6.conf.eno2.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.eno2.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.eno2.mtu = 1500
net.ipv6.conf.eno2.ndisc_notify = 0
net.ipv6.conf.eno2.optimistic_dad = 0
net.ipv6.conf.eno2.proxy_ndp = 0
net.ipv6.conf.eno2.regen_max_retry = 3
net.ipv6.conf.eno2.router_probe_interval = 60
net.ipv6.conf.eno2.router_solicitation_delay = 1
net.ipv6.conf.eno2.router_solicitation_interval = 4
net.ipv6.conf.eno2.router_solicitations = 0
net.ipv6.conf.eno2.temp_prefered_lft = 86400
net.ipv6.conf.eno2.temp_valid_lft = 604800
net.ipv6.conf.eno2.use_tempaddr = 0
net.ipv6.conf.enp1s0f0.accept_dad = 1
net.ipv6.conf.enp1s0f0.accept_ra = 0
net.ipv6.conf.enp1s0f0.accept_ra_defrtr = 0
net.ipv6.conf.enp1s0f0.accept_ra_pinfo = 0
net.ipv6.conf.enp1s0f0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp1s0f0.accept_ra_rtr_pref = 0
net.ipv6.conf.enp1s0f0.accept_redirects = 0
net.ipv6.conf.enp1s0f0.accept_source_route = 0
net.ipv6.conf.enp1s0f0.autoconf = 0
net.ipv6.conf.enp1s0f0.dad_transmits = 1
net.ipv6.conf.enp1s0f0.disable_ipv6 = 0
net.ipv6.conf.enp1s0f0.force_mld_version = 0
net.ipv6.conf.enp1s0f0.force_tllao = 0
net.ipv6.conf.enp1s0f0.forwarding = 1
net.ipv6.conf.enp1s0f0.hop_limit = 64
net.ipv6.conf.enp1s0f0.max_addresses = 16
net.ipv6.conf.enp1s0f0.max_desync_factor = 600
net.ipv6.conf.enp1s0f0.mc_forwarding = 0
net.ipv6.conf.enp1s0f0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp1s0f0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp1s0f0.mtu = 1500
net.ipv6.conf.enp1s0f0.ndisc_notify = 0
net.ipv6.conf.enp1s0f0.optimistic_dad = 0
net.ipv6.conf.enp1s0f0.proxy_ndp = 0
net.ipv6.conf.enp1s0f0.regen_max_retry = 3
net.ipv6.conf.enp1s0f0.router_probe_interval = 60
net.ipv6.conf.enp1s0f0.router_solicitation_delay = 1
net.ipv6.conf.enp1s0f0.router_solicitation_interval = 4
net.ipv6.conf.enp1s0f0.router_solicitations = 0
net.ipv6.conf.enp1s0f0.temp_prefered_lft = 86400
net.ipv6.conf.enp1s0f0.temp_valid_lft = 604800
net.ipv6.conf.enp1s0f0.use_tempaddr = 0
net.ipv6.conf.enp1s0f1.accept_dad = 1
net.ipv6.conf.enp1s0f1.accept_ra = 0
net.ipv6.conf.enp1s0f1.accept_ra_defrtr = 0
net.ipv6.conf.enp1s0f1.accept_ra_pinfo = 0
net.ipv6.conf.enp1s0f1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp1s0f1.accept_ra_rtr_pref = 0
net.ipv6.conf.enp1s0f1.accept_redirects = 0
net.ipv6.conf.enp1s0f1.accept_source_route = 0
net.ipv6.conf.enp1s0f1.autoconf = 0
net.ipv6.conf.enp1s0f1.dad_transmits = 1
net.ipv6.conf.enp1s0f1.disable_ipv6 = 0
net.ipv6.conf.enp1s0f1.force_mld_version = 0
net.ipv6.conf.enp1s0f1.force_tllao = 0
net.ipv6.conf.enp1s0f1.forwarding = 1
net.ipv6.conf.enp1s0f1.hop_limit = 64
net.ipv6.conf.enp1s0f1.max_addresses = 16
net.ipv6.conf.enp1s0f1.max_desync_factor = 600
net.ipv6.conf.enp1s0f1.mc_forwarding = 0
net.ipv6.conf.enp1s0f1.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp1s0f1.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp1s0f1.mtu = 1500
net.ipv6.conf.enp1s0f1.ndisc_notify = 0
net.ipv6.conf.enp1s0f1.optimistic_dad = 0
net.ipv6.conf.enp1s0f1.proxy_ndp = 0
net.ipv6.conf.enp1s0f1.regen_max_retry = 3
net.ipv6.conf.enp1s0f1.router_probe_interval = 60
net.ipv6.conf.enp1s0f1.router_solicitation_delay = 1
net.ipv6.conf.enp1s0f1.router_solicitation_interval = 4
net.ipv6.conf.enp1s0f1.router_solicitations = 0
net.ipv6.conf.enp1s0f1.temp_prefered_lft = 86400
net.ipv6.conf.enp1s0f1.temp_valid_lft = 604800
net.ipv6.conf.enp1s0f1.use_tempaddr = 0
net.ipv6.conf.enp4s0f0.accept_dad = 1
net.ipv6.conf.enp4s0f0.accept_ra = 0
net.ipv6.conf.enp4s0f0.accept_ra_defrtr = 0
net.ipv6.conf.enp4s0f0.accept_ra_pinfo = 0
net.ipv6.conf.enp4s0f0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp4s0f0.accept_ra_rtr_pref = 0
net.ipv6.conf.enp4s0f0.accept_redirects = 0
net.ipv6.conf.enp4s0f0.accept_source_route = 0
net.ipv6.conf.enp4s0f0.autoconf = 0
net.ipv6.conf.enp4s0f0.dad_transmits = 1
net.ipv6.conf.enp4s0f0.disable_ipv6 = 0
net.ipv6.conf.enp4s0f0.force_mld_version = 0
net.ipv6.conf.enp4s0f0.force_tllao = 0
net.ipv6.conf.enp4s0f0.forwarding = 1
net.ipv6.conf.enp4s0f0.hop_limit = 64
net.ipv6.conf.enp4s0f0.max_addresses = 16
net.ipv6.conf.enp4s0f0.max_desync_factor = 600
net.ipv6.conf.enp4s0f0.mc_forwarding = 0
net.ipv6.conf.enp4s0f0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp4s0f0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp4s0f0.mtu = 1500
net.ipv6.conf.enp4s0f0.ndisc_notify = 0
net.ipv6.conf.enp4s0f0.optimistic_dad = 0
net.ipv6.conf.enp4s0f0.proxy_ndp = 0
net.ipv6.conf.enp4s0f0.regen_max_retry = 3
net.ipv6.conf.enp4s0f0.router_probe_interval = 60
net.ipv6.conf.enp4s0f0.router_solicitation_delay = 1
net.ipv6.conf.enp4s0f0.router_solicitation_interval = 4
net.ipv6.conf.enp4s0f0.router_solicitations = 0
net.ipv6.conf.enp4s0f0.temp_prefered_lft = 86400
net.ipv6.conf.enp4s0f0.temp_valid_lft = 604800
net.ipv6.conf.enp4s0f0.use_tempaddr = 0
net.ipv6.conf.enp4s0f1.accept_dad = 1
net.ipv6.conf.enp4s0f1.accept_ra = 0
net.ipv6.conf.enp4s0f1.accept_ra_defrtr = 0
net.ipv6.conf.enp4s0f1.accept_ra_pinfo = 0
net.ipv6.conf.enp4s0f1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp4s0f1.accept_ra_rtr_pref = 0
net.ipv6.conf.enp4s0f1.accept_redirects = 0
net.ipv6.conf.enp4s0f1.accept_source_route = 0
net.ipv6.conf.enp4s0f1.autoconf = 0
net.ipv6.conf.enp4s0f1.dad_transmits = 1
net.ipv6.conf.enp4s0f1.disable_ipv6 = 0
net.ipv6.conf.enp4s0f1.force_mld_version = 0
net.ipv6.conf.enp4s0f1.force_tllao = 0
net.ipv6.conf.enp4s0f1.forwarding = 1
net.ipv6.conf.enp4s0f1.hop_limit = 64
net.ipv6.conf.enp4s0f1.max_addresses = 16
net.ipv6.conf.enp4s0f1.max_desync_factor = 600
net.ipv6.conf.enp4s0f1.mc_forwarding = 0
net.ipv6.conf.enp4s0f1.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp4s0f1.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp4s0f1.mtu = 1500
net.ipv6.conf.enp4s0f1.ndisc_notify = 0
net.ipv6.conf.enp4s0f1.optimistic_dad = 0
net.ipv6.conf.enp4s0f1.proxy_ndp = 0
net.ipv6.conf.enp4s0f1.regen_max_retry = 3
net.ipv6.conf.enp4s0f1.router_probe_interval = 60
net.ipv6.conf.enp4s0f1.router_solicitation_delay = 1
net.ipv6.conf.enp4s0f1.router_solicitation_interval = 4
net.ipv6.conf.enp4s0f1.router_solicitations = 0
net.ipv6.conf.enp4s0f1.temp_prefered_lft = 86400
net.ipv6.conf.enp4s0f1.temp_valid_lft = 604800
net.ipv6.conf.enp4s0f1.use_tempaddr = 0
net.ipv6.conf.enp6s0f0.accept_dad = 1
net.ipv6.conf.enp6s0f0.accept_ra = 0
net.ipv6.conf.enp6s0f0.accept_ra_defrtr = 0
net.ipv6.conf.enp6s0f0.accept_ra_pinfo = 0
net.ipv6.conf.enp6s0f0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp6s0f0.accept_ra_rtr_pref = 0
net.ipv6.conf.enp6s0f0.accept_redirects = 0
net.ipv6.conf.enp6s0f0.accept_source_route = 0
net.ipv6.conf.enp6s0f0.autoconf = 0
net.ipv6.conf.enp6s0f0.dad_transmits = 1
net.ipv6.conf.enp6s0f0.disable_ipv6 = 0
net.ipv6.conf.enp6s0f0.force_mld_version = 0
net.ipv6.conf.enp6s0f0.force_tllao = 0
net.ipv6.conf.enp6s0f0.forwarding = 1
net.ipv6.conf.enp6s0f0.hop_limit = 64
net.ipv6.conf.enp6s0f0.max_addresses = 16
net.ipv6.conf.enp6s0f0.max_desync_factor = 600
net.ipv6.conf.enp6s0f0.mc_forwarding = 0
net.ipv6.conf.enp6s0f0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp6s0f0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp6s0f0.mtu = 1500
net.ipv6.conf.enp6s0f0.ndisc_notify = 0
net.ipv6.conf.enp6s0f0.optimistic_dad = 0
net.ipv6.conf.enp6s0f0.proxy_ndp = 0
net.ipv6.conf.enp6s0f0.regen_max_retry = 3
net.ipv6.conf.enp6s0f0.router_probe_interval = 60
net.ipv6.conf.enp6s0f0.router_solicitation_delay = 1
net.ipv6.conf.enp6s0f0.router_solicitation_interval = 4
net.ipv6.conf.enp6s0f0.router_solicitations = 0
net.ipv6.conf.enp6s0f0.temp_prefered_lft = 86400
net.ipv6.conf.enp6s0f0.temp_valid_lft = 604800
net.ipv6.conf.enp6s0f0.use_tempaddr = 0
net.ipv6.conf.enp6s0f1.accept_dad = 1
net.ipv6.conf.enp6s0f1.accept_ra = 0
net.ipv6.conf.enp6s0f1.accept_ra_defrtr = 0
net.ipv6.conf.enp6s0f1.accept_ra_pinfo = 0
net.ipv6.conf.enp6s0f1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp6s0f1.accept_ra_rtr_pref = 0
net.ipv6.conf.enp6s0f1.accept_redirects = 0
net.ipv6.conf.enp6s0f1.accept_source_route = 0
net.ipv6.conf.enp6s0f1.autoconf = 0
net.ipv6.conf.enp6s0f1.dad_transmits = 1
net.ipv6.conf.enp6s0f1.disable_ipv6 = 0
net.ipv6.conf.enp6s0f1.force_mld_version = 0
net.ipv6.conf.enp6s0f1.force_tllao = 0
net.ipv6.conf.enp6s0f1.forwarding = 1
net.ipv6.conf.enp6s0f1.hop_limit = 64
net.ipv6.conf.enp6s0f1.max_addresses = 16
net.ipv6.conf.enp6s0f1.max_desync_factor = 600
net.ipv6.conf.enp6s0f1.mc_forwarding = 0
net.ipv6.conf.enp6s0f1.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp6s0f1.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp6s0f1.mtu = 1500
net.ipv6.conf.enp6s0f1.ndisc_notify = 0
net.ipv6.conf.enp6s0f1.optimistic_dad = 0
net.ipv6.conf.enp6s0f1.proxy_ndp = 0
net.ipv6.conf.enp6s0f1.regen_max_retry = 3
net.ipv6.conf.enp6s0f1.router_probe_interval = 60
net.ipv6.conf.enp6s0f1.router_solicitation_delay = 1
net.ipv6.conf.enp6s0f1.router_solicitation_interval = 4
net.ipv6.conf.enp6s0f1.router_solicitations = 0
net.ipv6.conf.enp6s0f1.temp_prefered_lft = 86400
net.ipv6.conf.enp6s0f1.temp_valid_lft = 604800
net.ipv6.conf.enp6s0f1.use_tempaddr = 0
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 0
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.lo.accept_ra_rtr_pref = 1
net.ipv6.conf.lo.accept_redirects = 0
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.autoconf = 0
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 1
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.lo.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.lo.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.optimistic_dad = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_probe_interval = 60
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitations = 3
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.icmp.ratelimit = 1000
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 600
net.ipv6.ip6frag_time = 60
net.ipv6.mld_max_msf = 64
net.ipv6.mld_qrv = 2
net.ipv6.neigh.bond0.anycast_delay = 100
net.ipv6.neigh.bond0.app_solicit = 0
net.ipv6.neigh.bond0.base_reachable_time_ms = 30000
net.ipv6.neigh.bond0.delay_first_probe_time = 5
net.ipv6.neigh.bond0.gc_stale_time = 60
net.ipv6.neigh.bond0.locktime = 0
net.ipv6.neigh.bond0.mcast_solicit = 3
net.ipv6.neigh.bond0.proxy_delay = 80
net.ipv6.neigh.bond0.proxy_qlen = 64
net.ipv6.neigh.bond0.retrans_time_ms = 1000
net.ipv6.neigh.bond0.ucast_solicit = 3
net.ipv6.neigh.bond0.unres_qlen = 31
net.ipv6.neigh.bond0.unres_qlen_bytes = 65536
net.ipv6.neigh.bond1.anycast_delay = 100
net.ipv6.neigh.bond1.app_solicit = 0
net.ipv6.neigh.bond1.base_reachable_time_ms = 30000
net.ipv6.neigh.bond1.delay_first_probe_time = 5
net.ipv6.neigh.bond1.gc_stale_time = 60
net.ipv6.neigh.bond1.locktime = 0
net.ipv6.neigh.bond1.mcast_solicit = 3
net.ipv6.neigh.bond1.proxy_delay = 80
net.ipv6.neigh.bond1.proxy_qlen = 64
net.ipv6.neigh.bond1.retrans_time_ms = 1000
net.ipv6.neigh.bond1.ucast_solicit = 3
net.ipv6.neigh.bond1.unres_qlen = 31
net.ipv6.neigh.bond1.unres_qlen_bytes = 65536
net.ipv6.neigh.bond1/112.anycast_delay = 100
net.ipv6.neigh.bond1/112.app_solicit = 0
net.ipv6.neigh.bond1/112.base_reachable_time_ms = 30000
net.ipv6.neigh.bond1/112.delay_first_probe_time = 5
net.ipv6.neigh.bond1/112.gc_stale_time = 60
net.ipv6.neigh.bond1/112.locktime = 0
net.ipv6.neigh.bond1/112.mcast_solicit = 3
net.ipv6.neigh.bond1/112.proxy_delay = 80
net.ipv6.neigh.bond1/112.proxy_qlen = 64
net.ipv6.neigh.bond1/112.retrans_time_ms = 1000
net.ipv6.neigh.bond1/112.ucast_solicit = 3
net.ipv6.neigh.bond1/112.unres_qlen = 31
net.ipv6.neigh.bond1/112.unres_qlen_bytes = 65536
net.ipv6.neigh.bond1/117.anycast_delay = 100
net.ipv6.neigh.bond1/117.app_solicit = 0
net.ipv6.neigh.bond1/117.base_reachable_time_ms = 30000
net.ipv6.neigh.bond1/117.delay_first_probe_time = 5
net.ipv6.neigh.bond1/117.gc_stale_time = 60
net.ipv6.neigh.bond1/117.locktime = 0
net.ipv6.neigh.bond1/117.mcast_solicit = 3
net.ipv6.neigh.bond1/117.proxy_delay = 80
net.ipv6.neigh.bond1/117.proxy_qlen = 64
net.ipv6.neigh.bond1/117.retrans_time_ms = 1000
net.ipv6.neigh.bond1/117.ucast_solicit = 3
net.ipv6.neigh.bond1/117.unres_qlen = 31
net.ipv6.neigh.bond1/117.unres_qlen_bytes = 65536
net.ipv6.neigh.bond1/118.anycast_delay = 100
net.ipv6.neigh.bond1/118.app_solicit = 0
net.ipv6.neigh.bond1/118.base_reachable_time_ms = 30000
net.ipv6.neigh.bond1/118.delay_first_probe_time = 5
net.ipv6.neigh.bond1/118.gc_stale_time = 60
net.ipv6.neigh.bond1/118.locktime = 0
net.ipv6.neigh.bond1/118.mcast_solicit = 3
net.ipv6.neigh.bond1/118.proxy_delay = 80
net.ipv6.neigh.bond1/118.proxy_qlen = 64
net.ipv6.neigh.bond1/118.retrans_time_ms = 1000
net.ipv6.neigh.bond1/118.ucast_solicit = 3
net.ipv6.neigh.bond1/118.unres_qlen = 31
net.ipv6.neigh.bond1/118.unres_qlen_bytes = 65536
net.ipv6.neigh.bond1/96.anycast_delay = 100
net.ipv6.neigh.bond1/96.app_solicit = 0
net.ipv6.neigh.bond1/96.base_reachable_time_ms = 30000
net.ipv6.neigh.bond1/96.delay_first_probe_time = 5
net.ipv6.neigh.bond1/96.gc_stale_time = 60
net.ipv6.neigh.bond1/96.locktime = 0
net.ipv6.neigh.bond1/96.mcast_solicit = 3
net.ipv6.neigh.bond1/96.proxy_delay = 80
net.ipv6.neigh.bond1/96.proxy_qlen = 64
net.ipv6.neigh.bond1/96.retrans_time_ms = 1000
net.ipv6.neigh.bond1/96.ucast_solicit = 3
net.ipv6.neigh.bond1/96.unres_qlen = 31
net.ipv6.neigh.bond1/96.unres_qlen_bytes = 65536
net.ipv6.neigh.bond1/97.anycast_delay = 100
net.ipv6.neigh.bond1/97.app_solicit = 0
net.ipv6.neigh.bond1/97.base_reachable_time_ms = 30000
net.ipv6.neigh.bond1/97.delay_first_probe_time = 5
net.ipv6.neigh.bond1/97.gc_stale_time = 60
net.ipv6.neigh.bond1/97.locktime = 0
net.ipv6.neigh.bond1/97.mcast_solicit = 3
net.ipv6.neigh.bond1/97.proxy_delay = 80
net.ipv6.neigh.bond1/97.proxy_qlen = 64
net.ipv6.neigh.bond1/97.retrans_time_ms = 1000
net.ipv6.neigh.bond1/97.ucast_solicit = 3
net.ipv6.neigh.bond1/97.unres_qlen = 31
net.ipv6.neigh.bond1/97.unres_qlen_bytes = 65536
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 31
net.ipv6.neigh.default.unres_qlen_bytes = 65536
net.ipv6.neigh.eno1.anycast_delay = 100
net.ipv6.neigh.eno1.app_solicit = 0
net.ipv6.neigh.eno1.base_reachable_time_ms = 30000
net.ipv6.neigh.eno1.delay_first_probe_time = 5
net.ipv6.neigh.eno1.gc_stale_time = 60
net.ipv6.neigh.eno1.locktime = 0
net.ipv6.neigh.eno1.mcast_solicit = 3
net.ipv6.neigh.eno1.proxy_delay = 80
net.ipv6.neigh.eno1.proxy_qlen = 64
net.ipv6.neigh.eno1.retrans_time_ms = 1000
net.ipv6.neigh.eno1.ucast_solicit = 3
net.ipv6.neigh.eno1.unres_qlen = 31
net.ipv6.neigh.eno1.unres_qlen_bytes = 65536
net.ipv6.neigh.eno2.anycast_delay = 100
net.ipv6.neigh.eno2.app_solicit = 0
net.ipv6.neigh.eno2.base_reachable_time_ms = 30000
net.ipv6.neigh.eno2.delay_first_probe_time = 5
net.ipv6.neigh.eno2.gc_stale_time = 60
net.ipv6.neigh.eno2.locktime = 0
net.ipv6.neigh.eno2.mcast_solicit = 3
net.ipv6.neigh.eno2.proxy_delay = 80
net.ipv6.neigh.eno2.proxy_qlen = 64
net.ipv6.neigh.eno2.retrans_time_ms = 1000
net.ipv6.neigh.eno2.ucast_solicit = 3
net.ipv6.neigh.eno2.unres_qlen = 31
net.ipv6.neigh.eno2.unres_qlen_bytes = 65536
net.ipv6.neigh.enp1s0f0.anycast_delay = 100
net.ipv6.neigh.enp1s0f0.app_solicit = 0
net.ipv6.neigh.enp1s0f0.base_reachable_time_ms = 30000
net.ipv6.neigh.enp1s0f0.delay_first_probe_time = 5
net.ipv6.neigh.enp1s0f0.gc_stale_time = 60
net.ipv6.neigh.enp1s0f0.locktime = 0
net.ipv6.neigh.enp1s0f0.mcast_solicit = 3
net.ipv6.neigh.enp1s0f0.proxy_delay = 80
net.ipv6.neigh.enp1s0f0.proxy_qlen = 64
net.ipv6.neigh.enp1s0f0.retrans_time_ms = 1000
net.ipv6.neigh.enp1s0f0.ucast_solicit = 3
net.ipv6.neigh.enp1s0f0.unres_qlen = 31
net.ipv6.neigh.enp1s0f0.unres_qlen_bytes = 65536
net.ipv6.neigh.enp1s0f1.anycast_delay = 100
net.ipv6.neigh.enp1s0f1.app_solicit = 0
net.ipv6.neigh.enp1s0f1.base_reachable_time_ms = 30000
net.ipv6.neigh.enp1s0f1.delay_first_probe_time = 5
net.ipv6.neigh.enp1s0f1.gc_stale_time = 60
net.ipv6.neigh.enp1s0f1.locktime = 0
net.ipv6.neigh.enp1s0f1.mcast_solicit = 3
net.ipv6.neigh.enp1s0f1.proxy_delay = 80
net.ipv6.neigh.enp1s0f1.proxy_qlen = 64
net.ipv6.neigh.enp1s0f1.retrans_time_ms = 1000
net.ipv6.neigh.enp1s0f1.ucast_solicit = 3
net.ipv6.neigh.enp1s0f1.unres_qlen = 31
net.ipv6.neigh.enp1s0f1.unres_qlen_bytes = 65536
net.ipv6.neigh.enp4s0f0.anycast_delay = 100
net.ipv6.neigh.enp4s0f0.app_solicit = 0
net.ipv6.neigh.enp4s0f0.base_reachable_time_ms = 30000
net.ipv6.neigh.enp4s0f0.delay_first_probe_time = 5
net.ipv6.neigh.enp4s0f0.gc_stale_time = 60
net.ipv6.neigh.enp4s0f0.locktime = 0
net.ipv6.neigh.enp4s0f0.mcast_solicit = 3
net.ipv6.neigh.enp4s0f0.proxy_delay = 80
net.ipv6.neigh.enp4s0f0.proxy_qlen = 64
net.ipv6.neigh.enp4s0f0.retrans_time_ms = 1000
net.ipv6.neigh.enp4s0f0.ucast_solicit = 3
net.ipv6.neigh.enp4s0f0.unres_qlen = 31
net.ipv6.neigh.enp4s0f0.unres_qlen_bytes = 65536
net.ipv6.neigh.enp4s0f1.anycast_delay = 100
net.ipv6.neigh.enp4s0f1.app_solicit = 0
net.ipv6.neigh.enp4s0f1.base_reachable_time_ms = 30000
net.ipv6.neigh.enp4s0f1.delay_first_probe_time = 5
net.ipv6.neigh.enp4s0f1.gc_stale_time = 60
net.ipv6.neigh.enp4s0f1.locktime = 0
net.ipv6.neigh.enp4s0f1.mcast_solicit = 3
net.ipv6.neigh.enp4s0f1.proxy_delay = 80
net.ipv6.neigh.enp4s0f1.proxy_qlen = 64
net.ipv6.neigh.enp4s0f1.retrans_time_ms = 1000
net.ipv6.neigh.enp4s0f1.ucast_solicit = 3
net.ipv6.neigh.enp4s0f1.unres_qlen = 31
net.ipv6.neigh.enp4s0f1.unres_qlen_bytes = 65536
net.ipv6.neigh.enp6s0f0.anycast_delay = 100
net.ipv6.neigh.enp6s0f0.app_solicit = 0
net.ipv6.neigh.enp6s0f0.base_reachable_time_ms = 30000
net.ipv6.neigh.enp6s0f0.delay_first_probe_time = 5
net.ipv6.neigh.enp6s0f0.gc_stale_time = 60
net.ipv6.neigh.enp6s0f0.locktime = 0
net.ipv6.neigh.enp6s0f0.mcast_solicit = 3
net.ipv6.neigh.enp6s0f0.proxy_delay = 80
net.ipv6.neigh.enp6s0f0.proxy_qlen = 64
net.ipv6.neigh.enp6s0f0.retrans_time_ms = 1000
net.ipv6.neigh.enp6s0f0.ucast_solicit = 3
net.ipv6.neigh.enp6s0f0.unres_qlen = 31
net.ipv6.neigh.enp6s0f0.unres_qlen_bytes = 65536
net.ipv6.neigh.enp6s0f1.anycast_delay = 100
net.ipv6.neigh.enp6s0f1.app_solicit = 0
net.ipv6.neigh.enp6s0f1.base_reachable_time_ms = 30000
net.ipv6.neigh.enp6s0f1.delay_first_probe_time = 5
net.ipv6.neigh.enp6s0f1.gc_stale_time = 60
net.ipv6.neigh.enp6s0f1.locktime = 0
net.ipv6.neigh.enp6s0f1.mcast_solicit = 3
net.ipv6.neigh.enp6s0f1.proxy_delay = 80
net.ipv6.neigh.enp6s0f1.proxy_qlen = 64
net.ipv6.neigh.enp6s0f1.retrans_time_ms = 1000
net.ipv6.neigh.enp6s0f1.ucast_solicit = 3
net.ipv6.neigh.enp6s0f1.unres_qlen = 31
net.ipv6.neigh.enp6s0f1.unres_qlen_bytes = 65536
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 31
net.ipv6.neigh.lo.unres_qlen_bytes = 65536
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 2097152
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.xfrm6_gc_thresh = 1024
Comment 7 mailinglists 2016-05-15 11:55:37 EDT
Also submitted a bug report in CentOS:
https://bugs.centos.org/view.php?id=10858

Please do backport the fix from upstream.
https://github.com/acassen/keepalived/commit/4d1bc9a1aee37657619688bbeeef96fff95d1635
Comment 8 mailinglists 2016-05-16 08:35:53 EDT
Below I am addressing your four points specifically, as advised on IRC to get the issue moving forward.

1. The alternate sollution by disabling garp_master_repeat and garp_master_refresh and only use garp_master delay does not work, because the same problem exists with unsolicited neighbour advertisement sent out at the time of taking over resources. Those settings by themselves do not cause the defaut gw to be deleted, but the unsolicited neighbour advertisement package which does not have router flag set, does make it (clien'ts default gw) dissapear.

2. Upstream patch has since been grately improved, to check if the unsolicited neighbour advertisement packet should have set router flag or not, based on the actual settings on the local system. Please take a look at: https://github.com/acassen/keepalived/commit/4d1bc9a1aee37657619688bbeeef96fff95d1635

3. Your thought has been addressed exactly in the before mentioned patch (point nbr. 2.)

4. On the other side, I am unable to produce a working system when IPv6 addresses migrating are on the actual routers. It fails every time, on every setup I make. Additionally I have provided detailed info about my the systems. If more is required, please do let me know.

Hopefully the above comments will be enough to at least get a response if not a fix backported.

Once again, we could fix this by building packages ourselves, but it's not in the spirit of open source and people contributing back to the system.
Comment 11 mailinglists 2017-03-16 13:33:44 EDT
So is this supposed to be fixed in keepalived-1.3.4-1.el7 and keepliaved will no longer remove default routes of clients?
Comment 12 Brandon Perkins 2017-03-16 14:19:20 EDT
Yes, the patch is included in 1.3.2 and can be found here:

https://github.com/acassen/keepalived/commit/4d1bc9a1aee37657619688bbeeef96fff95d1635
Comment 19 errata-xmlrpc 2017-08-01 15:36:38 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2169

Note You need to log in before you can comment on or make changes to this bug.