Bug 1426735
| Summary: | Backup HA router sending traffic, traffic from switch interrupted | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Aaron Smith <aasmith> | ||||
| Component: | openstack-neutron | Assignee: | Daniel Alvarez Sanchez <dalvarez> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Alexander Stafeyev <astafeye> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 10.0 (Newton) | CC: | aasmith, amuller, atelang, bfournie, chrisw, dalvarez, nyechiel, oblaut, samccann, srevivo | ||||
| Target Milestone: | z4 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 10.0 (Newton) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-neutron-9.3.1-9.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
Cause:
Neutron backup HA routers have the same IP/MAC addresses as the master instance and the backup HA routers have IPv6 forwarding enabled by default. This causes the backup HA routers to subscribe to different multicast groups and these backup HA routers may respond to queries coming from the external network.
Consequence:
This backup HA router traffic causes the upstream switch to learn MAC address on a different port, disrupting existing traffic to the master instance.
Fix:
Disable IPv6 forwarding on backup instances and restore it on failover.
Result:
Traffic will not leave the backup instance to go to the upstream switch thus not disrupt existing connections with the master instance.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1469048 1469051 (view as bug list) | Environment: | |||||
| Last Closed: | 2017-09-06 17:17:18 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1469048, 1469051 | ||||||
| Attachments: |
|
||||||
|
Description
Aaron Smith
2017-02-24 17:39:58 UTC
Created attachment 1257396 [details]
SOS report from on of the backup hosts
Note that /var/log/neutron seems empty in the SOS report. In which interface are you seeing those incoming RA packets? May you please paste the "ip a" command output on the qrouter namespace of the backup node? Mine doesn't seem to have any LL address on external interfaces:
[vagrant@worker ~]$ sudo ip netns exec qrouter-491fdd6f-3f7b-473e-a29f-4873eb2f34c4 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
534: ha-276f79ea-82: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:e0:b7:aa brd ff:ff:ff:ff:ff:ff
inet 169.254.192.4/18 brd 169.254.255.255 scope global ha-276f79ea-82
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fee0:b7aa/64 scope link
valid_lft forever preferred_lft forever
535: qg-d9149575-50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:d1:be:5c brd ff:ff:ff:ff:ff:ff
536: qr-7edb8430-ce: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:ef:74:d3 brd ff:ff:ff:ff:ff:ff
541: qr-fa95087c-34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:2d:22:b4 brd ff:ff:ff:ff:ff:ff
Forget my previous comment, I saw it in the sosreport:
$ cat ip_netns_exec_qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45_ip_address_show
[...]
30: qg-3f327e61-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:a7:ae:63 brd ff:ff:ff:ff:ff:ff
I wonder how a packet can reach this interface on the backup node from the outside though.
(In reply to Assaf Muller from comment #2) > Note that /var/log/neutron seems empty in the SOS report. Hi, I don't know why the neutron log is empty, but I am fairly new to SOS reports. We are transitioning our deployment to a different configuration so I will need a day or so to repeat the tests. (In reply to Daniel Alvarez Sanchez from comment #4) > Forget my previous comment, I saw it in the sosreport: > > $ cat > ip_netns_exec_qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45_ip_address_show > [...] > 30: qg-3f327e61-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue > state UNKNOWN qlen 1000 > link/ether fa:16:3e:a7:ae:63 brd ff:ff:ff:ff:ff:ff > > I wonder how a packet can reach this interface on the backup node from the > outside though. Hi here is a packet capture of an occurrence.. This packet capture is in the netns of the qrouter.... [root@overcloud-controller-1 ~]# ip netns ls qdhcp-8897a5a6-dd5e-44d3-852f-c0dcd870b228 qdhcp-9b61e335-1b6c-498f-82b4-f660180e6876 qdhcp-ec0bad75-3466-43fb-b240-2b7d53a1d2c7 qdhcp-f76fcc81-0b57-4258-9e1a-04aa0c786428 qdhcp-b918b2ce-ef92-444d-80a0-9bf1f93406f3 qdhcp-3e184953-6df2-45c0-a0d4-7ca837110740 qdhcp-06c553c8-0adb-4b95-b2c4-4dc163f95a4f qdhcp-184592ea-da08-4180-a9cb-a660125e5d1a qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 [root@overcloud-controller-1 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 31: ha-bce4b526-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000 link/ether fa:16:3e:2d:1c:2c brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-bce4b526-c4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe2d:1c2c/64 scope link valid_lft forever preferred_lft forever 32: qr-1656d4d1-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000 link/ether fa:16:3e:54:77:0b brd ff:ff:ff:ff:ff:ff 33: qg-3f327e61-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state UNKNOWN qlen 1000 link/ether fa:16:3e:a7:ae:63 brd ff:ff:ff:ff:ff:ff [root@overcloud-controller-1 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 bash [root@overcloud-controller-1 ~]# tcpdump -nnvv -e -i qg-3f327e61-8b tcpdump: WARNING: qg-3f327e61-8b: no IPv4 address assigned tcpdump: listening on qg-3f327e61-8b, link-type EN10MB (Ethernet), capture size 65535 bytes 14:24:09.814889 08:81:f4:a6:dc:01 > 01:00:5e:00:00:01, ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 1, id 30720, offset 0, flags [none], proto IGMP (2), length 32, options (RA)) 10.19.108.254 > 224.0.0.1: igmp query v2 14:24:13.799395 08:81:f4:a6:dc:01 > 33:33:00:00:00:0d, ethertype IPv6 (0x86dd), length 110: (class 0xc0, hlim 1, next-header PIM (103) payload length: 56) fe80:52:0:136c::fe > ff02::d: PIMv2, length 56 Hello, cksum 0x018f (correct) Hold Time Option (1), length 2, Value: 1m45s 0x0000: 0069 LAN Prune Delay Option (2), length 4, Value: T-bit=1, LAN delay 500ms, Override interval 2000ms 0x0000: 81f4 07d0 DR Priority Option (19), length 4, Value: 1 0x0000: 0000 0001 Address List Option (24), length 18, Value: 2620:52:0:136c::fe 0x0000: 0200 2620 0052 0000 136c 0000 0000 0000 0x0010: 00fe Generation ID Option (20), length 4, Value: 0x00ec032c 0x0000: 00ec 032c 14:24:21.481044 08:81:f4:a6:dc:01 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80:52:0:136c::fe > ff02::1: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener querymax resp delay: 10000 addr: :: 14:24:28.241400 fa:16:3e:ba:6f:6f > 33:33:ff:ba:6f:6f, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::f816:3eff:feba:6f6f > ff02::1:ffba:6f6f: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ffba:6f6f 14:24:38.883558 08:81:f4:a6:dc:01 > 01:00:5e:00:00:0d, ethertype IPv4 (0x0800), length 68: (tos 0xc0, ttl 1, id 52817, offset 0, flags [none], proto PIM (103), length 54) 10.19.108.254 > 224.0.0.13: PIMv2, length 34 Hello, cksum 0x4fa8 (correct) Hold Time Option (1), length 2, Value: 1m45s 0x0000: 0069 LAN Prune Delay Option (2), length 4, Value: T-bit=1, LAN delay 500ms, Override interval 2000ms 0x0000: 81f4 07d0 DR Priority Option (19), length 4, Value: 1 0x0000: 0000 0001 Generation ID Option (20), length 4, Value: 0x7c75897b 0x0000: 7c75 897b (In reply to Aaron Smith from comment #6) > 33: qg-3f327e61-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue > state UNKNOWN qlen 1000 > link/ether fa:16:3e:a7:ae:63 brd ff:ff:ff:ff:ff:ff I can't see any packets from fa:16:3e:2d:1c:2c. In your first capture I could see packets coming out the qg interface (source MAC was fa:16:3e:a7:ae:63) but in your last capture looks like there're no packets coming out from the qrouter to the external switch. I've been able to send RA packets and receive them on the qg interface of the qrouter but still I can't see the backup node responding: [vagrant@worker ipv6toolkit]$ sudo ip netns exec qrouter-491fdd6f-3f7b-473e-a29f-4873eb2f34c4 tcpdump -e -vvnn -i qg-d9149575-50 & [1] 2627 [vagrant@worker ipv6toolkit]$ tcpdump: WARNING: qg-d9149575-50: no IPv4 address assigned tcpdump: listening on qg-d9149575-50, link-type EN10MB (Ethernet), capture size 65535 bytes [vagrant@worker ipv6toolkit]$ sudo ./ra6 -i br-ex -D fa:16:3e:d1:be:5c -c 5 --lifetime 100 -o -e -vv Ethernet Source Address: 62:2f:80:4e:c6:45 (automatically-selected) Ethernet Destination Address: fa:16:3e:d1:be:5c IPv6 Source Address: fe80::602f:80ff:fe4e:c645 (automatically-selected) IPv6 Destination Address: ff02::1 (all-nodes link-local multicast) IPv6 Hop Limit: 255 (default) Cur Hop Limit: 5 Preference: high Flags: O Router Lifetime: 100 Reachable Time: 4294967295 Retrans Timer: 4000 Source Link-layer Address option -> Address: 62:2f:80:4e:c6:45 Initial attack packet(s) sent successfully. 22:54:04.579235 62:2f:80:4e:c6:45 > fa:16:3e:d1:be:5c, ethertype IPv6 (0x86dd), length 78: (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::602f:80ff:fe4e:c645 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 24 hop limit 5, Flags [other stateful], pref high, router lifetime 100s, reachable time 4294967295ms, retrans time 4000ms source link-address option (1), length 8 (1): 62:2f:80:4e:c6:45 0x0000: 622f 804e c645 My apologies, I copied the wrong controller window....
In the capture below, you will see several packets come in from
08:81:f4:a6:dc:01. Eventually, fa:16:3e:a7:ae:63 replies and ICMP
requests are redirected to this backup controller.
root@overcloud-controller-0 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
28: ha-94c36543-95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:52:98:a4 brd ff:ff:ff:ff:ff:ff
inet 169.254.192.8/18 brd 169.254.255.255 scope global ha-94c36543-95
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe52:98a4/64 scope link
valid_lft forever preferred_lft forever
29: qr-1656d4d1-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:54:77:0b brd ff:ff:ff:ff:ff:ff
30: qg-3f327e61-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:a7:ae:63 brd ff:ff:ff:ff:ff:ff
[root@overcloud-controller-0 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 bash
[root@overcloud-controller-0 ~]# tcpdump -nnvv -e -i qg-3f327e61-8b
tcpdump: WARNING: qg-3f327e61-8b: no IPv4 address assigned
tcpdump: listening on qg-3f327e61-8b, link-type EN10MB (Ethernet), capture size 65535 bytes
15:11:03.232415 08:81:f4:a6:dc:01 > 01:00:5e:00:00:0d, ethertype IPv4 (0x0800), length 68: (tos 0xc0, ttl 1, id 30322, offset 0, flags [none], proto PIM (103), length 54)
10.19.108.254 > 224.0.0.13: PIMv2, length 34
Hello, cksum 0x4fa8 (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Generation ID Option (20), length 4, Value: 0x7c75897b
0x0000: 7c75 897b
15:11:17.743272 08:81:f4:a6:dc:01 > 33:33:00:00:00:0d, ethertype IPv6 (0x86dd), length 110: (class 0xc0, hlim 1, next-header PIM (103) payload length: 56) fe80:52:0:136c::fe > ff02::d: PIMv2, length 56
Hello, cksum 0x018f (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Address List Option (24), length 18, Value:
2620:52:0:136c::fe
0x0000: 0200 2620 0052 0000 136c 0000 0000 0000
0x0010: 00fe
Generation ID Option (20), length 4, Value: 0x00ec032c
0x0000: 00ec 032c
15:11:31.219937 08:81:f4:a6:dc:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.19.108.155 tell 10.19.108.254, length 46
15:11:31.646955 08:81:f4:a6:dc:01 > 01:00:5e:00:00:0d, ethertype IPv4 (0x0800), length 68: (tos 0xc0, ttl 1, id 4988, offset 0, flags [none], proto PIM (103), length 54)
10.19.108.254 > 224.0.0.13: PIMv2, length 34
Hello, cksum 0x4fa8 (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Generation ID Option (20), length 4, Value: 0x7c75897b
0x0000: 7c75 897b
15:11:43.561269 08:81:f4:a6:dc:01 > 01:00:5e:00:00:0d, ethertype IPv4 (0x0800), length 90: (tos 0xc0, ttl 1, id 16320, offset 0, flags [none], proto PIM (103), length 76)
10.19.108.254 > 224.0.0.13: PIMv2, length 56
Bootstrap, cksum 0xac4c (correct) tag=934a hashmlen=30 BSRprio=100 BSR=10.16.253.254 (group0: 239.255.0.0/16 RPcnt=3 FRPcnt=3 RP0=10.16.253.242,holdtime=2m30s,prio=100 RP1=10.16.253.253,holdtime=2m30s,prio=0 RP2=10.16.253.254,holdtime=2m30s,prio=0)
15:11:46.972690 08:81:f4:a6:dc:01 > 33:33:00:00:00:0d, ethertype IPv6 (0x86dd), length 110: (class 0xc0, hlim 1, next-header PIM (103) payload length: 56) fe80:52:0:136c::fe > ff02::d: PIMv2, length 56
Hello, cksum 0x018f (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Address List Option (24), length 18, Value:
2620:52:0:136c::fe
0x0000: 0200 2620 0052 0000 136c 0000 0000 0000
0x0010: 00fe
Generation ID Option (20), length 4, Value: 0x00ec032c
0x0000: 00ec 032c
15:12:01.271824 08:81:f4:a6:dc:01 > 01:00:5e:00:00:0d, ethertype IPv4 (0x0800), length 68: (tos 0xc0, ttl 1, id 31628, offset 0, flags [none], proto PIM (103), length 54)
10.19.108.254 > 224.0.0.13: PIMv2, length 34
Hello, cksum 0x4fa8 (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Generation ID Option (20), length 4, Value: 0x7c75897b
0x0000: 7c75 897b
15:12:01.797760 08:81:f4:a6:dc:01 > 01:00:5e:00:00:01, ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 1, id 32087, offset 0, flags [none], proto IGMP (2), length 32, options (RA))
10.19.108.254 > 224.0.0.1: igmp query v2
15:12:13.462663 08:81:f4:a6:dc:01 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80:52:0:136c::fe > ff02::1: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener querymax resp delay: 10000 addr: ::
15:12:14.099551 08:81:f4:a6:dc:01 > 33:33:00:00:00:0d, ethertype IPv6 (0x86dd), length 110: (class 0xc0, hlim 1, next-header PIM (103) payload length: 56) fe80:52:0:136c::fe > ff02::d: PIMv2, length 56
Hello, cksum 0x018f (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Address List Option (24), length 18, Value:
2620:52:0:136c::fe
0x0000: 0200 2620 0052 0000 136c 0000 0000 0000
0x0010: 00fe
Generation ID Option (20), length 4, Value: 0x00ec032c
0x0000: 00ec 032c
15:12:16.982696 4e:14:50:cf:3a:ca > 33:33:ff:cf:3a:ca, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::4c14:50ff:fecf:3aca > ff02::1:ffcf:3aca: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ffcf:3aca
15:12:17.510559 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2
15:12:17.615963 08:81:f4:a6:e5:01 > fa:16:3e:a7:ae:63, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 59, id 38774, offset 0, flags [DF], proto ICMP (1), length 84)
10.18.81.21 > 10.19.108.161: ICMP echo request, id 4835, seq 18157, length 64
15:12:17.816109 08:81:f4:a6:e5:01 > fa:16:3e:a7:ae:63, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 59, id 38860, offset 0, flags [DF], proto ICMP (1), length 84)
10.18.81.21 > 10.19.108.161: ICMP echo request, id 4835, seq 18158, length 64
15:12:18.016286 08:81:f4:a6:e5:01 > fa:16:3e:a7:ae:63, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 59, id 38960, offset 0, flags [DF], proto ICMP (1), length 84)
10.18.81.21 > 10.19.108.161: ICMP echo request, id 4835, seq 18159, length 64
Can you please check whether radvd is running only on master node? I can't see the ps output in the sosreport. (In reply to Daniel Alvarez Sanchez from comment #9) > Can you please check whether radvd is running only on master node? I can't > see the ps output in the sosreport. I do not see radvd on the master or any of the controllers. That is expected as the bug report states that the HA routers are not connected to v6 subnets. Here is a hex capture of the packets just before and at the point of the problem... The last packet is the ICMP going to this controller.
17:31:39.711925 IP (tos 0xc0, ttl 1, id 27822, offset 0, flags [none], proto PIM (103), length 54)
10.19.108.254 > 224.0.0.13: PIMv2, length 34
Hello, cksum 0x4fa8 (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Generation ID Option (20), length 4, Value: 0x7c75897b
0x0000: 7c75 897b
0x0000: 0100 5e00 000d 0881 f4a6 dc01 0800 45c0 ..^...........E.
0x0010: 0036 6cae 0000 0167 f4d4 0a13 6cfe e000 .6l....g....l...
0x0020: 000d 2000 4fa8 0001 0002 0069 0002 0004 ....O......i....
0x0030: 81f4 07d0 0013 0004 0000 0001 0014 0004 ................
0x0040: 7c75 897b |u.{
17:31:53.407993 IP6 (class 0xc0, hlim 1, next-header PIM (103) payload length: 56) fe80:52:0:136c::fe > ff02::d: PIMv2, length 56
Hello, cksum 0x018f (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Address List Option (24), length 18, Value:
2620:52:0:136c::fe
0x0000: 0200 2620 0052 0000 136c 0000 0000 0000
0x0010: 00fe
Generation ID Option (20), length 4, Value: 0x00ec032c
0x0000: 00ec 032c
0x0000: 3333 0000 000d 0881 f4a6 dc01 86dd 6c00 33............l.
0x0010: 0000 0038 6701 fe80 0052 0000 136c 0000 ...8g....R...l..
0x0020: 0000 0000 00fe ff02 0000 0000 0000 0000 ................
0x0030: 0000 0000 000d 2000 018f 0001 0002 0069 ...............i
0x0040: 0002 0004 81f4 07d0 0013 0004 0000 0001 ................
0x0050: 0018 0012 0200 2620 0052 0000 136c 0000 ......&..R...l..
0x0060: 0000 0000 00fe 0014 0004 00ec 032c .............,
17:32:02.182207 IP (tos 0xc0, ttl 1, id 48897, offset 0, flags [none], proto IGMP (2), length 32, options (RA))
10.19.108.254 > 224.0.0.1: igmp query v2
0x0000: 0100 5e00 0001 0881 f4a6 dc01 0800 46c0 ..^...........F.
0x0010: 0020 bf01 0000 0102 0e04 0a13 6cfe e000 ............l...
0x0020: 0001 9404 0000 1164 ee9b 0000 0000 0000 .......d........
0x0030: 0000 0000 0000 0000 0000 0000 ............
17:32:06.872488 IP (tos 0xc0, ttl 1, id 52629, offset 0, flags [none], proto PIM (103), length 54)
10.19.108.254 > 224.0.0.13: PIMv2, length 34
Hello, cksum 0x4fa8 (correct)
Hold Time Option (1), length 2, Value: 1m45s
0x0000: 0069
LAN Prune Delay Option (2), length 4, Value:
T-bit=1, LAN delay 500ms, Override interval 2000ms
0x0000: 81f4 07d0
DR Priority Option (19), length 4, Value: 1
0x0000: 0000 0001
Generation ID Option (20), length 4, Value: 0x7c75897b
0x0000: 7c75 897b
0x0000: 0100 5e00 000d 0881 f4a6 dc01 0800 45c0 ..^...........E.
0x0010: 0036 cd95 0000 0167 93ed 0a13 6cfe e000 .6.....g....l...
0x0020: 000d 2000 4fa8 0001 0002 0069 0002 0004 ....O......i....
0x0030: 81f4 07d0 0013 0004 0000 0001 0014 0004 ................
0x0040: 7c75 897b |u.{
17:32:13.857399 IP6 (hlim 1, next-header Options (0) payload length: 32) fe80:52:0:136c::fe > ff02::1: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener querymax resp delay: 10000 addr: ::
0x0000: 3333 0000 0001 0881 f4a6 dc01 86dd 6000 33............`.
0x0010: 0000 0020 0001 fe80 0052 0000 136c 0000 .........R...l..
0x0020: 0000 0000 00fe ff02 0000 0000 0000 0000 ................
0x0030: 0000 0000 0001 3a00 0502 0000 0100 8200 ......:.........
0x0040: 445c 2710 0000 0000 0000 0000 0000 0000 D\'.............
0x0050: 0000 0000 0000 ......
17:32:15.275383 IP (tos 0xc0, ttl 1, id 59107, offset 0, flags [none], proto PIM (103), length 76)
10.19.108.254 > 224.0.0.13: PIMv2, length 56
Bootstrap, cksum 0x8b76 (correct) tag=b420 hashmlen=30 BSRprio=100 BSR=10.16.253.254 (group0: 239.255.0.0/16 RPcnt=3 FRPcnt=3 RP0=10.16.253.242,holdtime=2m30s,prio=100 RP1=10.16.253.253,holdtime=2m30s,prio=0 RP2=10.16.253.254,holdtime=2m30s,prio=0)
0x0000: 0100 5e00 000d 0881 f4a6 dc01 0800 45c0 ..^...........E.
0x0010: 004c e6e3 0000 0167 7a89 0a13 6cfe e000 .L.....gz...l...
0x0020: 000d 2400 8b76 b420 1e64 0100 0a10 fdfe ..$..v...d......
0x0030: 0100 0010 efff 0000 0303 0000 0100 0a10 ................
0x0040: fdf2 0096 6400 0100 0a10 fdfd 0096 0000 ....d...........
0x0050: 0100 0a10 fdfe 0096 0000 ..........
17:32:17.153695 IP6 (hlim 1, next-header Options (0) payload length: 32) fe80::609d:29ff:fe00:3183 > ff02::1:ff00:3183: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff00:3183
0x0000: 3333 ff00 3183 629d 2900 3183 86dd 6000 33..1.b.).1...`.
0x0010: 0000 0020 0001 fe80 0000 0000 0000 609d ..............`.
0x0020: 29ff fe00 3183 ff02 0000 0000 0000 0000 )...1...........
0x0030: 0001 ff00 3183 3a00 0502 0000 0100 8300 ....1.:.........
0x0040: 64fb 0000 0000 ff02 0000 0000 0000 0000 d...............
0x0050: 0001 ff00 3183 ....1.
17:32:20.353683 IP6 (hlim 1, next-header Options (0) payload length: 32) fe80::f816:3eff:fe1a:4c00 > ff02::1:ff1a:4c00: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff1a:4c00
0x0000: 3333 ff1a 4c00 fa16 3e1a 4c00 86dd 6000 33..L...>.L...`.
0x0010: 0000 0020 0001 fe80 0000 0000 0000 f816 ................
0x0020: 3eff fe1a 4c00 ff02 0000 0000 0000 0000 >...L...........
0x0030: 0001 ff1a 4c00 3a00 0502 0000 0100 8300 ....L.:.........
0x0040: 68bc 0000 0000 ff02 0000 0000 0000 0000 h...............
0x0050: 0001 ff1a 4c00 ....L.
17:32:20.753453 IP6 (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2
0x0000: 3333 0000 0002 fa16 3ea7 ae63 86dd 6000 33......>..c..`.
0x0010: 0000 0020 0001 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 ff02 0000 0000 0000 0000 ................
0x0030: 0000 0000 0002 3a00 0502 0000 0100 8300 ......:.........
0x0040: 7ea3 0000 0000 ff02 0000 0000 0000 0000 ~...............
0x0050: 0000 0000 0002 ......
17:32:20.806009 IP (tos 0x0, ttl 59, id 19083, offset 0, flags [DF], proto ICMP (1), length 84)
10.18.81.21 > 10.19.108.161: ICMP echo request, id 9877, seq 498, length 64
0x0000: fa16 3ea7 ae63 0881 f4a6 e501 0800 4500 ..>..c........E.
0x0010: 0054 4a8b 4000 3b01 2343 0a12 5115 0a13 .TJ.@.;.#C..Q...
0x0020: 6ca1 0800 a034 2695 01f2 26b4 b558 0000 l....4&...&..X..
0x0030: 0000 8d64 0700 0000 0000 1011 1213 1415 ...d............
0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 ...........!"#$%
0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345
0x0060: 3637 67
(In reply to Aaron Smith from comment #0) > > Additional info: > Setting accept_ra=0 on the backup host routers stops the problem from > happening. Unfortunately, on a reboot, we loose the setting. The current > sysctl files have accept_ra=0. So if accept_ra is disabled when rebooting and it then gets enabled, the only reason I can see for that is that the role was set to "master" at some point. I'd need to check logs to confirm that the qrouter was set to "master" and, therefore, ra was enabled on the gateway interface. Whenever possible, please upload a new sosreport with neutron logs so that we can check. From the code, I can see that it's only enabled when the following conditions are met: * IPv6 is enabled on the platform * IPv6 gateway config flag is not set * External network associated to the router doesn't include any IPv6 subnet * Router is set to "master" Maybe we should consider disabling accept_ra if the above are not met. (In reply to Daniel Alvarez Sanchez from comment #13) > (In reply to Aaron Smith from comment #0) > > > > Additional info: > > Setting accept_ra=0 on the backup host routers stops the problem from > > happening. Unfortunately, on a reboot, we loose the setting. The current > > sysctl files have accept_ra=0. > > So if accept_ra is disabled when rebooting and it then gets enabled, the > only reason I can see for that is that the role was set to "master" at some > point. I'd need to check logs to confirm that the qrouter was set to > "master" and, therefore, ra was enabled on the gateway interface. Whenever > possible, please upload a new sosreport with neutron logs so that we can > check. > > From the code, I can see that it's only enabled when the following > conditions are met: > > * IPv6 is enabled on the platform > * IPv6 gateway config flag is not set > * External network associated to the router doesn't include any IPv6 subnet > * Router is set to "master" > > Maybe we should consider disabling accept_ra if the above are not met. Here is the accept_ra setting on the back controllers... [root@overcloud-controller-1 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 bash [root@overcloud-controller-1 ~]# cat /proc/sys/net/ipv6/ anycast_src_echo_reply conf/ ip6frag_high_thresh ip6frag_time neigh/ xfrm6_gc_thresh bindv6only icmp/ ip6frag_low_thresh ip_nonlocal_bind route/ [root@overcloud-controller-1 ~]# cat /proc/sys/net/ipv6/conf/ all/ default/ ha-bce4b526-c4/ lo/ qg-3f327e61-8b/ qr-1656d4d1-2c/ [root@overcloud-controller-1 ~]# cat /proc/sys/net/ipv6/conf/qg-3f327e61-8b/accept_ra 2 It looks like ra_accept will be set to 2 in neutron if a IPv6 gateway
is not set but perhaps I'm misreading the code.
neutron/agent/linux/interface.py
def configure_ipv6_ra(namespace, dev_name):
"""Configure acceptance of IPv6 route advertisements on an intf."""
# Learn the default router's IP address via RAs
ip_lib.IPWrapper(namespace=namespace).netns.execute(
['sysctl', '-w', 'net.ipv6.conf.%s.accept_ra=2' % dev_name])
neutron/agent/l3/router_info.py
if self.use_ipv6 and not self.is_v6_gateway_set(gateway_ips):
# There is no IPv6 gw_ip, use RouterAdvt for default route.
enable_ra_on_gw = True
Neutron versions... [root@overcloud-controller-2 ~]# rpm -qa | grep neut openstack-neutron-lbaas-9.1.0-1.el7ost.noarch openstack-neutron-ml2-9.1.0-8.el7ost.noarch python-neutronclient-6.0.0-2.el7ost.noarch python-neutron-lib-0.4.0-1.el7ost.noarch openstack-neutron-bigswitch-lldp-9.40.0-1.1.el7ost.noarch openstack-neutron-metering-agent-9.1.0-8.el7ost.noarch puppet-neutron-9.4.2-1.el7ost.noarch python-neutron-9.1.0-8.el7ost.noarch python-neutron-tests-9.1.0-8.el7ost.noarch python-neutron-lbaas-9.1.0-1.el7ost.noarch openstack-neutron-openvswitch-9.1.0-8.el7ost.noarch openstack-neutron-bigswitch-agent-9.40.0-1.1.el7ost.noarch openstack-neutron-sriov-nic-agent-9.1.0-8.el7ost.noarch openstack-neutron-common-9.1.0-8.el7ost.noarch openstack-neutron-9.1.0-8.el7ost.noarch (In reply to Aaron Smith from comment #15) > It looks like ra_accept will be set to 2 in neutron if a IPv6 gateway > is not set but perhaps I'm misreading the code. If I'm not wrong, RA should only be enabled on master [0]. However, when the interface is created, it's assigned the default value which, in my case is 1: [vagrant@worker neutron]$ cat /proc/sys/net/ipv6/conf/default/accept_ra 1 I think there's a bug in [0] which doesn't disable RA in the else clause. In my opinion that should be done because if it's enabled by default (which is usually the case apparently) or the router was master previously, then it won't be disabled. I'll report the bug and propose the fix but can you please confirm that after setting accept_ra to 0, it's only set back to 2 (or 1) when the machine is rebooted? It should remain to 0 even though you restart all neutron services. Also, is there a chance that I can access your test setup (if any) to confirm the above? Thanks, Daniel [0] https://github.com/openstack/neutron/blob/347778a9f9ce955562c3535b413642baf9b12111/neutron/agent/l3/ha.py#L137 (In reply to Daniel Alvarez Sanchez from comment #17) > (In reply to Aaron Smith from comment #15) > > It looks like ra_accept will be set to 2 in neutron if a IPv6 gateway > > is not set but perhaps I'm misreading the code. > > If I'm not wrong, RA should only be enabled on master [0]. However, when the > interface is created, it's assigned the default value which, in my case is 1: > > [vagrant@worker neutron]$ cat /proc/sys/net/ipv6/conf/default/accept_ra > 1 > > I think there's a bug in [0] which doesn't disable RA in the else clause. In > my opinion that should be done because if it's enabled by default (which is > usually the case apparently) or the router was master previously, then it > won't be disabled. > > I'll report the bug and propose the fix but can you please confirm that > after setting accept_ra to 0, it's only set back to 2 (or 1) when the > machine is rebooted? It should remain to 0 even though you restart all > neutron services. > > Also, is there a chance that I can access your test setup (if any) to > confirm the above? > > Thanks, > Daniel > > [0] > https://github.com/openstack/neutron/blob/ > 347778a9f9ce955562c3535b413642baf9b12111/neutron/agent/l3/ha.py#L137 I will run the tests today and post the results. Thanks. Thanks Aaron for giving me access to a lab setup with the issue: I've written a patch to disable 'accept_ra' on the gateway interface on router instances. After restarting neutron-l3-agent service I can see that the accept_ra value goes from 1 to 0. However, tcpdump still shows packets being sent from that MAC address in the gw interface every ~2 minutes: [root@overcloud-controller-0 neutron]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 cat /proc/sys/net/ipv6/conf/qg-3f327e61-8b/accept_ra 0 [root@overcloud-controller-0 neutron]# tcpdump -nnvv -i qg-3f327e61-8b -e ether host fa:16:3e:a7:ae:63 tcpdump: WARNING: qg-3f327e61-8b: no IPv4 address assigned tcpdump: listening on qg-3f327e61-8b, link-type EN10MB (Ethernet), capture size 65535 bytes 16:18:37.073690 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2 16:20:44.043696 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2 16:22:46.202693 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2 16:24:54.811697 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2 Are you sure that seting accept_ra to 0 solved the issue? BTW, the code running on controller 0 has my fix so you can either reboot the instance or restart neutron services and it'll set accept_ra to 0 in the gateway interface of the qrouter namespace. (In reply to Daniel Alvarez Sanchez from comment #19) > Thanks Aaron for giving me access to a lab setup with the issue: > > I've written a patch to disable 'accept_ra' on the gateway interface on > router instances. After restarting neutron-l3-agent service I can see that > the accept_ra value goes from 1 to 0. However, tcpdump still shows packets > being sent from that MAC address in the gw interface every ~2 minutes: > > [root@overcloud-controller-0 neutron]# ip netns exec > qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 cat > /proc/sys/net/ipv6/conf/qg-3f327e61-8b/accept_ra > 0 > > > [root@overcloud-controller-0 neutron]# tcpdump -nnvv -i qg-3f327e61-8b -e > ether host fa:16:3e:a7:ae:63 > tcpdump: WARNING: qg-3f327e61-8b: no IPv4 address assigned > tcpdump: listening on qg-3f327e61-8b, link-type EN10MB (Ethernet), capture > size 65535 bytes > > > 16:18:37.073690 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 > (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast > listener reportmax resp delay: 0 addr: ff02::2 > > 16:20:44.043696 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 > (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast > listener reportmax resp delay: 0 addr: ff02::2 > > 16:22:46.202693 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 > (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast > listener reportmax resp delay: 0 addr: ff02::2 > > 16:24:54.811697 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 > (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast > listener reportmax resp delay: 0 addr: ff02::2 > > Are you sure that seting accept_ra to 0 solved the issue? > > BTW, the code running on controller 0 has my fix so you can either reboot > the instance or restart neutron services and it'll set accept_ra to 0 in the > gateway interface of the qrouter namespace. We had been seeing the failure every 10 minutes or so. We set accept_ra=0 on both backup controllers and did not see the issue for about an hour when we resumed our reboot testing. It seemed at the time that setting accept_ra=0 stopped the failure from happening. It looks like the packets sent from the backup nodes are answers to "multicast listener" queries rather than due to RA traffic (although it has to be disabled and the bug is still valid IMO): ==QUERY== 16:43:36.017944 08:81:f4:a6:dc:01 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80:52:0:136c::fe > ff02::1: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener querymax resp delay: 10000 addr: :: ==RESPONSE== 16:43:37.859699 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2 ==QUERY== 16:45:41.010437 08:81:f4:a6:dc:01 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80:52:0:136c::fe > ff02::1: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener querymax resp delay: 10000 addr: :: ==RESPONSE== 16:45:41.843696 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2 From [0]: "Routers are devices that have their procfs forwarding entry, /proc/sys/net/ipv6/conf/all/forwarding, set. Routers join three multicast address groups, in addition to the two multicast group that each host joins and that were mentioned earlier. These are the link-local all-routers multicast group (ff02::2), interface-local all routers multicast group (ff01::2), and site-local all routers multicast group (ff05::2)." In our case, the forwarding value is set to '1' so the backup router belongs to the multicast group and, thus, receives the queries and sends the corresponding answers, plus it will send periodic reports according to mldv1_unsolicited_report_interval and mldv2_unsolicited_report_interval values: [root@overcloud-controller-0 neutron]# cat /proc/sys/net/ipv6/conf/qg-3f327e61-8b/forwarding 1 I have set 'forwarding' to 0 and I cannot see any more packets coming out the gateway interface so that apparently solves the issue. I need to investigate if this has any unwanted side-effect. Another option would be to completely disable ipv6 on the gateway interface of backup instances by setting "disable_ipv6" to 1. I'll leave the controller0 with this setting (forwarding=0) so that you can check but I saw no packets from the past 5 minutes. [0] http://apprize.info/linux/kernel/9.html I've proposed the following patch (the backport shouldn't be too hard): https://review.openstack.org/442518 For our deployment, we have an external network (VLAN) that is also the floating IP network for the cloud. The external network connects to the internal network through a router. The router has the external network as the gateway. This is the router that is exposed to the external/provider network. This is a typical setup when you want VMs inside the cloud to have Floating IPs for external access. The network has the following setup... openstack network create --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 100 external openstack subnet create external --network external --dhcp --allocation-pool start=10.19.108.150,end=10.19.108.250 --gateway 10.19.108.254 --subnet-range 10.19.108.0/24 --dns-nameserver 10.19.5.19 openstack router create external openstack router add subnet external default neutron router-gateway-set external external The cloud is deployed using Director with an "Isolated Networks" scheme. The external network in the standard "Isolated Networks" template is access by using the commands above. Aaron So, even though I think the patch should fix the bug, I've tried to reproduce it on a freshly created devstack environment and, for some reason, it behaves differently: - Two nodes, devstack and worker. devstack node has the HA router running as 'master' while worker is the backup: [vagrant@worker ~]$ neutron l3-agent-list-hosting-router r2 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+----------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------+----------------+-------+----------+ | 7399ef7d-d42b-4e9e-b1e5-728af626f02f | devstack | True | :-) | active | | d988f563-2834-473b-9429-22085c06bd43 | worker | True | :-) | standby | +--------------------------------------+----------+----------------+-------+----------+ Master: [vagrant@devstack ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 cat /proc/sys/net/ipv6/conf/qg-0b634c73-f9/forwarding 1 Backup: [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 cat /proc/sys/net/ipv6/conf/qg-0b634c73-f9/forwarding cat: /proc/sys/net/ipv6/conf/qg-0b634c73-f9/forwarding: No such file or directory Then I block VRRP traffic on the backup instance to force the failover: [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 iptables -A INPUT -p vrrp -i ha-0dae3275-16 -j DROP Both nodes become active now: [vagrant@worker ~]$ neutron l3-agent-list-hosting-router r2 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+----------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------+----------------+-------+----------+ | 7399ef7d-d42b-4e9e-b1e5-728af626f02f | devstack | True | :-) | active | | d988f563-2834-473b-9429-22085c06bd43 | worker | True | :-) | active | +--------------------------------------+----------+----------------+-------+----------+ I remove the DROP rule and worker goes to backup again: [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 cat /proc/sys/net/ipv6/conf/qg-0b634c73-f9/forwarding 1 [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 iptables -D INPUT -p vrrp -i ha-0dae3275-16 -j DROP [vagrant@worker ~]$ neutron l3-agent-list-hosting-router r2 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+----------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------+----------------+-------+----------+ | 7399ef7d-d42b-4e9e-b1e5-728af626f02f | devstack | True | :-) | active | | d988f563-2834-473b-9429-22085c06bd43 | worker | True | :-) | standby | +--------------------------------------+----------+----------------+-------+----------+ [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 cat /proc/sys/net/ipv6/conf/qg-0b634c73-f9/forwarding cat: /proc/sys/net/ipv6/conf/qg-0b634c73-f9/forwarding: No such file or directory The important thing here is that the qg-xxxx interface is up in both nodes but it doesn't show up in the proc entry for IPv6 (but it does for IPv4): Backup: [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 ls /proc/sys/net/ipv6/conf/ all default ha-0dae3275-16 lo [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 ls /proc/sys/net/ipv4/conf/ all default ha-0dae3275-16 lo qg-0b634c73-f9 Master: [vagrant@devstack ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 ls /proc/sys/net/ipv6/conf/ all default ha-29b6ee28-fd lo qg-0b634c73-f9 [vagrant@devstack ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 ls /proc/sys/net/ipv4/conf/ all default ha-29b6ee28-fd lo qg-0b634c73-f9 As far as I can tell after looking through Neutron code, it must be done somewhere else and I suspected of keepalived but I guess we're both running the same keepalived version, right?: [vagrant@worker ~]$ keepalived -v Keepalived v1.2.13 (11/05,2016) I'll try to dig further into it and try to find why the qg interface does not show up for ipv6 on backup instances by default in my setup. This would solve the issue without my patch and hence without making us dependent on the l3 agent. And then, repeating the tests it doesn't behave that way: [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 iptables -A INPUT -p vrrp -i ha-0dae3275-16 -j DROP [vagrant@worker ~]$ neutron l3-agent-list-hosting-router r2 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+----------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------+----------------+-------+----------+ | 7399ef7d-d42b-4e9e-b1e5-728af626f02f | devstack | True | :-) | active | | d988f563-2834-473b-9429-22085c06bd43 | worker | True | :-) | standby | +--------------------------------------+----------+----------------+-------+----------+ [vagrant@worker ~]$ neutron l3-agent-list-hosting-router r2 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+----------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------+----------------+-------+----------+ | 7399ef7d-d42b-4e9e-b1e5-728af626f02f | devstack | True | :-) | active | | d988f563-2834-473b-9429-22085c06bd43 | worker | True | :-) | active | +--------------------------------------+----------+----------------+-------+----------+ [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 cat /proc/sys/net/ipv6/conf/all/forwarding 1 [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 iptables -D INPUT -p vrrp -i ha-0dae3275-16 -j DROP [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 cat /proc/sys/net/ipv6/conf/all/forwarding 1 [vagrant@worker ~]$ neutron l3-agent-list-hosting-router r2 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+----------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------+----------------+-------+----------+ | 7399ef7d-d42b-4e9e-b1e5-728af626f02f | devstack | True | :-) | active | | d988f563-2834-473b-9429-22085c06bd43 | worker | True | :-) | active | +--------------------------------------+----------+----------------+-------+----------+ [vagrant@worker ~]$ neutron l3-agent-list-hosting-router r2 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+----------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------+----------------+-------+----------+ | 7399ef7d-d42b-4e9e-b1e5-728af626f02f | devstack | True | :-) | active | | d988f563-2834-473b-9429-22085c06bd43 | worker | True | :-) | standby | +--------------------------------------+----------+----------------+-------+----------+ [vagrant@worker ~]$ sudo ip netns exec qrouter-d0809e46-bf8d-43c6-b053-7f491fab8173 cat /proc/sys/net/ipv6/conf/all/forwarding 1 I need to find out why the behavior is not consistent across tests. However my patch should fix it either way although making it dependent on the l3 agent for IPv6 failover. Fixed on master, backports up to upstream Newton and Ocata. Hi Aaron , I need clarification for reproduction steps This problem is reproducible in our environment as follows: 1. Deploy OSP10 2. Create external network 3. Create external subnet (IPv4) 4. Create an internal network and VM 5. Attach floating ip 6. ssh into the VM through the FIP or ping the FIP 7. you will start to see ssh freeze or the ping fail occasionally 4. ipv4 or ipv6 network? tnx heat-admin@controller-1 ~]$ sudo -i [root@controller-1 ~]# sudo ip netns exec qrouter-76951c1f-10b8-4918-8ce5-1732777ca700 cat /proc/sys/net/ipv6/conf/qg-b737c03f-93/forwarding 1 [root@controller-1 ~]# reboot Connection to 192.168.24.17 closed by remote host. Connection to 192.168.24.17 closed. [stack@undercloud-0 ~]$ ssh heat-admin.24.17 ssh: connect to host 192.168.24.17 port 22: Connection refused [stack@undercloud-0 ~]$ ssh heat-admin.24.17 Last login: Thu Aug 17 11:14:16 2017 from 192.168.24.1 [heat-admin@controller-1 ~]$ sudo -i [root@controller-1 ~]# sudo ip netns exec qrouter-76951c1f-10b8-4918-8ce5-1732777ca700 cat /proc/sys/net/ipv6/conf/qg-b737c03f-93/forwarding 0 After switching to backup the qg-b737c03f-93/forwarding goes from 1 to 0 (In reply to Alexander Stafeyev from comment #35) > Hi Aaron , I need clarification for reproduction steps > > > This problem is reproducible in our environment as follows: > > 1. Deploy OSP10 > 2. Create external network > 3. Create external subnet (IPv4) > 4. Create an internal network and VM > 5. Attach floating ip > 6. ssh into the VM through the FIP or ping the FIP > 7. you will start to see ssh freeze or the ping fail occasionally > > 4. ipv4 or ipv6 network? > > tnx ipv4 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2663 |