1310783 – IPv6 NDP seems to cause packet drops in qrouter namespace.

Bug 1310783 - IPv6 NDP seems to cause packet drops in qrouter namespace.

Summary: IPv6 NDP seems to cause packet drops in qrouter namespace.

Keywords:
Status:	CLOSED DUPLICATE of bug 1300580
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	7.0 (Kilo)
Assignee:	Assaf Muller
QA Contact:	Toni Freger
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-02-22 16:19 UTC by Jeremy
Modified:	2023-09-14 03:18 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-06-05 14:32:31 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1520517	0	None	None	None	2016-02-23 12:55:08 UTC

Description Jeremy 2016-02-22 16:19:40 UTC

Description of problem:IPv6 NDP seems to cause packet drops in qrouter namespace. Examples shown in Additional info section.


Version-Release number of selected component (if applicable):
openstack-neutron-2015.1.2-3.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1.ping the controller from within the qrouter namespace
2.note packet drop approximately 1 sec after  IPV6 NDP
3.

Actual results:
packet drop

Expected results:
no packet drop

Additional info:

Following controller qg gateway qg-9a031758-ff is the provider network gateway router.

[root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 ifconfig -a
ha-b720d7cf-7e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.192.11  netmask 255.255.192.0  broadcast 169.254.255.255
        inet6 fe80::f816:3eff:fe83:94d4  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:83:94:d4  txqueuelen 0  (Ethernet)
        RX packets 5  bytes 430 (430.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4266  bytes 205164 (200.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qg-9a031758-ff: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.10.210  netmask 255.255.255.0  broadcast 0.0.0.0
        ether fa:16:3e:84:cc:2e  txqueuelen 0  (Ethernet)
        RX packets 1576834  bytes 452385685 (431.4 MiB)
        RX errors 0  dropped 12919  overruns 0  frame 0
        TX packets 1339968  bytes 24531846098 (22.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qr-fe506d80-31: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.202.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::f816:3eff:fe93:502f  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:93:50:2f  txqueuelen 0  (Ethernet)
        RX packets 1337817  bytes 24531796561 (22.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1502831  bytes 439733880 (419.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

If I am pinging from controller through ip netns as following command:
[root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 ping 192.168.10.203
PING 192.168.10.203 (192.168.10.203) 56(84) bytes of data.
64 bytes from 192.168.10.203: icmp_seq=1 ttl=64 time=0.316 ms
64 bytes from 192.168.10.203: icmp_seq=2 ttl=64 time=0.122 ms
64 bytes from 192.168.10.203: icmp_seq=4 ttl=64 time=0.088 ms
64 bytes from 192.168.10.203: icmp_seq=5 ttl=64 time=0.097 ms
64 bytes from 192.168.10.203: icmp_seq=6 ttl=64 time=0.122 ms
64 bytes from 192.168.10.203: icmp_seq=7 ttl=64 time=0.123 ms
64 bytes from 192.168.10.203: icmp_seq=8 ttl=64 time=0.130 ms
64 bytes from 192.168.10.203: icmp_seq=9 ttl=64 time=0.106 ms
64 bytes from 192.168.10.203: icmp_seq=10 ttl=64 time=0.099 ms
64 bytes from 192.168.10.203: icmp_seq=11 ttl=64 time=0.107 ms
64 bytes from 192.168.10.203: icmp_seq=12 ttl=64 time=0.188 ms
64 bytes from 192.168.10.203: icmp_seq=14 ttl=64 time=0.812 ms
64 bytes from 192.168.10.203: icmp_seq=15 ttl=64 time=0.122 ms
64 bytes from 192.168.10.203: icmp_seq=16 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=17 ttl=64 time=0.201 ms
64 bytes from 192.168.10.203: icmp_seq=18 ttl=64 time=0.144 ms
64 bytes from 192.168.10.203: icmp_seq=19 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=20 ttl=64 time=0.101 ms
64 bytes from 192.168.10.203: icmp_seq=21 ttl=64 time=0.090 ms
64 bytes from 192.168.10.203: icmp_seq=22 ttl=64 time=0.142 ms
64 bytes from 192.168.10.203: icmp_seq=24 ttl=64 time=0.267 ms
64 bytes from 192.168.10.203: icmp_seq=25 ttl=64 time=0.085 ms
64 bytes from 192.168.10.203: icmp_seq=26 ttl=64 time=0.136 ms
64 bytes from 192.168.10.203: icmp_seq=27 ttl=64 time=0.133 ms
64 bytes from 192.168.10.203: icmp_seq=28 ttl=64 time=0.117 ms
64 bytes from 192.168.10.203: icmp_seq=29 ttl=64 time=0.681 ms
64 bytes from 192.168.10.203: icmp_seq=30 ttl=64 time=0.108 ms
64 bytes from 192.168.10.203: icmp_seq=31 ttl=64 time=0.136 ms
64 bytes from 192.168.10.203: icmp_seq=32 ttl=64 time=0.109 ms
64 bytes from 192.168.10.203: icmp_seq=35 ttl=64 time=0.284 ms
64 bytes from 192.168.10.203: icmp_seq=36 ttl=64 time=0.098 ms
64 bytes from 192.168.10.203: icmp_seq=37 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=38 ttl=64 time=0.708 ms
64 bytes from 192.168.10.203: icmp_seq=39 ttl=64 time=0.156 ms
64 bytes from 192.168.10.203: icmp_seq=40 ttl=64 time=0.121 ms
64 bytes from 192.168.10.203: icmp_seq=41 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=42 ttl=64 time=0.175 ms
64 bytes from 192.168.10.203: icmp_seq=45 ttl=64 time=0.242 ms
64 bytes from 192.168.10.203: icmp_seq=46 ttl=64 time=0.104 ms
64 bytes from 192.168.10.203: icmp_seq=47 ttl=64 time=0.107 ms
64 bytes from 192.168.10.203: icmp_seq=48 ttl=64 time=0.101 ms
64 bytes from 192.168.10.203: icmp_seq=49 ttl=64 time=0.130 ms
64 bytes from 192.168.10.203: icmp_seq=50 ttl=64 time=0.128 ms
64 bytes from 192.168.10.203: icmp_seq=51 ttl=64 time=0.125 ms
64 bytes from 192.168.10.203: icmp_seq=52 ttl=64 time=0.140 ms
64 bytes from 192.168.10.203: icmp_seq=54 ttl=64 time=0.256 ms
64 bytes from 192.168.10.203: icmp_seq=55 ttl=64 time=0.121 ms
64 bytes from 192.168.10.203: icmp_seq=56 ttl=64 time=0.130 ms
64 bytes from 192.168.10.203: icmp_seq=57 ttl=64 time=0.141 ms
64 bytes from 192.168.10.203: icmp_seq=58 ttl=64 time=0.656 ms
64 bytes from 192.168.10.203: icmp_seq=59 ttl=64 time=0.081 ms
64 bytes from 192.168.10.203: icmp_seq=60 ttl=64 time=0.123 ms
64 bytes from 192.168.10.203: icmp_seq=61 ttl=64 time=0.124 ms
64 bytes from 192.168.10.203: icmp_seq=62 ttl=64 time=0.135 ms
64 bytes from 192.168.10.203: icmp_seq=65 ttl=64 time=0.308 ms
64 bytes from 192.168.10.203: icmp_seq=66 ttl=64 time=0.116 ms
64 bytes from 192.168.10.203: icmp_seq=67 ttl=64 time=0.125 ms
64 bytes from 192.168.10.203: icmp_seq=68 ttl=64 time=0.126 ms


On second window, every 10 seconds, I see these ICMPv6 neighbor advertisement.  One second after, the reply ICMP packet is dropped.
You can review  ping sequence 53 as example 

[root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 tcpdump -n -i qg-9a031758-ff ether host fa:16:3e:84:cc:2e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qg-9a031758-ff, link-type EN10MB (Ethernet), capture size 65535 bytes
22:18:39.871654 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 40, length 64
22:18:39.871749 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 40, length 64
22:18:40.871624 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 41, length 64
22:18:40.871719 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 41, length 64
22:18:41.871645 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 42, length 64
22:18:41.871785 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 42, length 64
22:18:42.026738 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026757 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026796 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026801 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026822 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026827 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026848 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026853 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026872 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026877 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.871632 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 43, length 64
22:18:43.871648 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 44, length 64
22:18:44.871631 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 45, length 64
22:18:44.871850 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 45, length 64
22:18:45.871626 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 46, length 64
22:18:45.871706 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 46, length 64
22:18:46.871630 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 47, length 64
22:18:46.871714 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 47, length 64
22:18:47.871653 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 48, length 64
22:18:47.871728 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 48, length 64
22:18:48.871658 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 49, length 64
22:18:48.871764 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 49, length 64
22:18:49.871653 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 50, length 64
22:18:49.871756 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 50, length 64
22:18:50.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 51, length 64
22:18:50.871734 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 51, length 64
22:18:51.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 52, length 64
22:18:51.871749 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 52, length 64
22:18:52.032028 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032072 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032131 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032138 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032184 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032191 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032242 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032248 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032284 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032295 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.871654 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 53, length 64
22:18:53.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 54, length 64
22:18:53.871864 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 54, length 64
22:18:54.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 55, length 64
22:18:54.871730 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 55, length 64
22:18:55.871628 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 56, length 64
22:18:55.871732 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 56, length 64
22:18:56.871658 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 57, length 64
22:18:56.871774 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 57, length 64
^C
53 packets captured
53 packets received by filter
0 packets dropped by kernel


As work around, if I disable IPv6 inside the QG router, then the packet drop stopped.
# ip netns exec qrouter-42eb5a3e-39b9-4769-8b79-cf3e3e3e0be0 sysctl -w net.ipv6.conf.qg-9318c696-b6.disable_ipv6=1

Seems OVS is not able to handle IPv6 Network Discovery and cause drop packet on IPv4.
If I disable IPv6 on QG router, then that means I won't have IPv6 capability.

Comment 1 Jakub Libosvar 2016-02-23 12:55:09 UTC

The symptoms appear to be pretty much similar to https://bugs.launchpad.net/neutron/+bug/1520517

Comment 8 Assaf Muller 2017-06-05 14:32:31 UTC

We delivered a fix in [1], we provided a hotfix for this particular customer over a year ago and the customer ticket has since been closed. I'm closing this bug as a duplicate.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1300580

*** This bug has been marked as a duplicate of bug 1300580 ***

Comment 9 Red Hat Bugzilla 2023-09-14 03:18:20 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days

Note You need to log in before you can comment on or make changes to this bug.