Bug 1310783 - IPv6 NDP seems to cause packet drops in qrouter namespace. [NEEDINFO]
IPv6 NDP seems to cause packet drops in qrouter namespace.
Status: CLOSED DUPLICATE of bug 1300580
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
medium Severity medium
: ---
: 7.0 (Kilo)
Assigned To: Assaf Muller
Toni Freger
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-22 11:19 EST by Jeremy
Modified: 2017-06-05 10:32 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-05 10:32:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
amuller: needinfo? (jmelvin)
amuller: needinfo? (jmelvin)


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1520517 None None None 2016-02-23 07:55 EST

  None (edit)
Description Jeremy 2016-02-22 11:19:40 EST
Description of problem:IPv6 NDP seems to cause packet drops in qrouter namespace. Examples shown in Additional info section.


Version-Release number of selected component (if applicable):
openstack-neutron-2015.1.2-3.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1.ping the controller from within the qrouter namespace
2.note packet drop approximately 1 sec after  IPV6 NDP
3.

Actual results:
packet drop

Expected results:
no packet drop

Additional info:

Following controller qg gateway qg-9a031758-ff is the provider network gateway router.

[root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 ifconfig -a
ha-b720d7cf-7e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.192.11  netmask 255.255.192.0  broadcast 169.254.255.255
        inet6 fe80::f816:3eff:fe83:94d4  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:83:94:d4  txqueuelen 0  (Ethernet)
        RX packets 5  bytes 430 (430.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4266  bytes 205164 (200.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qg-9a031758-ff: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.10.210  netmask 255.255.255.0  broadcast 0.0.0.0
        ether fa:16:3e:84:cc:2e  txqueuelen 0  (Ethernet)
        RX packets 1576834  bytes 452385685 (431.4 MiB)
        RX errors 0  dropped 12919  overruns 0  frame 0
        TX packets 1339968  bytes 24531846098 (22.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qr-fe506d80-31: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.202.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::f816:3eff:fe93:502f  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:93:50:2f  txqueuelen 0  (Ethernet)
        RX packets 1337817  bytes 24531796561 (22.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1502831  bytes 439733880 (419.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

If I am pinging from controller through ip netns as following command:
[root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 ping 192.168.10.203
PING 192.168.10.203 (192.168.10.203) 56(84) bytes of data.
64 bytes from 192.168.10.203: icmp_seq=1 ttl=64 time=0.316 ms
64 bytes from 192.168.10.203: icmp_seq=2 ttl=64 time=0.122 ms
64 bytes from 192.168.10.203: icmp_seq=4 ttl=64 time=0.088 ms
64 bytes from 192.168.10.203: icmp_seq=5 ttl=64 time=0.097 ms
64 bytes from 192.168.10.203: icmp_seq=6 ttl=64 time=0.122 ms
64 bytes from 192.168.10.203: icmp_seq=7 ttl=64 time=0.123 ms
64 bytes from 192.168.10.203: icmp_seq=8 ttl=64 time=0.130 ms
64 bytes from 192.168.10.203: icmp_seq=9 ttl=64 time=0.106 ms
64 bytes from 192.168.10.203: icmp_seq=10 ttl=64 time=0.099 ms
64 bytes from 192.168.10.203: icmp_seq=11 ttl=64 time=0.107 ms
64 bytes from 192.168.10.203: icmp_seq=12 ttl=64 time=0.188 ms
64 bytes from 192.168.10.203: icmp_seq=14 ttl=64 time=0.812 ms
64 bytes from 192.168.10.203: icmp_seq=15 ttl=64 time=0.122 ms
64 bytes from 192.168.10.203: icmp_seq=16 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=17 ttl=64 time=0.201 ms
64 bytes from 192.168.10.203: icmp_seq=18 ttl=64 time=0.144 ms
64 bytes from 192.168.10.203: icmp_seq=19 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=20 ttl=64 time=0.101 ms
64 bytes from 192.168.10.203: icmp_seq=21 ttl=64 time=0.090 ms
64 bytes from 192.168.10.203: icmp_seq=22 ttl=64 time=0.142 ms
64 bytes from 192.168.10.203: icmp_seq=24 ttl=64 time=0.267 ms
64 bytes from 192.168.10.203: icmp_seq=25 ttl=64 time=0.085 ms
64 bytes from 192.168.10.203: icmp_seq=26 ttl=64 time=0.136 ms
64 bytes from 192.168.10.203: icmp_seq=27 ttl=64 time=0.133 ms
64 bytes from 192.168.10.203: icmp_seq=28 ttl=64 time=0.117 ms
64 bytes from 192.168.10.203: icmp_seq=29 ttl=64 time=0.681 ms
64 bytes from 192.168.10.203: icmp_seq=30 ttl=64 time=0.108 ms
64 bytes from 192.168.10.203: icmp_seq=31 ttl=64 time=0.136 ms
64 bytes from 192.168.10.203: icmp_seq=32 ttl=64 time=0.109 ms
64 bytes from 192.168.10.203: icmp_seq=35 ttl=64 time=0.284 ms
64 bytes from 192.168.10.203: icmp_seq=36 ttl=64 time=0.098 ms
64 bytes from 192.168.10.203: icmp_seq=37 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=38 ttl=64 time=0.708 ms
64 bytes from 192.168.10.203: icmp_seq=39 ttl=64 time=0.156 ms
64 bytes from 192.168.10.203: icmp_seq=40 ttl=64 time=0.121 ms
64 bytes from 192.168.10.203: icmp_seq=41 ttl=64 time=0.118 ms
64 bytes from 192.168.10.203: icmp_seq=42 ttl=64 time=0.175 ms
64 bytes from 192.168.10.203: icmp_seq=45 ttl=64 time=0.242 ms
64 bytes from 192.168.10.203: icmp_seq=46 ttl=64 time=0.104 ms
64 bytes from 192.168.10.203: icmp_seq=47 ttl=64 time=0.107 ms
64 bytes from 192.168.10.203: icmp_seq=48 ttl=64 time=0.101 ms
64 bytes from 192.168.10.203: icmp_seq=49 ttl=64 time=0.130 ms
64 bytes from 192.168.10.203: icmp_seq=50 ttl=64 time=0.128 ms
64 bytes from 192.168.10.203: icmp_seq=51 ttl=64 time=0.125 ms
64 bytes from 192.168.10.203: icmp_seq=52 ttl=64 time=0.140 ms
64 bytes from 192.168.10.203: icmp_seq=54 ttl=64 time=0.256 ms
64 bytes from 192.168.10.203: icmp_seq=55 ttl=64 time=0.121 ms
64 bytes from 192.168.10.203: icmp_seq=56 ttl=64 time=0.130 ms
64 bytes from 192.168.10.203: icmp_seq=57 ttl=64 time=0.141 ms
64 bytes from 192.168.10.203: icmp_seq=58 ttl=64 time=0.656 ms
64 bytes from 192.168.10.203: icmp_seq=59 ttl=64 time=0.081 ms
64 bytes from 192.168.10.203: icmp_seq=60 ttl=64 time=0.123 ms
64 bytes from 192.168.10.203: icmp_seq=61 ttl=64 time=0.124 ms
64 bytes from 192.168.10.203: icmp_seq=62 ttl=64 time=0.135 ms
64 bytes from 192.168.10.203: icmp_seq=65 ttl=64 time=0.308 ms
64 bytes from 192.168.10.203: icmp_seq=66 ttl=64 time=0.116 ms
64 bytes from 192.168.10.203: icmp_seq=67 ttl=64 time=0.125 ms
64 bytes from 192.168.10.203: icmp_seq=68 ttl=64 time=0.126 ms


On second window, every 10 seconds, I see these ICMPv6 neighbor advertisement.  One second after, the reply ICMP packet is dropped.
You can review  ping sequence 53 as example 

[root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 tcpdump -n -i qg-9a031758-ff ether host fa:16:3e:84:cc:2e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qg-9a031758-ff, link-type EN10MB (Ethernet), capture size 65535 bytes
22:18:39.871654 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 40, length 64
22:18:39.871749 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 40, length 64
22:18:40.871624 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 41, length 64
22:18:40.871719 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 41, length 64
22:18:41.871645 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 42, length 64
22:18:41.871785 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 42, length 64
22:18:42.026738 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026757 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026796 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026801 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026822 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026827 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026848 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026853 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.026872 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:42.026877 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:42.871632 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 43, length 64
22:18:43.871648 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 44, length 64
22:18:44.871631 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 45, length 64
22:18:44.871850 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 45, length 64
22:18:45.871626 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 46, length 64
22:18:45.871706 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 46, length 64
22:18:46.871630 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 47, length 64
22:18:46.871714 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 47, length 64
22:18:47.871653 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 48, length 64
22:18:47.871728 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 48, length 64
22:18:48.871658 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 49, length 64
22:18:48.871764 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 49, length 64
22:18:49.871653 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 50, length 64
22:18:49.871756 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 50, length 64
22:18:50.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 51, length 64
22:18:50.871734 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 51, length 64
22:18:51.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 52, length 64
22:18:51.871749 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 52, length 64
22:18:52.032028 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032072 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032131 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032138 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032184 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032191 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032242 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032248 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.032284 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28
22:18:52.032295 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32
22:18:52.871654 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 53, length 64
22:18:53.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 54, length 64
22:18:53.871864 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 54, length 64
22:18:54.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 55, length 64
22:18:54.871730 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 55, length 64
22:18:55.871628 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 56, length 64
22:18:55.871732 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 56, length 64
22:18:56.871658 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 57, length 64
22:18:56.871774 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 57, length 64
^C
53 packets captured
53 packets received by filter
0 packets dropped by kernel


As work around, if I disable IPv6 inside the QG router, then the packet drop stopped.
# ip netns exec qrouter-42eb5a3e-39b9-4769-8b79-cf3e3e3e0be0 sysctl -w net.ipv6.conf.qg-9318c696-b6.disable_ipv6=1

Seems OVS is not able to handle IPv6 Network Discovery and cause drop packet on IPv4.
If I disable IPv6 on QG router, then that means I won't have IPv6 capability.
Comment 1 Jakub Libosvar 2016-02-23 07:55:09 EST
The symptoms appear to be pretty much similar to https://bugs.launchpad.net/neutron/+bug/1520517
Comment 8 Assaf Muller 2017-06-05 10:32:31 EDT
We delivered a fix in [1], we provided a hotfix for this particular customer over a year ago and the customer ticket has since been closed. I'm closing this bug as a duplicate.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1300580

*** This bug has been marked as a duplicate of bug 1300580 ***

Note You need to log in before you can comment on or make changes to this bug.