Bug 1310783
| Summary: | IPv6 NDP seems to cause packet drops in qrouter namespace. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jeremy <jmelvin> |
| Component: | openstack-neutron | Assignee: | Assaf Muller <amuller> |
| Status: | CLOSED DUPLICATE | QA Contact: | Toni Freger <tfreger> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.0 (Kilo) | CC: | amuller, chrisw, jlibosva, jmelvin, nyechiel, srevivo |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | 7.0 (Kilo) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-06-05 14:32:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The symptoms appear to be pretty much similar to https://bugs.launchpad.net/neutron/+bug/1520517 We delivered a fix in [1], we provided a hotfix for this particular customer over a year ago and the customer ticket has since been closed. I'm closing this bug as a duplicate. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1300580 *** This bug has been marked as a duplicate of bug 1300580 *** The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem:IPv6 NDP seems to cause packet drops in qrouter namespace. Examples shown in Additional info section. Version-Release number of selected component (if applicable): openstack-neutron-2015.1.2-3.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1.ping the controller from within the qrouter namespace 2.note packet drop approximately 1 sec after IPV6 NDP 3. Actual results: packet drop Expected results: no packet drop Additional info: Following controller qg gateway qg-9a031758-ff is the provider network gateway router. [root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 ifconfig -a ha-b720d7cf-7e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 169.254.192.11 netmask 255.255.192.0 broadcast 169.254.255.255 inet6 fe80::f816:3eff:fe83:94d4 prefixlen 64 scopeid 0x20<link> ether fa:16:3e:83:94:d4 txqueuelen 0 (Ethernet) RX packets 5 bytes 430 (430.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 4266 bytes 205164 (200.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qg-9a031758-ff: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.10.210 netmask 255.255.255.0 broadcast 0.0.0.0 ether fa:16:3e:84:cc:2e txqueuelen 0 (Ethernet) RX packets 1576834 bytes 452385685 (431.4 MiB) RX errors 0 dropped 12919 overruns 0 frame 0 TX packets 1339968 bytes 24531846098 (22.8 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qr-fe506d80-31: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.16.202.1 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::f816:3eff:fe93:502f prefixlen 64 scopeid 0x20<link> ether fa:16:3e:93:50:2f txqueuelen 0 (Ethernet) RX packets 1337817 bytes 24531796561 (22.8 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1502831 bytes 439733880 (419.3 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 If I am pinging from controller through ip netns as following command: [root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 ping 192.168.10.203 PING 192.168.10.203 (192.168.10.203) 56(84) bytes of data. 64 bytes from 192.168.10.203: icmp_seq=1 ttl=64 time=0.316 ms 64 bytes from 192.168.10.203: icmp_seq=2 ttl=64 time=0.122 ms 64 bytes from 192.168.10.203: icmp_seq=4 ttl=64 time=0.088 ms 64 bytes from 192.168.10.203: icmp_seq=5 ttl=64 time=0.097 ms 64 bytes from 192.168.10.203: icmp_seq=6 ttl=64 time=0.122 ms 64 bytes from 192.168.10.203: icmp_seq=7 ttl=64 time=0.123 ms 64 bytes from 192.168.10.203: icmp_seq=8 ttl=64 time=0.130 ms 64 bytes from 192.168.10.203: icmp_seq=9 ttl=64 time=0.106 ms 64 bytes from 192.168.10.203: icmp_seq=10 ttl=64 time=0.099 ms 64 bytes from 192.168.10.203: icmp_seq=11 ttl=64 time=0.107 ms 64 bytes from 192.168.10.203: icmp_seq=12 ttl=64 time=0.188 ms 64 bytes from 192.168.10.203: icmp_seq=14 ttl=64 time=0.812 ms 64 bytes from 192.168.10.203: icmp_seq=15 ttl=64 time=0.122 ms 64 bytes from 192.168.10.203: icmp_seq=16 ttl=64 time=0.118 ms 64 bytes from 192.168.10.203: icmp_seq=17 ttl=64 time=0.201 ms 64 bytes from 192.168.10.203: icmp_seq=18 ttl=64 time=0.144 ms 64 bytes from 192.168.10.203: icmp_seq=19 ttl=64 time=0.118 ms 64 bytes from 192.168.10.203: icmp_seq=20 ttl=64 time=0.101 ms 64 bytes from 192.168.10.203: icmp_seq=21 ttl=64 time=0.090 ms 64 bytes from 192.168.10.203: icmp_seq=22 ttl=64 time=0.142 ms 64 bytes from 192.168.10.203: icmp_seq=24 ttl=64 time=0.267 ms 64 bytes from 192.168.10.203: icmp_seq=25 ttl=64 time=0.085 ms 64 bytes from 192.168.10.203: icmp_seq=26 ttl=64 time=0.136 ms 64 bytes from 192.168.10.203: icmp_seq=27 ttl=64 time=0.133 ms 64 bytes from 192.168.10.203: icmp_seq=28 ttl=64 time=0.117 ms 64 bytes from 192.168.10.203: icmp_seq=29 ttl=64 time=0.681 ms 64 bytes from 192.168.10.203: icmp_seq=30 ttl=64 time=0.108 ms 64 bytes from 192.168.10.203: icmp_seq=31 ttl=64 time=0.136 ms 64 bytes from 192.168.10.203: icmp_seq=32 ttl=64 time=0.109 ms 64 bytes from 192.168.10.203: icmp_seq=35 ttl=64 time=0.284 ms 64 bytes from 192.168.10.203: icmp_seq=36 ttl=64 time=0.098 ms 64 bytes from 192.168.10.203: icmp_seq=37 ttl=64 time=0.118 ms 64 bytes from 192.168.10.203: icmp_seq=38 ttl=64 time=0.708 ms 64 bytes from 192.168.10.203: icmp_seq=39 ttl=64 time=0.156 ms 64 bytes from 192.168.10.203: icmp_seq=40 ttl=64 time=0.121 ms 64 bytes from 192.168.10.203: icmp_seq=41 ttl=64 time=0.118 ms 64 bytes from 192.168.10.203: icmp_seq=42 ttl=64 time=0.175 ms 64 bytes from 192.168.10.203: icmp_seq=45 ttl=64 time=0.242 ms 64 bytes from 192.168.10.203: icmp_seq=46 ttl=64 time=0.104 ms 64 bytes from 192.168.10.203: icmp_seq=47 ttl=64 time=0.107 ms 64 bytes from 192.168.10.203: icmp_seq=48 ttl=64 time=0.101 ms 64 bytes from 192.168.10.203: icmp_seq=49 ttl=64 time=0.130 ms 64 bytes from 192.168.10.203: icmp_seq=50 ttl=64 time=0.128 ms 64 bytes from 192.168.10.203: icmp_seq=51 ttl=64 time=0.125 ms 64 bytes from 192.168.10.203: icmp_seq=52 ttl=64 time=0.140 ms 64 bytes from 192.168.10.203: icmp_seq=54 ttl=64 time=0.256 ms 64 bytes from 192.168.10.203: icmp_seq=55 ttl=64 time=0.121 ms 64 bytes from 192.168.10.203: icmp_seq=56 ttl=64 time=0.130 ms 64 bytes from 192.168.10.203: icmp_seq=57 ttl=64 time=0.141 ms 64 bytes from 192.168.10.203: icmp_seq=58 ttl=64 time=0.656 ms 64 bytes from 192.168.10.203: icmp_seq=59 ttl=64 time=0.081 ms 64 bytes from 192.168.10.203: icmp_seq=60 ttl=64 time=0.123 ms 64 bytes from 192.168.10.203: icmp_seq=61 ttl=64 time=0.124 ms 64 bytes from 192.168.10.203: icmp_seq=62 ttl=64 time=0.135 ms 64 bytes from 192.168.10.203: icmp_seq=65 ttl=64 time=0.308 ms 64 bytes from 192.168.10.203: icmp_seq=66 ttl=64 time=0.116 ms 64 bytes from 192.168.10.203: icmp_seq=67 ttl=64 time=0.125 ms 64 bytes from 192.168.10.203: icmp_seq=68 ttl=64 time=0.126 ms On second window, every 10 seconds, I see these ICMPv6 neighbor advertisement. One second after, the reply ICMP packet is dropped. You can review ping sequence 53 as example [root@overcloud-controller-2 ~]# ip netns exec qrouter-9f915195-5b0a-4513-a6f5-de789ef57950 tcpdump -n -i qg-9a031758-ff ether host fa:16:3e:84:cc:2e tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on qg-9a031758-ff, link-type EN10MB (Ethernet), capture size 65535 bytes 22:18:39.871654 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 40, length 64 22:18:39.871749 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 40, length 64 22:18:40.871624 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 41, length 64 22:18:40.871719 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 41, length 64 22:18:41.871645 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 42, length 64 22:18:41.871785 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 42, length 64 22:18:42.026738 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:42.026757 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:42.026796 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:42.026801 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:42.026822 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:42.026827 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:42.026848 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:42.026853 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:42.026872 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:42.026877 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:42.871632 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 43, length 64 22:18:43.871648 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 44, length 64 22:18:44.871631 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 45, length 64 22:18:44.871850 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 45, length 64 22:18:45.871626 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 46, length 64 22:18:45.871706 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 46, length 64 22:18:46.871630 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 47, length 64 22:18:46.871714 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 47, length 64 22:18:47.871653 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 48, length 64 22:18:47.871728 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 48, length 64 22:18:48.871658 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 49, length 64 22:18:48.871764 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 49, length 64 22:18:49.871653 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 50, length 64 22:18:49.871756 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 50, length 64 22:18:50.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 51, length 64 22:18:50.871734 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 51, length 64 22:18:51.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 52, length 64 22:18:51.871749 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 52, length 64 22:18:52.032028 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:52.032072 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:52.032131 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:52.032138 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:52.032184 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:52.032191 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:52.032242 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:52.032248 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:52.032284 ARP, Request who-has 192.168.10.210 (Broadcast) tell 192.168.10.210, length 28 22:18:52.032295 IP6 fe80::f816:3eff:fe84:cc2e > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:fe84:cc2e, length 32 22:18:52.871654 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 53, length 64 22:18:53.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 54, length 64 22:18:53.871864 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 54, length 64 22:18:54.871633 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 55, length 64 22:18:54.871730 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 55, length 64 22:18:55.871628 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 56, length 64 22:18:55.871732 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 56, length 64 22:18:56.871658 IP 192.168.10.210 > 192.168.10.203: ICMP echo request, id 13147, seq 57, length 64 22:18:56.871774 IP 192.168.10.203 > 192.168.10.210: ICMP echo reply, id 13147, seq 57, length 64 ^C 53 packets captured 53 packets received by filter 0 packets dropped by kernel As work around, if I disable IPv6 inside the QG router, then the packet drop stopped. # ip netns exec qrouter-42eb5a3e-39b9-4769-8b79-cf3e3e3e0be0 sysctl -w net.ipv6.conf.qg-9318c696-b6.disable_ipv6=1 Seems OVS is not able to handle IPv6 Network Discovery and cause drop packet on IPv4. If I disable IPv6 on QG router, then that means I won't have IPv6 capability.