Bug 1854376
Summary: | ICMPv6 packets are going through the wrong path in br-int | |||
---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Slawek Kaplonski <skaplons> | |
Component: | openvswitch | Assignee: | Adrián Moreno <amorenoz> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Eran Kuris <ekuris> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 16.1 (Train) | CC: | ahleihel, akatz, apevec, bcafarel, chrisw, eolivare, fhallal, i.maximets, jlibosva, mgarciac, mleitner, ralonsoh, rhos-maint, tredaelli | |
Target Milestone: | --- | Keywords: | AutomationBlocker, Triaged | |
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | openvswitch2.13-2.13.0-49.el8fdp | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1862153 (view as bug list) | Environment: | ||
Last Closed: | 2020-11-04 13:38:32 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: |
Description
Slawek Kaplonski
2020-07-07 10:34:19 UTC
After some regression tests I found that this commit: commit dbf4a92800d0365cc3ec3c0e99df56e2ba676cb7 Author: Eli Britstein <elibr> Date: Thu Mar 21 07:44:16 2019 +0000 odp-util: Do not rewrite fields with the same values as matched To improve performance and avoid wasting resources for HW offloaded flows, do not rewrite fields that are matched with the same value. Reviewed-by: Roi Dayan <roid> Signed-off-by: Eli Britstein <elibr> Signed-off-by: Ben Pfaff <blp> introduced the issue. We still need to figure out if it's a real bug introduced by the commit or if it's only necessary to change some flows Adrian, Timothy, could you try this: https://patchwork.ozlabs.org/project/openvswitch/list/?series=192595 I don't know if it will fix the issue, but it seems relevant. After Timothy kindly bisected the problem I did some experimentation and what I found is: - At the ofproto level, there is no explicit flow for traffic coming from the qrouter port in br-int so it falls back to ACTION_NORMAL VM interface: 27(qvo5b0e0123-0a): addr:fe:19:7e:45:b4:87 QRouter Interface: 28(qr-2ba2a180-bc): addr:fa:16:3e:bd:9a:76 * On a working version this gets translated into: ( port 10: qvo5b0e0123-0a port 11: vxlan_sys_4789 (vxlan: packet_type=ptap) port 12: qr-2ba2a180-bc (internal) ) recirc_id(0),in_port(12),eth(src=fa:16:3e:bd:9a:76,dst=fa:16:3e:5c:47:bb),eth_type(0x86dd),ipv6(frag=no), packets:1, bytes:118, used:3.485s, actions:10 * However, on a non-working version (containing the patch pointed out by Timothy) the datapath rule is: recirc_id(0),in_port(12),skb_mark(0),eth(src=fa:16:3e:bd:9a:76),eth_type(0x86dd),ipv6(tclass=0/0x3,frag=no), packets:268, bytes:28992, used:1.297s, actions:set(tunnel(tun_id=0x1,src=172.17.2.107,dst=172.17.2.59,ttl=64,tp_dst=4789,flags(df|key))),set(eth(src=fa:16:3f:41:1d:2f)),11 - According to the flow above, all ipv6 (not just icmpv6) traffic coming from the qrouter gets sent through the vxlan interface. As long as there is traffic matching that flow it stays configured in the datapath and the problem persists. - Capturing in the input port (qr-2ba2a180-bc): 16:02:55.478056 fa:16:3e:5c:47:bb > fa:16:3e:bd:9a:76, ethertype 802.1Q (0x8100), length 122: vlan 10, p 0, ethertype IPv6, 2001:db8::f816:3eff:fe5c:47bb > 2001:db8::1: ICMP6, echo request, seq 1, length 64 16:02:55.478101 fa:16:3e:bd:9a:76 > fa:16:3e:5c:47:bb, ethertype 802.1Q (0x8100), length 122: vlan 10, p 0, ethertype IPv6, 2001:db8::1 > 2001:db8::f816:3eff:fe5c:47bb: ICMP6, echo reply, seq 1, length 64 - During this situation, ofproto/trace reports the expected behavior (not the observed one): $ ovs-appctl ofproto/trace br-int in_port=qr-2ba2a180-bc,dl_src=fa:16:3e:bd:9a:76,dl_dst=fa:16:3e:5c:47:bb,dl_vlan=10,icmp6 Flow: icmp6,in_port=28,dl_vlan=10,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=fa:16:3e:bd:9a:76,dl_dst=fa:16:3e:5c:47:bb,ipv6_src=::,ipv6_dst=::,ipv6_label=0x00000,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0 bridge("br-int") ---------------- 0. priority 0, cookie 0xf522faab0d3e9672 goto_table:60 60. dl_vlan=10,dl_dst=fa:16:3e:5c:47:bb, priority 20, cookie 0xf522faab0d3e9672 pop_vlan output:27 Final flow: icmp6,in_port=28,vlan_tci=0x0000,dl_src=fa:16:3e:bd:9a:76,dl_dst=fa:16:3e:5c:47:bb,ipv6_src=::,ipv6_dst=::,ipv6_label=0x00000,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0 Megaflow: recirc_id=0,eth,ipv6,in_port=28,dl_vlan=10,dl_vlan_pcp=0,dl_dst=fa:16:3e:5c:47:bb,nw_frag=no Datapath actions: pop_vlan,10 - If "ovs-appctl dpctl/del-flows" is run in the affected node to trigger a re-translation of the flows, the correct flow will be configured: - Also, if a 5~10 delay is introduced before running the ping6 test, the situation won't happen I am not familiar with the code modified by the patch but from what I've seen it seems that there is an initial ipv6 flow that then get's modified but dl_dst fails to get added to the mask. I'm fine tuning the the testing scripts in order to catch those initial seconds that seem crucial. (In reply to Ilya Maximets from comment #4) > Adrian, Timothy, could you try this: > https://patchwork.ozlabs.org/project/openvswitch/list/?series=192595 > > I don't know if it will fix the issue, but it seems relevant. It does! Thanks Ilya. For the sake of completeness, although Ilya's commit message explains the issue perfectly (https://patchwork.ozlabs.org/project/openvswitch/patch/20200727185848.4089473-1-i.maximets@ovn.org/), the flow that gets configured during the first seconds of VM boot time is: recirc_id(0),in_port(12),skb_mark(0),eth(src=fa:16:3e:bd:9a:76),eth_type(0x86dd),ipv6(tclass=0/0x3,frag=no), packets:8, bytes:776, used:7.574s, actions:push_vlan(vid=10,pcp=0),2,set(tunnel(tun_id=0x1,src=172.17.2.107,dst=172.17.2.59,ttl=64,tp_dst=4789,flags(df|key))),set(eth(src=fa:16:3f:41:1d:2f)),pop_vlan,11,set(tunnel(tun_id=0x1,src=172.17.2.107,dst=172.17.2.139,ttl=64,tp_dst=4789,flags(df|key))),11,set(tunnel(tun_id=0x1,src=172.17.2.107,dst=172.17.2.106,ttl=64,tp_dst=4789,flags(df|key))),11,set(tunnel(tun_id=0x1,src=172.17.2.107,dst=172.17.2.27,ttl=64,tp_dst=4789,flags(df|key))),11,set(eth(src=fa:16:3e:bd:9a:76)),10,13 Without Ilya's patch, this gets modified into: recirc_id(0),in_port(12),skb_mark(0),eth(src=fa:16:3e:bd:9a:76),eth_type(0x86dd),ipv6(tclass=0/0x3,frag=no), packets:0, bytes:0, used:never, actions:set(tunnel(tun_id=0x1,src=172.17.2.107,dst=172.17.2.59,ttl=64,tp_dst=4789,flags(df|key))),set(eth(src=fa:16:3f:41:1d:2f)),11 If you wait long enough until the first rule disappears from the datapath, a correct one is inserted instead. This confirms the issue is related with flow modification as Ilya explains. We are already use openvswitch2.13.x86_64-2.13.0-57.el8fdp and it looks like the issue was fixed. changing to verified. Publicly-available containers (see openstack-openvswitch-base-16.1.2-8 on the Catalog) have version openvswitch2.13-2.13.0-57.el8fdp. Moving to CLOSED CURRENTRELEASE. |