Bug 1773605
| Summary: | OCP IPv6 - Packets dropped on upcall from kernel after SNAT - Failed to acquire udpif_key corresponding to unexpected flow | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Mark McLoughlin <markmc> | |
| Component: | ovn2.11 | Assignee: | Numan Siddique <nusiddiq> | |
| Status: | VERIFIED --- | QA Contact: | Jianlin Shi <jishi> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | RHEL 7.7 | CC: | ctrautma, dcbw, fleitner, jhsiao, mmichels, nusiddiq, ovs-qe, ovs-team, qding, ralongi | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1773598 | |||
| : | 1775160 1776969 1776973 1776994 (view as bug list) | Environment: | ||
| Last Closed: | Type: | --- | ||
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1775160, 1776969, 1776973, 1776994 | |||
|
Description
Mark McLoughlin
2019-11-18 14:45:45 UTC
This could very well be an ovn issue. I will take this BZ and investigate. Patch posted by Numan: https://patchwork.ozlabs.org/patch/1197423/ The fix is available in ovn2.11-2.11.1-20 reproduced with following steps on openvswitch2.12-2.12.0-4.el7fdp.x86_64 and ovn2.12-2.12.0-14.el7fdp.x86_64:
server:
#!/bin/bash
systemctl start openvswitch
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.29.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.29.25
systemctl restart ovn-controller
ip netns add server0
ip link add veth0_s0 type veth peer name veth0_s0_p
ip link set veth0_s0 netns server0
ip netns exec server0 ip link set lo up
ip netns exec server0 ip link set veth0_s0 up
ip netns exec server0 ip link set veth0_s0 address 00:00:00:01:01:02
ip netns exec server0 ip addr add 172.16.2.1/24 dev veth0_s0
ip netns exec server0 ip addr add 2001::1/64 dev veth0_s0
ip netns exec server0 ip route add default via 172.16.2.254 dev veth0_s0
ip netns exec server0 ip -6 route add default via 2001::a dev veth0_s0
ovs-vsctl add-port br-int veth0_s0_p
ip link set veth0_s0_p up
ovs-vsctl set interface veth0_s0_p external_ids:iface-id=ls2p1
ovn-nbctl ls-add ls1
ovn-nbctl lsp-add ls1 ls1p1
ovn-nbctl lsp-set-addresses ls1p1 00:00:00:01:00:02
ovn-nbctl ls-add ls2
ovn-nbctl lsp-add ls2 ls2p1
ovn-nbctl lsp-set-addresses ls2p1 00:00:00:01:01:02
ovn-nbctl lr-add lr1
ovn-nbctl lrp-add lr1 lr1-ls1 00:de:ad:ff:01:01 172.16.1.254/24 2000::a/64
ovn-nbctl lrp-add lr1 lr1-ls2 00:de:ad:ff:01:02 172.16.2.254/24 2001::a/64
ovn-nbctl lrp-add lr1 lr1-ls0 00:de:ad:ff:01:03 192.168.111.254/24 3000::a/64
ovn-nbctl lsp-add ls1 ls1-lr1
ovn-nbctl lsp-set-type ls1-lr1 router
ovn-nbctl lsp-set-addresses ls1-lr1 00:de:ad:ff:01:01
ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1
ovn-nbctl lsp-add ls2 ls2-lr1
ovn-nbctl lsp-set-type ls2-lr1 router
ovn-nbctl lsp-set-addresses ls2-lr1 00:de:ad:ff:01:02
ovn-nbctl lsp-set-options ls2-lr1 router-port=lr1-ls2
ovn-nbctl set Logical_Router_Port lr1-ls2 options:redirect-chassis=hv1
# for public
ovn-nbctl ls-add ls0
ovn-nbctl lsp-add ls0 ls0-lr1
ovn-nbctl lsp-set-addresses ls0-lr1 00:de:ad:ff:01:03
ovn-nbctl lsp-set-type ls0-lr1 router
ovn-nbctl lsp-set-options ls0-lr1 router-port=lr1-ls0
ovn-nbctl lsp-add ls0 ln_port
ovn-nbctl lsp-set-addresses ln_port unknown
ovn-nbctl lsp-set-type ln_port localnet
ovn-nbctl lsp-set-options ln_port network_name=nattest
ovn-nbctl set logical_router lr1 options:chassis=hv1
ovs-vsctl add-br br-nat
ovs-vsctl set open . external_ids:ovn-bridge-mappings=nattest:br-nat
ip link set br-nat up
ip netns add server1 up
ip link add veth0_s1 type veth peer name veth0_s1_p
ip link set veth0_s1 netns server1
ip netns exec server1 ip link set veth0_s1 up
ip netns exec server1 ip addr add 192.168.111.1/24 dev veth0_s1
ip netns exec server1 ip addr add 3000::1/64 dev veth0_s1
ip netns exec server1 ip route add default via 192.168.111.254
ip netns exec server1 ip -6 route add default via 3000::a
ovs-vsctl add-port br-nat veth0_s1_p
ip link set veth0_s1_p up
ovn-nbctl lr-nat-add lr1 dnat_and_snat 3000::55 2001::1
on client:
systemctl start openvswitch
ovs-vsctl set open . external_ids:system-id=hv0 external_ids:ovn-remote=tcp:20.0.29.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.29.26
systemctl restart ovn-controller
ip netns add client0
ip link add veth0_c0 type veth peer name veth0_c0_p
ip link set veth0_c0 netns client0
ip netns exec client0 ip link set lo up
ip netns exec client0 ip link set veth0_c0 up
ip netns exec client0 ip link set veth0_c0 address 00:00:00:01:00:02
ip netns exec client0 ip addr add 172.16.1.1/24 dev veth0_c0
ip netns exec client0 ip addr add 2000::1/64 dev veth0_c0
ip netns exec client0 ip route add default via 172.16.1.254 dev veth0_c0
ip netns exec client0 ip -6 route add default via 2000::a dev veth0_c0
ovs-vsctl add-port br-int veth0_c0_p
ip link set veth0_c0_p up
ovs-vsctl set interface veth0_c0_p external_ids:iface-id=ls1p1
then run ping6 on server:
ip netns exec server0 ping6 3000::2 -c 1
then get the error log in /var/log/openvswitch/ovs-vswitchd.log:
[root@dell-per740-12 bz1773605]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log
2019-11-22T09:02:04.063Z|00001|ofproto_dpif_upcall(handler50)|INFO|received packet on unassociated datapath port 4294967295
2019-11-22T09:02:04.083Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:cfa90710-073d-4516-abfe-5117d64e82ae
with setup on comment 7. on ovn2.12.0-7: when I tried to ping 3000::1 on server0 with ip netns exec server0 ping6 3000::1 -c 1. server1 can't receive the ipv6 ns, then ping6 would always fail. [root@dell-per740-12 bz1773605]# ip netns exec server0 ping6 3000::1 -c 2 PING 3000::1(3000::1) 56 data bytes --- 3000::1 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1001ms [root@dell-per740-12 ~]# ip netns exec server1 tcpdump -i any -nnle tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 04:31:04.816918 B 00:de:ad:ff:01:03 ethertype ARP (0x0806), length 44: Request who-has 192.168.111.254 tell 192.168.111.254, length 28 ^C 1 packet captured 1 packet received by filter 0 packets dropped by kernel <=== no NS received on server1 and on ovn2.12.0-15: [root@dell-per740-12 bz1773605]# rpm -qa | grep ovn ovn2.12-2.12.0-15.el7fdp.x86_64 ovn2.12-host-2.12.0-15.el7fdp.x86_64 ovn2.12-central-2.12.0-15.el7fdp.x86_64 [root@dell-per740-12 bz1773605]# ip netns exec server0 ping6 3000::1 -c 2 PING 3000::1(3000::1) 56 data bytes 64 bytes from 3000::1: icmp_seq=2 ttl=63 time=2.22 ms --- 3000::1 ping statistics --- 2 packets transmitted, 1 received, 50% packet loss, time 1000ms rtt min/avg/max/mdev = 2.228/2.228/2.228/0.000 ms [root@dell-per740-12 ~]# ip netns exec server1 tcpdump -i any -nnle tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 04:46:17.376666 M 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 88: 2001::1 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 3000::1, length 32 <=== NS received 04:46:17.376726 Out fe:89:60:69:07:bf ethertype IPv6 (0x86dd), length 88: 3000::1 > 2001::1: ICMP6, neighbor advertisement, tgt is 3000::1, length 32 04:46:18.375131 In 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 120: 3000::55 > 3000::1: ICMP6, echo request, seq 2, length 64 04:46:18.375226 Out fe:89:60:69:07:bf ethertype IPv6 (0x86dd), length 88: 3000::1 > ff02::1:ff00:55: ICMP6, neighbor solicitation, who has 3000::55, length 32 04:46:18.376061 In 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 88: 3000::55 > 3000::1: ICMP6, neighbor advertisement, tgt is 3000::55, length 32 04:46:18.376094 Out fe:89:60:69:07:bf ethertype IPv6 (0x86dd), length 120: 3000::1 > 3000::55: ICMP6, echo reply, seq 2, length 64 but the error message would also occur the first time ping6 run: [root@dell-per740-12 bz1773605]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log 2019-11-22T09:46:17.388Z|00001|ofproto_dpif_upcall(handler52)|INFO|received packet on unassociated datapath port 4294967295 2019-11-22T09:46:17.389Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:161ddea7-25f6-4ebb-8eec-0f63feaac816 ovs i used is 2.12.0-4: [root@dell-per740-12 bz1773605]# rpm -qa | grep openvs openvswitch2.12-2.12.0-4.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch This bug addresses the issue in OVN so that the IPv6 NS packet generated by ovn-controller is not dropped. Since you have verified that the NS packet is received and not dropped, I would say you can verify this BZ. I have clone this bug on openvswitch2.12 component which should address this issue properly in ovs-vswitchd and those logs should not be seen. reproduced on ovn2.11.1-19: [root@dell-per740-12 bz1773605]# ip netns exec server0 ping6 3000::2 -c 1 PING 3000::2(3000::2) 56 data bytes --- 3000::2 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [root@dell-per740-12 bz1773605]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log 2019-11-28T01:38:32.281Z|00001|ofproto_dpif_upcall(handler50)|INFO|received packet on unassociated datapath port 4294967295 2019-11-28T01:38:32.295Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:8c542e09-56ac-404d-8756-21b3352e6a31 [root@dell-per740-12 bz1773605]# ip netns exec server0 ping6 3000::2 -c 1 PING 3000::2(3000::2) 56 data bytes --- 3000::2 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms <==== also failed to ping6 3000::1 [root@dell-per740-12 bz1773605]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log 2019-11-28T01:38:32.281Z|00001|ofproto_dpif_upcall(handler50)|INFO|received packet on unassociated datapath port 4294967295 2019-11-28T01:38:32.295Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:8c542e09-56ac-404d-8756-21b3352e6a31 2019-11-28T01:38:50.696Z|00002|ofproto_dpif_upcall(handler50)|INFO|received packet on unassociated datapath port 4294967295 2019-11-28T01:38:50.852Z|00002|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:36540ac4-8a29-44fb-835d-dc89aa5ae5be <===== error message [root@dell-per740-12 bz1773605]# rpm -qa | grep -E "openvswitch|ovn" ovn2.11-host-2.11.1-19.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch ovn2.11-central-2.11.1-19.el7fdp.x86_64 openvswitch2.11-2.11.0-26.el7fdp.x86_64 ovn2.11-2.11.1-19.el7fdp.x86_64 [root@dell-per740-12 ~]# ip netns exec server1 tcpdump -i any -nnle tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel <==== no NS received Verified on ovn2.11.1-20: [root@dell-per740-12 bz1773605]# ip netns exec server0 ping6 3000::1 -c 1 PING 3000::1(3000::1) 56 data bytes 64 bytes from 3000::1: icmp_seq=1 ttl=63 time=3.86 ms --- 3000::1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 3.864/3.864/3.864/0.000 ms <==== passed [root@dell-per740-12 bz1773605]# rpm -qa | grep -E "openvswitch|ovn" ovn2.11-2.11.1-20.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch ovn2.11-central-2.11.1-20.el7fdp.x86_64 openvswitch2.11-2.11.0-26.el7fdp.x86_64 ovn2.11-host-2.11.1-20.el7fdp.x86_64 [root@dell-per740-12 bz1773605]# ip netns exec server0 ping6 3000::2 -c 1 PING 3000::2(3000::2) 56 data bytes --- 3000::2 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [root@dell-per740-12 ~]# ip netns exec server1 tcpdump -i any -nnle tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 20:45:07.387857 In 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 120: 3000::55 > 3000::1: ICMP6, echo request, seq 1, length 64 20:45:07.387895 Out ba:81:86:09:e0:cd ethertype IPv6 (0x86dd), length 120: 3000::1 > 3000::55: ICMP6, echo reply, seq 1, length 64 20:46:57.014295 M 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 88: 2001::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has 3000::2, length 32 <==== NS received [root@dell-per740-12 bz1773605]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log 2019-11-28T01:38:32.281Z|00001|ofproto_dpif_upcall(handler50)|INFO|received packet on unassociated datapath port 4294967295 2019-11-28T01:38:32.295Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:8c542e09-56ac-404d-8756-21b3352e6a31 <=== also the error message still appear as NS can be received, based on comment 11, set VERIFIED |