Bug 1992705
Summary: | FFWD2 OSP13->OSP16.2 stopped working for ovn deployments | |||
---|---|---|---|---|
Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Lukas Bezdicka <lbezdick> | |
Component: | OVN | Assignee: | OVN Team <ovnteam> | |
Status: | CLOSED ERRATA | QA Contact: | Jianlin Shi <jishi> | |
Severity: | urgent | Docs Contact: | ||
Priority: | urgent | |||
Version: | FDP 21.F | CC: | ccamposr, ctrautma, dalvarez, dcbw, dceara, jiji, jlibosva, jpretori, kfida, mburns, mmichels, spower | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | ovn-2021-21.06.0-24.el8fdp ovn-2021-21.09.0-5.el8fdp | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 2006744 (view as bug list) | Environment: | ||
Last Closed: | 2021-09-07 18:03:43 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1983106, 1993154, 2006744, 2009741, 2019446 |
Description
Lukas Bezdicka
2021-08-11 15:34:30 UTC
The impact is OVN can't talk to the local ovsdb and e.g. create patch ports between br-int and provider bridges. FYI, it appears that the "datapaths" column was added in August 2019 to OVS, which means it would be present in OVS 2.12+. In general, the OVN team recommends using whatever version of OVS is contemporary with the current version of OVN. When it comes to versions of OVN from the 20.XX series onward (ovn2.13 and ovn-2021 in RHEL), those should be compatible with OVS 2.13+. In this case, trying to upgrade OVN but using OVS 2.11 is going to cause this issue. OVS needs to be at 2.13+. Question for OSP: does 21.06.0-13 work? A bunch of the datapath related code in 21.06 was added to 21.06.0-14 for "ovn-controller: Detect OVS datapath capabilities." @Dan, I don't think that will help in this case. The problem here is a schema mismatch, and that causes errors upon connection when ovn-controller sends its "monitor_cond" (or "monitor_cond_since", I'm not sure which exactly) request. ovn-controller is interested in these columns that do not exist in the current version of OVS, and there's not a way for ovn-controller to fall back to requesting monitoring based on an older schema. Changing issue to MODIFIED. I was wrong with my reply to Dan earlier. It turns out the probing for capabilities relied on a database table that was added during the OVS 2.12 cycle. As such, this exposed the incompatibility with OVS 2.11. The offending commits have been reverted so as to provide a build. A longer-term solution will be created to allow for us to check datapath capabilities without relying on assumptions about the schema. reproduce steps: 1. install openvswitch2.11-2.11.3-86.el8fdp and ovn-2021-21.06.0-17.el8fdp on client 2. install openvswitch2.15-2.15.0-26.el8fdp and ovn-2021-21.06.0-17.el8fdp on server 3. start ovn on server: systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.25 systemctl restart ovn-controller ovn-nbctl ls-add ls1 ovn-nbctl lsp-add ls1 ls1p1 ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1" ovn-nbctl lsp-add ls1 ls1p2 ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:01:02 192.168.1.2" ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1 ip netns add ls1p1 ip link set ls1p1 netns ls1p1 ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01 ip netns exec ls1p1 ip link set ls1p1 up ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1 4. start ovn on client systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv0 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.26 systemctl restart ovn-controller ovs-vsctl set bridge br-int protocols=OpenFlow13,OpenFlow15 ovs-vsctl add-port br-int ls1p2 -- set interface ls1p2 type=internal external_ids:iface-id=ls1p2 ip netns add ls1p2 ip link set ls1p2 netns ls1p2 ip netns exec ls1p2 ip link set ls1p2 address 00:00:00:01:01:02 ip netns exec ls1p2 ip link set ls1p2 up ip netns exec ls1p2 ip addr add 192.168.1.2/24 dev ls1p2 result on client: + ovs-vsctl set bridge br-int protocols=OpenFlow13,OpenFlow15 ovs-vsctl: no row "br-int" in table Bridge + ovs-vsctl add-port br-int ls1p2 -- set interface ls1p2 type=internal external_ids:iface-id=ls1p2 ovs-vsctl: no bridge named br-int + ip netns add ls1p2 + ip link set ls1p2 netns ls1p2 Cannot find device "ls1p2" + ip netns exec ls1p2 ip link set ls1p2 address 00:00:00:01:01:02 Cannot find device "ls1p2" + ip netns exec ls1p2 ip link set ls1p2 up Cannot find device "ls1p2" + ip netns exec ls1p2 ip addr add 192.168.1.2/24 dev ls1p2 Cannot find device "ls1p2" [root@wsfd-advnetlab17 bz1992705]# grep WARN /var/log/ovn/ovn-controller.log 2021-09-01T08:04:24.020Z|00004|ovsdb_idl|WARN|Open_vSwitch database lacks Datapath table (database needs upgrade?) 2021-09-01T08:04:24.020Z|00005|ovsdb_idl|WARN|Open_vSwitch table in Open_vSwitch database lacks datapaths column (database needs upgrade?) 2021-09-01T08:04:24.020Z|00006|ovsdb_idl|WARN|Open_vSwitch database lacks Datapath table (database needs upgrade?) 2021-09-01T08:04:24.020Z|00007|ovsdb_idl|WARN|Open_vSwitch table in Open_vSwitch database lacks datapaths column (database needs upgrade?) 2021-09-01T08:04:24.024Z|00013|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"bridges\",\"datapaths\"]"} 2021-09-01T08:04:24.024Z|00014|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"bridges\",\"datapaths\"]"} 2021-09-01T08:04:24.026Z|00015|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"bridges\",\"datapaths\"]"} 2021-09-01T08:04:24.027Z|00016|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"bridges\",\"datapaths\"]"} 2021-09-01T08:04:24.027Z|00017|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"bridges\",\"datapaths\"]"} 2021-09-01T08:04:24.030Z|00020|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: connection failed (No such file or directory) [root@wsfd-advnetlab17 bz1992705]# rpm -qa | grep -E "openvswitch2.11|ovn-2021" ovn-2021-21.06.0-17.el8fdp.x86_64 openvswitch2.11-2.11.3-86.el8fdp.x86_64 ovn-2021-central-21.06.0-17.el8fdp.x86_64 ovn-2021-host-21.06.0-17.el8fdp.x86_64 Verified on ovn-2021-21.06.0-24: [root@wsfd-advnetlab17 bz1992705]# bash -x client.sh + systemctl start openvswitch + systemctl start ovn-northd + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set open . external_ids:system-id=hv0 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.26 + systemctl restart ovn-controller + ovs-vsctl set bridge br-int protocols=OpenFlow13,OpenFlow15 + ovs-vsctl add-port br-int ls1p2 -- set interface ls1p2 type=internal external_ids:iface-id=ls1p2 + ip netns add ls1p2 + ip link set ls1p2 netns ls1p2 + ip netns exec ls1p2 ip link set ls1p2 address 00:00:00:01:01:02 + ip netns exec ls1p2 ip link set ls1p2 up + ip netns exec ls1p2 ip addr add 192.168.1.2/24 dev ls1p2 [root@wsfd-advnetlab17 bz1992705]# rpm -qa | grep -E "openvswitch2.11|ovn-2021" ovn-2021-central-21.06.0-24.el8fdp.x86_64 openvswitch2.11-2.11.3-86.el8fdp.x86_64 ovn-2021-21.06.0-24.el8fdp.x86_64 ovn-2021-host-21.06.0-24.el8fdp.x86_64 [root@wsfd-advnetlab16 bz1992705]# ovn-sbctl show Chassis hv1 hostname: wsfd-advnetlab16.anl.lab.eng.bos.redhat.com Encap geneve ip: "20.0.175.25" options: {csum="true"} Port_Binding ls1p1 Chassis hv0 hostname: wsfd-advnetlab17.anl.lab.eng.bos.redhat.com Encap geneve ip: "20.0.175.26" options: {csum="true"} Port_Binding ls1p2 [root@wsfd-advnetlab16 bz1992705]# ip netns exec ls1p1 ping 192.168.1.2 -c 1 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. 64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=2.21 ms --- 192.168.1.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 2.205/2.205/2.205/0.000 ms Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3450 *** Bug 2016343 has been marked as a duplicate of this bug. *** 1518 packets transmitted, 1518 received, 0% packet loss, time 1547210ms ovn-2021-21.09.0-5.el8fdp.x86_64 |