Updating ovn-controller from ovn2.13-host-2.13.0-37.el8fdp.x86_64 to ovn2.13-20.09 breaks datapath connectivity. When only ovn-controller is updated, it rejects all logical flows with ct_commit action and a warning is seen in the ovn-controller log. Because of which datapath disruption is seen if ACLs with allow-related are configured on logical switches/port groups.
Steps to reproduce. Install OVN version ovn2.13-20.06.2-15 # Create OVN resources. ovn-nbctl ls-add sw0 ovn-nbctl lsp-add sw0 sw0-port1 ovn-nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3" ovn-nbctl lsp-add sw0 sw0-port2 ovn-nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 10.0.0.4" ovn-nbctl acl-add sw0 to-lport 1002 "ip" allow-related ovn-nbctl acl-add sw0 to-lport 1000 "ip" drop # Create namespace for sw0-port1 and sw0-port2 # Ping from sw0-port1 to sw0-port2. It should work fine. ovs-ofctl dump-flows br-int table=47 | grep commit # Stop ovn-controller. Note --restart is important. Otherwise ovn-controller will unregister # and delete all flows. ovn-appctl -t ovn-controller exit --restart # Update only ovn-controller to ovn2.13-20.09.1-15 # Start ovn-controller. # ping from sw0-port1 to sw0-port2 should work. # There should be commit OF flows with ct action. ovs-ofctl dump-flows br-int table=47 | grep commit
Verified on ovn2.13-host-20.09.0-17.el8fdp.x86_64: [root@wsfd-advnetlab17 bz1900484]# rpm -ivh ~/2.13.0-37/ovn2.13-* Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-2.13.0-37.el8fdp ################################# [ 33%] Unit ovn-northd.service could not be found. 2:ovn2.13-central-2.13.0-37.el8fdp ################################# [ 67%] Unit ovn-controller.service could not be found. 3:ovn2.13-host-2.13.0-37.el8fdp ################################# [100%] [root@wsfd-advnetlab17 bz1900484]# bash -x rep.sh + systemctl start openvswitch + systemctl start ovn-northd + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.151.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.151.25 + systemctl restart ovn-controller + ovn-nbctl ls-add sw0 + ovn-nbctl lsp-add sw0 sw0-port1 + ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3' + ovn-nbctl lsp-add sw0 sw0-port2 + ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4' + ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related + ovn-nbctl acl-add sw0 to-lport 1000 ip drop + ip netns add sw0p1 + ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1 + ip link set sw0p1 netns sw0p1 + ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03 + ip netns exec sw0p1 ip link set sw0p1 up + ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1 + ip netns add sw0p2 + ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2 + ip link set sw0p2 netns sw0p2 + ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04 + ip netns exec sw0p2 ip link set sw0p2 up + ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2 + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1019 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=1.79 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.597 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 22ms rtt min/avg/max/mdev = 0.597/340.620/1019.469/480.018 ms, pipe 2 + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x75a74162, duration=1.440s, table=47, n_packets=2, n_bytes=196, idle_age=1, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])), resubmit(,48) cookie=0x75a74162, duration=1.439s, table=47, n_packets=0, n_bytes=0, idle_age=1, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])), resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + exit 0 [root@wsfd-advnetlab17 bz1900484]# rpm -Uvh ~/20.09.0-17/ovn2.13-host-20.09.0-17.el8fdp.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-17.el8fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el8fdp ################################# [100%] [root@wsfd-advnetlab17 bz1900484]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.706 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.262 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 2 received, 33.3333% packet loss, time 85ms rtt min/avg/max/mdev = 0.262/0.484/0.706/0.222 ms <=== ping works + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x75a74162, duration=2.102s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0x75a74162, duration=2.102s, table=47, n_packets=0, n_bytes=0, idle_age=2, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) [root@wsfd-advnetlab17 bz1900484]# rpm -qa | grep ovn2.13 ovn2.13-host-20.09.0-17.el8fdp.x86_64 ovn2.13-2.13.0-37.el8fdp.x86_64 ovn2.13-central-2.13.0-37.el8fdp.x86_64
Verified on rhel8 version: [root@wsfd-advnetlab16 bz1900484]# rpm -Uvh ~/20.09.0-17/ovn2.13-host-20.09.0-17.el7fdp.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-17.el7fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el7fdp ################################# [100%] [root@wsfd-advnetlab16 bz1900484]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=1.17 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 1 received, 66% packet loss, time 1999ms rtt min/avg/max/mdev = 1.175/1.175/1.175/0.000 ms <=== ping passed + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x714b946, duration=11.012s, table=47, n_packets=1, n_bytes=98, idle_age=10, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0x714b946, duration=11.012s, table=47, n_packets=0, n_bytes=0, idle_age=11, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) [root@wsfd-advnetlab16 bz1900484]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-63.el7fdp.x86_64 ovn2.13-central-2.13.0-37.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch ovn2.13-2.13.0-37.el7fdp.x86_64 ovn2.13-host-20.09.0-17.el7fdp.x86_64
set VERIFIED per comment 2 and comment 3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5308