Hide Forgot
+++ This bug was initially created as a clone of Bug #1895220 +++ Description of problem: Version-Release number of selected component (if applicable): Update from RHOSP 16.1.0 to 16.1.2 How reproducible: During the update phase from RHOSP 16.1.0 to 16.1.2, when the controller nodes are updated, the Instance using the OVN Provider Network will experience network communication problems. Steps to Reproduce: 1. Update Director node from 16.1.0 to 16.1.2 2. Update controller nodes from 16.1.0 to 16.1.2 3. Network communication problem occurs in instances using OVN Provider Network Additional info: Director and controller nodes are virtual machines. Compute nodes are physical servers. Instance uses Provider Network. And it communicates directly with the external gateway. Other potential solution at this stage: 0. prevent paunch from updating the ovndb container during update-run on the controller. This should then happen automatically during converge: - pro: - the update workflow stays exactly the same from a customer point of view cons: - the ovndb container won't be updated until convergence. Maybe we can also add a post-update external task to reduce this time if that's a problem. - full canary testing is still strictly speaking lost, but at least it can be done even if it's against an old ovndb db 1. force ordering during the update: all computes node need to be done first (what I'm proposing as a current workaround) - pro: - doesn't have to maintain workaround; - cons: - force customer to update all their automation around rhosp; - rolling update still goes away - we lose the ability to update all roles in parallel. 2. remove the security group rule (with low priority) added by neutron ML2/OVN to drop all packets which are not matched by other rules. - pro: - solve this issue cons: - this will solve this one, but not future potential breakage - this relax security rule, which may not be an option. 3. convince ovn to follow some sort of semantic versioning, so that we don't have breaking db change within an EUS stream: - pro: nothing change for update within the same EUS stream; - cons: - we would still have to deal with update from one EUS stream to another; - it seems that this has already been ruled out by OVN. 4. dive deeper in the problem to find a more elegant solution if any. Currently, option 0 would be my personal first choice for investigation. But first thing first, we need to totally understand why CI FIP testing isn't currently affected by the issue. Thanks, ... ..
https://patchwork.ozlabs.org/project/ovn/patch/20201120105253.2160203-1-numans@ovn.org/
reproduced on ovn20.09.0-2 with following steps: install ovn2.13.0-37: [root@wsfd-advnetlab16 bz1899936]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-63.el7fdp.x86_64 ovn2.13-central-2.13.0-37.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch ovn2.13-2.13.0-37.el7fdp.x86_64 ovn2.13-host-2.13.0-37.el7fdp.x86_64 [root@wsfd-advnetlab16 bz1899936]# bash -x rep.sh + systemctl start openvswitch + systemctl start ovn-northd + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.150.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.150.25 + systemctl restart ovn-controller + ovn-nbctl ls-add sw0 + ovn-nbctl lsp-add sw0 sw0-port1 + ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3' + ovn-nbctl lsp-add sw0 sw0-port2 + ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4' + ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related + ovn-nbctl acl-add sw0 to-lport 1000 ip drop + ip netns add sw0p1 + ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1 + ip link set sw0p1 netns sw0p1 + ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03 + ip netns exec sw0p1 ip link set sw0p1 up + ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1 + ip netns add sw0p2 + ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2 + ip link set sw0p2 netns sw0p2 + ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04 + ip netns exec sw0p2 ip link set sw0p2 up + ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2 + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=2.51 ms 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1003 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.450 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.450/335.587/1003.793/472.493 ms, pipe 2 + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x8a31167b, duration=1.390s, table=47, n_packets=2, n_bytes=196, idle_age=1, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0x8a31167b, duration=1.388s, table=47, n_packets=0, n_bytes=0, idle_age=1, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=true + exit 0 [root@wsfd-advnetlab16 bz1899936]# rpm -Uvh ~/20.09.0-2/ovn2.13-host-20.09.0-2.el7fdp.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-2.el7fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el7fdp ################################# [100%] <=== upgrade only ovn-controller to 20.09.0-2 [root@wsfd-advnetlab16 bz1899936]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-63.el7fdp.x86_64 ovn2.13-central-2.13.0-37.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch ovn2.13-2.13.0-37.el7fdp.x86_64 ovn2.13-host-20.09.0-2.el7fdp.x86_64 [root@wsfd-advnetlab16 bz1899936]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. --- 10.0.0.4 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 1999ms <=== ping failed + ovs-ofctl dump-flows br-int table=47 + grep commit <=== no commit flow + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=false + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. --- 10.0.0.4 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2000ms + ovs-ofctl dump-flows br-int table=47 + grep commit Verified on ovn20.09.0-15: [root@wsfd-advnetlab16 bz1899936]# rpm -Uvh ~/20.09.0-15/ovn2.13-host-20.09.0-15.el7fdp.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-15.el7fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el7fdp ################################# [100%] [root@wsfd-advnetlab16 bz1899936]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-63.el7fdp.x86_64 ovn2.13-central-2.13.0-37.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch ovn2.13-2.13.0-37.el7fdp.x86_64 ovn2.13-host-20.09.0-15.el7fdp.x86_64 [root@wsfd-advnetlab16 bz1899936]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.49 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.712 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.080 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 0.080/0.761/1.491/0.577 ms <=== ping passed if ovn-match-northd-version=true + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x1e480b, duration=31.646s, table=47, n_packets=3, n_bytes=294, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0x1e480b, duration=31.645s, table=47, n_packets=0, n_bytes=0, idle_age=31, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) <=== commit flow + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=false + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. --- 10.0.0.4 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 1999ms <== ping failed if ovn-match-northd-version=false + ovs-ofctl dump-flows br-int table=47 + grep commit <=== no commit flow
Verified on rhel8 version: [root@wsfd-advnetlab17 bz1899936]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-72.el8fdp.x86_64 openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch ovn2.13-2.13.0-37.el8fdp.x86_64 ovn2.13-host-2.13.0-37.el8fdp.x86_64 ovn2.13-central-2.13.0-37.el8fdp.x86_64 [root@wsfd-advnetlab17 bz1899936]# bash -x rep.sh + systemctl start openvswitch + systemctl start ovn-northd + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.151.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.151.25 + systemctl restart ovn-controller + ovn-nbctl ls-add sw0 + ovn-nbctl lsp-add sw0 sw0-port1 + ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3' + ovn-nbctl lsp-add sw0 sw0-port2 + ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4' + ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related + ovn-nbctl acl-add sw0 to-lport 1000 ip drop + ip netns add sw0p1 + ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1 + ip link set sw0p1 netns sw0p1 + ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03 + ip netns exec sw0p1 ip link set sw0p1 up + ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1 + ip netns add sw0p2 + ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2 + ip link set sw0p2 netns sw0p2 + ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04 + ip netns exec sw0p2 ip link set sw0p2 up + ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2 + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=1.82 ms 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1053 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.586 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 55ms rtt min/avg/max/mdev = 0.586/351.879/1053.232/495.931 ms, pipe 2 + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x5609c6e1, duration=1.455s, table=47, n_packets=2, n_bytes=196, idle_age=1, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0x5609c6e1, duration=1.453s, table=47, n_packets=0, n_bytes=0, idle_age=1, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=true + exit 0 [root@wsfd-advnetlab17 bz1899936]# rpm -Uvh ~/20.09.0-15/ovn2.13-host-20.09.0-15.el8fdp.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-15.el8fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el8fdp ################################# [100%] [root@wsfd-advnetlab17 bz1899936]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.49 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.589 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.073 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 65ms rtt min/avg/max/mdev = 0.073/0.716/1.486/0.583 ms <=== ping passed when ovn-match-northd-version=true + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x5609c6e1, duration=39.105s, table=47, n_packets=3, n_bytes=294, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0x5609c6e1, duration=39.103s, table=47, n_packets=0, n_bytes=0, idle_age=39, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=false + systemctl start ovn-controller + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. --- 10.0.0.4 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 72ms + ovs-ofctl dump-flows br-int table=47 + grep commit [root@wsfd-advnetlab17 bz1899936]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-72.el8fdp.x86_64 ovn2.13-host-20.09.0-15.el8fdp.x86_64 openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch ovn2.13-2.13.0-37.el8fdp.x86_64 ovn2.13-central-2.13.0-37.el8fdp.x86_64
reproduced the dhcp issue described in comment 6: [root@wsfd-advnetlab17 test]# bash -x rep.sh + systemctl start openvswitch + systemctl start ovn-northd + ovn-nbctl set-connection ptcp:6641 + ovn-sbctl set-connection ptcp:6642 + ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.151.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.151.25 + systemctl restart ovn-controller + ovn-nbctl ls-add sw0 -- add Logical_Switch sw0 other_config subnet=10.0.0.0/24 + ovn-nbctl lsp-add sw0 sw0-port1 + ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3' + ovn-nbctl lsp-add sw0 sw0-port2 + ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4' + ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related + ovn-nbctl acl-add sw0 to-lport 1000 ip drop ++ ovn-nbctl create DHCP_Options cidr=10.0.0.0/24 'options="server_id"="10.0.0.254" "server_mac"="00:de:ad:ff:01:02" "router"="10.0.0.254" "lease_time"="3600"' + dhcp_102=23e6f8f1-4f9d-4694-b6ad-5907fc9f41f3 + ovn-nbctl lsp-set-dhcpv4-options sw0-port2 23e6f8f1-4f9d-4694-b6ad-5907fc9f41f3 + ip netns add sw0p1 + ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1 + ip link set sw0p1 netns sw0p1 + ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03 + ip netns exec sw0p1 ip link set sw0p1 up + ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1 + ip netns add sw0p2 + ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2 + ip link set sw0p2 netns sw0p2 + ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04 + ip netns exec sw0p2 ip link set sw0p2 up + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.3.6 Copyright 2004-2017 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 3 (xid=0xb055626d) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 4 (xid=0xb055626d) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xb055626d) DHCPOFFER from 10.0.0.254 DHCPACK from 10.0.0.254 (xid=0xb055626d) bound to 10.0.0.4 -- renewal in 1570 seconds. + ip netns exec sw0p2 pkill dhclient + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.83 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.706 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.099 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 34ms rtt min/avg/max/mdev = 0.099/0.878/1.829/0.716 ms + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x76bd6f30, duration=6.550s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),r esubmit(,48) cookie=0x76bd6f30, duration=6.547s, table=47, n_packets=0, n_bytes=0, idle_age=6, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])), resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=true + exit 0 [root@wsfd-advnetlab17 test]# rpm -Uvh ~/20.09.0-15/ovn2.13-host-20.09.0-15.el8fdp.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-15.el8fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el8fdp ################################# [100%] [root@wsfd-advnetlab17 test]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.3.6 Copyright 2004-2017 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xc946c00a) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xc946c00a) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xc946c00a) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 3 (xid=0xe92dfc4c) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 6 (xid=0xe92dfc4c) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 12 (xid=0xe92dfc4c) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 14 (xid=0xe92dfc4c) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 9 (xid=0xe92dfc4c) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 8 (xid=0xe92dfc4c) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 9 (xid=0xe92dfc4c) No DHCPOFFERS received. Trying recorded lease 10.0.0.4 PING 10.0.0.254 (10.0.0.254) from 10.0.0.4 sw0p2: 56(84) bytes of data. --- 10.0.0.254 ping statistics --- 10 packets transmitted, 0 received, 100% packet loss, time 263ms No working leases in persistent database - sleeping. <=== fail to get ip through dhcp, when ovn-match-northd-version=true as version mismatched + ip netns exec sw0p2 pkill dhclient + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.36 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.654 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.088 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 11ms rtt min/avg/max/mdev = 0.088/0.699/1.357/0.519 ms + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x76bd6f30, duration=413.662s, table=47, n_packets=2, n_bytes=196, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0]) ),resubmit(,48) cookie=0x76bd6f30, duration=413.659s, table=47, n_packets=0, n_bytes=0, idle_age=413, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0 ])),resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=false + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.3.6 Copyright 2004-2017 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x96911165) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x96911165) DHCPACK from 10.0.0.254 (xid=0x96911165) bound to 10.0.0.4 -- renewal in 1399 seconds. <=== dhcp works, when ovn-match-northd-version=false + ip netns exec sw0p2 pkill dhclient + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. --- 10.0.0.4 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 62ms + ovs-ofctl dump-flows br-int table=47 + grep commit [root@wsfd-advnetlab17 test]# rpm -qa | grep ovn2.13 ovn2.13-host-20.09.0-15.el8fdp.x86_64 ovn2.13-2.13.0-37.el8fdp.x86_64 ovn2.13-central-2.13.0-37.el8fdp.x86_64
reproduced on rhel7 version: [root@wsfd-advnetlab16 test]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.2.5 Copyright 2004-2013 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x28bf77e6) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x28bf77e6) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x28bf77e6) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 5 (xid=0x21f7ab44) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 9 (xid=0x21f7ab44) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 20 (xid=0x21f7ab44) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 13 (xid=0x21f7ab44) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 11 (xid=0x21f7ab44) DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 3 (xid=0x21f7ab44) No DHCPOFFERS received. Trying recorded lease 10.0.0.4 PING 10.0.0.254 (10.0.0.254) from 10.0.0.4 sw0p2: 56(84) bytes of data. --- 10.0.0.254 ping statistics --- 10 packets transmitted, 0 received, 100% packet loss, time 9003ms No working leases in persistent database - sleeping. <=== fail to get ip through dhcp [root@wsfd-advnetlab16 test]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-63.el7fdp.x86_64 ovn2.13-central-2.13.0-37.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch ovn2.13-2.13.0-37.el7fdp.x86_64 ovn2.13-host-20.09.0-15.el7fdp.x86_64
Verified on 20.09.0-17.el7: [root@wsfd-advnetlab16 20.09.0-17]# rpm -Uvh ovn2.13-host-20.09.0-17.el7fdp.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-17.el7fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el7fdp ################################# [100%] [root@wsfd-advnetlab16 20.09.0-17]# cd [root@wsfd-advnetlab16 ~]# cd test/ [root@wsfd-advnetlab16 test]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.2.5 Copyright 2004-2013 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x54b6300e) DHCPACK from 10.0.0.254 (xid=0x54b6300e) bound to 10.0.0.4 -- renewal in 1749 seconds. <=== dhcp works + ip netns exec sw0p2 pkill dhclient + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.22 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.552 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.088 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.088/0.621/1.224/0.466 ms + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0xbf1ca58b, duration=80.249s, table=47, n_packets=2, n_bytes=196, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])) ,resubmit(,48) cookie=0xbf1ca58b, duration=80.248s, table=47, n_packets=0, n_bytes=0, idle_age=80, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0]) ),resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=false + systemctl start ovn-controller + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.2.5 Copyright 2004-2013 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x4f67c945) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x4f67c945) DHCPACK from 10.0.0.254 (xid=0x4f67c945) bound to 10.0.0.4 -- renewal in 1646 seconds. + ip netns exec sw0p2 pkill dhclient + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=0.111 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.075 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.071 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.071/0.085/0.111/0.020 ms + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0xbf1ca58b, duration=9.051s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0xbf1ca58b, duration=9.051s, table=47, n_packets=0, n_bytes=0, idle_age=9, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) [root@wsfd-advnetlab16 test]# rpm -qa | grep -E "openvswitch|ovn" openvswitch2.13-2.13.0-63.el7fdp.x86_64 ovn2.13-central-2.13.0-37.el7fdp.x86_64 openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch ovn2.13-2.13.0-37.el7fdp.x86_64 ovn2.13-host-20.09.0-17.el7fdp.x86_64
also verified on rhel8 version: [root@wsfd-advnetlab17 test]# rpm -Uvh ~/20.09.0-17/ovn2.13-host-20.09.0-17.el8fdp.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ovn2.13-host-20.09.0-17.el8fdp ################################# [ 50%] Cleaning up / removing... 2:ovn2.13-host-2.13.0-37.el8fdp ################################# [100%] [root@wsfd-advnetlab17 test]# bash -x rep_step2.sh + systemctl start ovn-controller + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.3.6 Copyright 2004-2017 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x1da97f75) DHCPACK from 10.0.0.254 (xid=0x1da97f75) bound to 10.0.0.4 -- renewal in 1492 seconds. <=== dhcp works + ip netns exec sw0p2 pkill dhclient + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.41 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.615 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.080 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 23ms rtt min/avg/max/mdev = 0.080/0.702/1.413/0.548 ms + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x728ad313, duration=31.903s, table=47, n_packets=2, n_bytes=196, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])) ,resubmit(,48) cookie=0x728ad313, duration=31.901s, table=47, n_packets=0, n_bytes=0, idle_age=31, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0]) ),resubmit(,48) + ovn-appctl -t ovn-controller exit --restart + ovs-vsctl set open . external_ids:ovn-match-northd-version=false + systemctl start ovn-controller + ip netns exec sw0p2 dhclient -v sw0p2 Internet Systems Consortium DHCP Client 4.3.6 Copyright 2004-2017 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/sw0p2/50:54:00:00:00:04 Sending on LPF/sw0p2/50:54:00:00:00:04 Sending on Socket/fallback DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xe18e7b16) DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xe18e7b16) DHCPACK from 10.0.0.254 (xid=0xe18e7b16) bound to 10.0.0.4 -- renewal in 1586 seconds. + ip netns exec sw0p2 pkill dhclient + ip netns exec sw0p1 ping 10.0.0.4 -c 3 PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data. 64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=0.669 ms 64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.073 ms 64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.062 ms --- 10.0.0.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 76ms rtt min/avg/max/mdev = 0.062/0.268/0.669/0.283 ms + ovs-ofctl dump-flows br-int table=47 + grep commit cookie=0x728ad313, duration=9.585s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) cookie=0x728ad313, duration=9.585s, table=47, n_packets=0, n_bytes=0, idle_age=9, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) [root@wsfd-advnetlab17 test]# rpm -qa | grep ovn2.13 ovn2.13-host-20.09.0-17.el8fdp.x86_64 ovn2.13-2.13.0-37.el8fdp.x86_64 ovn2.13-central-2.13.0-37.el8fdp.x86_64
set VERIFIED per above comments
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5308