Bug 1899936

Summary: After updating the controller nodes to RHOSP 16.1.2, there was a problem with VM network communication.
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Numan Siddique <nusiddiq>
Component: ovn2.13Assignee: Numan Siddique <nusiddiq>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: high Docs Contact:
Priority: high    
Version: FDP 20.ECC: akaris, amuller, astupnik, atragler, bdobreli, bhaley, bshephar, chrisw, ctrautma, ealcaniz, ekuris, jishi, jlibosva, jpretori, kfida, knoha, mburns, nusiddiq, ralongi, sathlang, scohen, sgolovat, smooney, sputhenp, tkajinam, youngkim
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1895220 Environment:
Last Closed: 2020-12-01 15:07:14 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1895220, 1900484    

Description Numan Siddique 2020-11-20 12:14:40 UTC 
+++ This bug was initially created as a clone of Bug #1895220 +++

Description of problem:


Version-Release number of selected component (if applicable):
Update from RHOSP 16.1.0 to 16.1.2


How reproducible:
During the update phase from RHOSP 16.1.0 to 16.1.2, when the controller nodes are updated, the Instance using the OVN Provider Network will experience network communication problems.


Steps to Reproduce:
1. Update Director node from 16.1.0 to 16.1.2
2. Update controller nodes from 16.1.0 to 16.1.2
3. Network communication problem occurs in instances using OVN Provider Network


Additional info:
Director and controller nodes are virtual machines.
Compute nodes are physical servers.
Instance uses Provider Network. And it communicates directly with the external gateway.


Other potential solution at this stage:

 0. prevent paunch from updating the ovndb container during update-run on the controller.  This should then happen automatically during converge:
    - pro: - the update workflow stays exactly the same from a customer point of view
      cons: - the ovndb container won't be updated until convergence.  Maybe we can also add a post-update external task to reduce this
              time if that's a problem.
            - full canary testing is still strictly speaking lost, but at least it can be done even if it's against an old ovndb db
 1. force ordering during the update: all computes node need to be done first (what I'm proposing as a current workaround)
    - pro:  - doesn't have to maintain workaround;
    - cons: - force customer to update all their automation around rhosp;
            - rolling update still goes away
            - we lose the ability to update all roles in parallel.
 2. remove the security group rule (with low priority) added by neutron ML2/OVN to drop all packets which are not matched by other rules.
    - pro: - solve this issue
      cons: - this will solve this one, but not future potential breakage
            - this relax security rule, which may not be an option.

 3. convince ovn to follow some sort of semantic versioning, so that we don't have breaking
    db change within an EUS stream:
    - pro: nothing change for update within the same EUS stream;
    - cons: - we would still have to deal with update from one EUS stream to another;
            - it seems that this has already been ruled out by OVN.

 4. dive deeper in the problem to find a more elegant solution if any.

Currently, option 0 would be my personal first choice for investigation.

But first thing first, we need to totally understand why CI FIP testing isn't currently affected by the issue.

Thanks,

...
..
Comment 1 Numan Siddique 2020-11-20 12:16:24 UTC 
https://patchwork.ozlabs.org/project/ovn/patch/20201120105253.2160203-1-numans@ovn.org/
Comment 4 Jianlin Shi 2020-11-23 02:39:35 UTC 
reproduced on ovn20.09.0-2 with following steps:

install ovn2.13.0-37:

[root@wsfd-advnetlab16 bz1899936]# rpm -qa | grep -E "openvswitch|ovn"                                
openvswitch2.13-2.13.0-63.el7fdp.x86_64                                                               
ovn2.13-central-2.13.0-37.el7fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch                                                 
ovn2.13-2.13.0-37.el7fdp.x86_64                                                                       
ovn2.13-host-2.13.0-37.el7fdp.x86_64

[root@wsfd-advnetlab16 bz1899936]# bash -x rep.sh                                                     
+ systemctl start openvswitch                                                                         
+ systemctl start ovn-northd                                                                          
+ ovn-nbctl set-connection ptcp:6641                                                                  
+ ovn-sbctl set-connection ptcp:6642                                                                  
+ ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.150.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.150.25
+ systemctl restart ovn-controller                                                                    
+ ovn-nbctl ls-add sw0                                                                                
+ ovn-nbctl lsp-add sw0 sw0-port1                                                                     
+ ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3'                                  
+ ovn-nbctl lsp-add sw0 sw0-port2                                                                     
+ ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4'                                  
+ ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related                                                
+ ovn-nbctl acl-add sw0 to-lport 1000 ip drop                                                         
+ ip netns add sw0p1                                                                                  
+ ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1
+ ip link set sw0p1 netns sw0p1                                                                       
+ ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03                                     
+ ip netns exec sw0p1 ip link set sw0p1 up                                                            
+ ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1                                               
+ ip netns add sw0p2                                                                                  
+ ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2
+ ip link set sw0p2 netns sw0p2                                                                       
+ ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04                                     
+ ip netns exec sw0p2 ip link set sw0p2 up                                                            
+ ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2                                               
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3                                                              
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.                                                        
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=2.51 ms                                                
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1003 ms                                                
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.450 ms                                               
                                                                                                      
--- 10.0.0.4 ping statistics ---                                                                      
3 packets transmitted, 3 received, 0% packet loss, time 2004ms                                        
rtt min/avg/max/mdev = 0.450/335.587/1003.793/472.493 ms, pipe 2                                      
+ ovs-ofctl dump-flows br-int table=47                                                                
+ grep commit                                                                                         
 cookie=0x8a31167b, duration=1.390s, table=47, n_packets=2, n_bytes=196, idle_age=1, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)                                                                                         
 cookie=0x8a31167b, duration=1.388s, table=47, n_packets=0, n_bytes=0, idle_age=1, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)                                                                                         
+ ovn-appctl -t ovn-controller exit --restart                                                         
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=true                                     
+ exit 0

[root@wsfd-advnetlab16 bz1899936]# rpm -Uvh ~/20.09.0-2/ovn2.13-host-20.09.0-2.el7fdp.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:ovn2.13-host-20.09.0-2.el7fdp    ################################# [ 50%]
Cleaning up / removing...
   2:ovn2.13-host-2.13.0-37.el7fdp    ################################# [100%]

<=== upgrade only ovn-controller to 20.09.0-2

[root@wsfd-advnetlab16 bz1899936]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-63.el7fdp.x86_64
ovn2.13-central-2.13.0-37.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch
ovn2.13-2.13.0-37.el7fdp.x86_64
ovn2.13-host-20.09.0-2.el7fdp.x86_64
[root@wsfd-advnetlab16 bz1899936]# bash -x rep_step2.sh 
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

<=== ping failed

+ ovs-ofctl dump-flows br-int table=47
+ grep commit

<=== no commit flow

+ ovn-appctl -t ovn-controller exit --restart
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=false
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms

+ ovs-ofctl dump-flows br-int table=47
+ grep commit

Verified on ovn20.09.0-15:

[root@wsfd-advnetlab16 bz1899936]# rpm -Uvh ~/20.09.0-15/ovn2.13-host-20.09.0-15.el7fdp.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:ovn2.13-host-20.09.0-15.el7fdp   ################################# [ 50%]
Cleaning up / removing...
   2:ovn2.13-host-2.13.0-37.el7fdp    ################################# [100%]
[root@wsfd-advnetlab16 bz1899936]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-63.el7fdp.x86_64
ovn2.13-central-2.13.0-37.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch
ovn2.13-2.13.0-37.el7fdp.x86_64
ovn2.13-host-20.09.0-15.el7fdp.x86_64
[root@wsfd-advnetlab16 bz1899936]# bash -x rep_step2.sh 
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.49 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.712 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.080 ms

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 0.080/0.761/1.491/0.577 ms

<=== ping passed if ovn-match-northd-version=true

+ ovs-ofctl dump-flows br-int table=47
+ grep commit
 cookie=0x1e480b, duration=31.646s, table=47, n_packets=3, n_bytes=294, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)
 cookie=0x1e480b, duration=31.645s, table=47, n_packets=0, n_bytes=0, idle_age=31, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)

<=== commit flow

+ ovn-appctl -t ovn-controller exit --restart
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=false
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

<== ping failed if ovn-match-northd-version=false

+ ovs-ofctl dump-flows br-int table=47
+ grep commit

<=== no commit flow
Comment 5 Jianlin Shi 2020-11-23 02:47:28 UTC 
Verified on rhel8 version:

[root@wsfd-advnetlab17 bz1899936]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-72.el8fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
ovn2.13-2.13.0-37.el8fdp.x86_64
ovn2.13-host-2.13.0-37.el8fdp.x86_64                                                                  
ovn2.13-central-2.13.0-37.el8fdp.x86_64
[root@wsfd-advnetlab17 bz1899936]# bash -x rep.sh                                                     
+ systemctl start openvswitch
+ systemctl start ovn-northd
+ ovn-nbctl set-connection ptcp:6641
+ ovn-sbctl set-connection ptcp:6642
+ ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.151.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.151.25
+ systemctl restart ovn-controller
+ ovn-nbctl ls-add sw0
+ ovn-nbctl lsp-add sw0 sw0-port1
+ ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3'
+ ovn-nbctl lsp-add sw0 sw0-port2                                                                     
+ ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4'                                  
+ ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related                                                
+ ovn-nbctl acl-add sw0 to-lport 1000 ip drop
+ ip netns add sw0p1
+ ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1
+ ip link set sw0p1 netns sw0p1                                                                       
+ ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03                                     
+ ip netns exec sw0p1 ip link set sw0p1 up
+ ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1                                               
+ ip netns add sw0p2
+ ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2
+ ip link set sw0p2 netns sw0p2
+ ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04
+ ip netns exec sw0p2 ip link set sw0p2 up                                                            
+ ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.                                                        
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=1.82 ms                                                
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1053 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.586 ms                                               

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 55ms                                          
rtt min/avg/max/mdev = 0.586/351.879/1053.232/495.931 ms, pipe 2                                      
+ ovs-ofctl dump-flows br-int table=47
+ grep commit                                                                                         
 cookie=0x5609c6e1, duration=1.455s, table=47, n_packets=2, n_bytes=196, idle_age=1, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)                                                                                         
 cookie=0x5609c6e1, duration=1.453s, table=47, n_packets=0, n_bytes=0, idle_age=1, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)                                                                                         
+ ovn-appctl -t ovn-controller exit --restart
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=true                                     
+ exit 0

[root@wsfd-advnetlab17 bz1899936]# rpm -Uvh ~/20.09.0-15/ovn2.13-host-20.09.0-15.el8fdp.x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:ovn2.13-host-20.09.0-15.el8fdp   ################################# [ 50%]
Cleaning up / removing...
   2:ovn2.13-host-2.13.0-37.el8fdp    ################################# [100%]
[root@wsfd-advnetlab17 bz1899936]# bash -x rep_step2.sh 
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.49 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.589 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.073 ms

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 65ms
rtt min/avg/max/mdev = 0.073/0.716/1.486/0.583 ms

<=== ping passed when ovn-match-northd-version=true

+ ovs-ofctl dump-flows br-int table=47
+ grep commit
 cookie=0x5609c6e1, duration=39.105s, table=47, n_packets=3, n_bytes=294, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)
 cookie=0x5609c6e1, duration=39.103s, table=47, n_packets=0, n_bytes=0, idle_age=39, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)
+ ovn-appctl -t ovn-controller exit --restart
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=false
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 72ms

+ ovs-ofctl dump-flows br-int table=47
+ grep commit
[root@wsfd-advnetlab17 bz1899936]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch2.13-2.13.0-72.el8fdp.x86_64
ovn2.13-host-20.09.0-15.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch
ovn2.13-2.13.0-37.el8fdp.x86_64
ovn2.13-central-2.13.0-37.el8fdp.x86_64
Comment 8 Jianlin Shi 2020-11-23 07:23:01 UTC 
reproduced the dhcp issue described in comment 6:

[root@wsfd-advnetlab17 test]# bash -x rep.sh                     
+ systemctl start openvswitch                                                                                                                                                                               
+ systemctl start ovn-northd                                    
+ ovn-nbctl set-connection ptcp:6641                                                                                                                                                                        
+ ovn-sbctl set-connection ptcp:6642                                        
+ ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.151.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.151.25
+ systemctl restart ovn-controller                                           
+ ovn-nbctl ls-add sw0 -- add Logical_Switch sw0 other_config subnet=10.0.0.0/24
+ ovn-nbctl lsp-add sw0 sw0-port1                                           
+ ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3'        
+ ovn-nbctl lsp-add sw0 sw0-port2                                           
+ ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4'
+ ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related   
+ ovn-nbctl acl-add sw0 to-lport 1000 ip drop                                                
++ ovn-nbctl create DHCP_Options cidr=10.0.0.0/24 'options="server_id"="10.0.0.254" "server_mac"="00:de:ad:ff:01:02"         "router"="10.0.0.254" "lease_time"="3600"'
+ dhcp_102=23e6f8f1-4f9d-4694-b6ad-5907fc9f41f3                               
+ ovn-nbctl lsp-set-dhcpv4-options sw0-port2 23e6f8f1-4f9d-4694-b6ad-5907fc9f41f3
+ ip netns add sw0p1                                                          
+ ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1
+ ip link set sw0p1 netns sw0p1                                               
+ ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03
+ ip netns exec sw0p1 ip link set sw0p1 up                  
+ ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1 
+ ip netns add sw0p2                                      
+ ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2
+ ip link set sw0p2 netns sw0p2                                                                                                                                                                             
+ ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04                                                                     
+ ip netns exec sw0p2 ip link set sw0p2 up                                                                                                                                                                  
+ ip netns exec sw0p2 dhclient -v sw0p2          
Internet Systems Consortium DHCP Client 4.3.6
Copyright 2004-2017 Internet Systems Consortium.                 
All rights reserved.                                                                                                                                                                                        
For info, please visit https://www.isc.org/software/dhcp/       
                                                                                                                                                                                                            
Listening on LPF/sw0p2/50:54:00:00:00:04                                    
Sending on   LPF/sw0p2/50:54:00:00:00:04 
Sending on   Socket/fallback
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 3 (xid=0xb055626d)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 4 (xid=0xb055626d)
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xb055626d)
DHCPOFFER from 10.0.0.254
DHCPACK from 10.0.0.254 (xid=0xb055626d)
bound to 10.0.0.4 -- renewal in 1570 seconds.
+ ip netns exec sw0p2 pkill dhclient 
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.83 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.706 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.099 ms

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 34ms
rtt min/avg/max/mdev = 0.099/0.878/1.829/0.716 ms

+ ovs-ofctl dump-flows br-int table=47
+ grep commit
 cookie=0x76bd6f30, duration=6.550s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),r
esubmit(,48)
 cookie=0x76bd6f30, duration=6.547s, table=47, n_packets=0, n_bytes=0, idle_age=6, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),
resubmit(,48)
+ ovn-appctl -t ovn-controller exit --restart
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=true
+ exit 0

[root@wsfd-advnetlab17 test]# rpm -Uvh ~/20.09.0-15/ovn2.13-host-20.09.0-15.el8fdp.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...              
   1:ovn2.13-host-20.09.0-15.el8fdp   ################################# [ 50%]
Cleaning up / removing...
   2:ovn2.13-host-2.13.0-37.el8fdp    ################################# [100%]

[root@wsfd-advnetlab17 test]# bash -x rep_step2.sh
+ systemctl start ovn-controller
+ ip netns exec sw0p2 dhclient -v sw0p2
Internet Systems Consortium DHCP Client 4.3.6
Copyright 2004-2017 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/sw0p2/50:54:00:00:00:04
Sending on   LPF/sw0p2/50:54:00:00:00:04
Sending on   Socket/fallback
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xc946c00a)
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xc946c00a)
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xc946c00a)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 3 (xid=0xe92dfc4c)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 6 (xid=0xe92dfc4c)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 12 (xid=0xe92dfc4c)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 14 (xid=0xe92dfc4c)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 9 (xid=0xe92dfc4c)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 8 (xid=0xe92dfc4c)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 9 (xid=0xe92dfc4c)
No DHCPOFFERS received.
Trying recorded lease 10.0.0.4
PING 10.0.0.254 (10.0.0.254) from 10.0.0.4 sw0p2: 56(84) bytes of data.

--- 10.0.0.254 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 263ms

No working leases in persistent database - sleeping.


<=== fail to get ip through dhcp, when ovn-match-northd-version=true as version mismatched

+ ip netns exec sw0p2 pkill dhclient
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.36 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.654 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.088 ms

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 11ms
rtt min/avg/max/mdev = 0.088/0.699/1.357/0.519 ms
+ ovs-ofctl dump-flows br-int table=47
+ grep commit
 cookie=0x76bd6f30, duration=413.662s, table=47, n_packets=2, n_bytes=196, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])
),resubmit(,48)
 cookie=0x76bd6f30, duration=413.659s, table=47, n_packets=0, n_bytes=0, idle_age=413, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0
])),resubmit(,48)
+ ovn-appctl -t ovn-controller exit --restart
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=false

+ ip netns exec sw0p2 dhclient -v sw0p2
Internet Systems Consortium DHCP Client 4.3.6
Copyright 2004-2017 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/sw0p2/50:54:00:00:00:04
Sending on   LPF/sw0p2/50:54:00:00:00:04
Sending on   Socket/fallback
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x96911165)
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x96911165)
DHCPACK from 10.0.0.254 (xid=0x96911165)
bound to 10.0.0.4 -- renewal in 1399 seconds.

<=== dhcp works, when ovn-match-northd-version=false

+ ip netns exec sw0p2 pkill dhclient
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 62ms

+ ovs-ofctl dump-flows br-int table=47
+ grep commit

[root@wsfd-advnetlab17 test]# rpm -qa | grep ovn2.13                                                                                                                                                       
ovn2.13-host-20.09.0-15.el8fdp.x86_64
ovn2.13-2.13.0-37.el8fdp.x86_64
ovn2.13-central-2.13.0-37.el8fdp.x86_64
Comment 9 Jianlin Shi 2020-11-23 07:31:12 UTC 
reproduced on rhel7 version:

[root@wsfd-advnetlab16 test]# bash -x rep_step2.sh
+ systemctl start ovn-controller                                
+ ip netns exec sw0p2 dhclient -v sw0p2                         
Internet Systems Consortium DHCP Client 4.2.5                   
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.                                             
For info, please visit https://www.isc.org/software/dhcp/                                                                                                                                                   
                                                                                                                                                                                                            
Listening on LPF/sw0p2/50:54:00:00:00:04                                                                                                                                                                    
Sending on   LPF/sw0p2/50:54:00:00:00:04                                                                                                                                                                    
Sending on   Socket/fallback                 
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x28bf77e6)  
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x28bf77e6)  
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x28bf77e6)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 5 (xid=0x21f7ab44)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 9 (xid=0x21f7ab44)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 20 (xid=0x21f7ab44)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 13 (xid=0x21f7ab44)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 11 (xid=0x21f7ab44)
DHCPDISCOVER on sw0p2 to 255.255.255.255 port 67 interval 3 (xid=0x21f7ab44)
No DHCPOFFERS received.                  
Trying recorded lease 10.0.0.4           
PING 10.0.0.254 (10.0.0.254) from 10.0.0.4 sw0p2: 56(84) bytes of data.
                                                                
--- 10.0.0.254 ping statistics ---                              
10 packets transmitted, 0 received, 100% packet loss, time 9003ms
                                             
No working leases in persistent database - sleeping.

<=== fail to get ip through dhcp

[root@wsfd-advnetlab16 test]# rpm -qa | grep -E "openvswitch|ovn"                                                                                                                                          
openvswitch2.13-2.13.0-63.el7fdp.x86_64
ovn2.13-central-2.13.0-37.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch
ovn2.13-2.13.0-37.el7fdp.x86_64
ovn2.13-host-20.09.0-15.el7fdp.x86_64
Comment 10 Jianlin Shi 2020-11-23 07:36:21 UTC 
Verified on 20.09.0-17.el7:

[root@wsfd-advnetlab16 20.09.0-17]# rpm -Uvh ovn2.13-host-20.09.0-17.el7fdp.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...              
   1:ovn2.13-host-20.09.0-17.el7fdp   ################################# [ 50%]
Cleaning up / removing...                                                                                                                                                                                   
   2:ovn2.13-host-2.13.0-37.el7fdp    ################################# [100%]
[root@wsfd-advnetlab16 20.09.0-17]# cd                                                                                                                                                                      
[root@wsfd-advnetlab16 ~]# cd test/
[root@wsfd-advnetlab16 test]# bash -x rep_step2.sh
+ systemctl start ovn-controller                                  
+ ip netns exec sw0p2 dhclient -v sw0p2
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.                            
For info, please visit https://www.isc.org/software/dhcp/
                                                         
Listening on LPF/sw0p2/50:54:00:00:00:04
Sending on   LPF/sw0p2/50:54:00:00:00:04
Sending on   Socket/fallback            
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x54b6300e)
DHCPACK from 10.0.0.254 (xid=0x54b6300e)                        
bound to 10.0.0.4 -- renewal in 1749 seconds.    

<=== dhcp works

               
+ ip netns exec sw0p2 pkill dhclient    
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3     
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.22 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.552 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.088 ms
                                                       
--- 10.0.0.4 ping statistics ---                       
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.088/0.621/1.224/0.466 ms
+ ovs-ofctl dump-flows br-int table=47                        
+ grep commit                                    
 cookie=0xbf1ca58b, duration=80.249s, table=47, n_packets=2, n_bytes=196, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0]))
,resubmit(,48)
 cookie=0xbf1ca58b, duration=80.248s, table=47, n_packets=0, n_bytes=0, idle_age=80, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])
),resubmit(,48)
+ ovn-appctl -t ovn-controller exit --restart                                                                                                                                                               
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=false

+ systemctl start ovn-controller
+ ip netns exec sw0p2 dhclient -v sw0p2
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/sw0p2/50:54:00:00:00:04
Sending on   LPF/sw0p2/50:54:00:00:00:04
Sending on   Socket/fallback
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x4f67c945)
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x4f67c945)
DHCPACK from 10.0.0.254 (xid=0x4f67c945)
bound to 10.0.0.4 -- renewal in 1646 seconds.
+ ip netns exec sw0p2 pkill dhclient
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=0.111 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.075 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.071 ms

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.071/0.085/0.111/0.020 ms
+ ovs-ofctl dump-flows br-int table=47
+ grep commit
 cookie=0xbf1ca58b, duration=9.051s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48) 
 cookie=0xbf1ca58b, duration=9.051s, table=47, n_packets=0, n_bytes=0, idle_age=9, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)

[root@wsfd-advnetlab16 test]# rpm -qa | grep -E "openvswitch|ovn"                                                                                                                                          
openvswitch2.13-2.13.0-63.el7fdp.x86_64 
ovn2.13-central-2.13.0-37.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch
ovn2.13-2.13.0-37.el7fdp.x86_64
ovn2.13-host-20.09.0-17.el7fdp.x86_64
Comment 11 Jianlin Shi 2020-11-23 07:39:50 UTC 
also verified on rhel8 version:

[root@wsfd-advnetlab17 test]# rpm -Uvh ~/20.09.0-17/ovn2.13-host-20.09.0-17.el8fdp.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...                                    
   1:ovn2.13-host-20.09.0-17.el8fdp   ################################# [ 50%]
Cleaning up / removing...             
   2:ovn2.13-host-2.13.0-37.el8fdp    ################################# [100%]

[root@wsfd-advnetlab17 test]# bash -x rep_step2.sh
+ systemctl start ovn-controller                                  
+ ip netns exec sw0p2 dhclient -v sw0p2                  
Internet Systems Consortium DHCP Client 4.3.6            
Copyright 2004-2017 Internet Systems Consortium.
All rights reserved.                            
For info, please visit https://www.isc.org/software/dhcp/
                                                                
Listening on LPF/sw0p2/50:54:00:00:00:04                        
Sending on   LPF/sw0p2/50:54:00:00:00:04                        
Sending on   Socket/fallback            
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0x1da97f75)
DHCPACK from 10.0.0.254 (xid=0x1da97f75)                        
bound to 10.0.0.4 -- renewal in 1492 seconds. 

<=== dhcp works

                  
+ ip netns exec sw0p2 pkill dhclient                   
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3               
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.         
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1.41 ms 
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.615 ms     
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.080 ms 
                                                            
--- 10.0.0.4 ping statistics ---                       
3 packets transmitted, 3 received, 0% packet loss, time 23ms                                                                                                                                                
rtt min/avg/max/mdev = 0.080/0.702/1.413/0.548 ms
+ ovs-ofctl dump-flows br-int table=47                                                                                                                                                                      
+ grep commit                                    
 cookie=0x728ad313, duration=31.903s, table=47, n_packets=2, n_bytes=196, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0]))
,resubmit(,48)                                                    
 cookie=0x728ad313, duration=31.901s, table=47, n_packets=0, n_bytes=0, idle_age=31, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])
),resubmit(,48)                        
+ ovn-appctl -t ovn-controller exit --restart                                                                                                                                                               
+ ovs-vsctl set open . external_ids:ovn-match-northd-version=false
+ systemctl start ovn-controller                    
+ ip netns exec sw0p2 dhclient -v sw0p2                  
Internet Systems Consortium DHCP Client 4.3.6
Copyright 2004-2017 Internet Systems Consortium.
All rights reserved.                    
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/sw0p2/50:54:00:00:00:04
Sending on   LPF/sw0p2/50:54:00:00:00:04
Sending on   Socket/fallback
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xe18e7b16)
DHCPREQUEST on sw0p2 to 255.255.255.255 port 67 (xid=0xe18e7b16)
DHCPACK from 10.0.0.254 (xid=0xe18e7b16)
bound to 10.0.0.4 -- renewal in 1586 seconds.
+ ip netns exec sw0p2 pkill dhclient 
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.073 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.062 ms

--- 10.0.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 76ms
rtt min/avg/max/mdev = 0.062/0.268/0.669/0.283 ms

+ ovs-ofctl dump-flows br-int table=47
+ grep commit 
 cookie=0x728ad313, duration=9.585s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)
 cookie=0x728ad313, duration=9.585s, table=47, n_packets=0, n_bytes=0, idle_age=9, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)


[root@wsfd-advnetlab17 test]# rpm -qa | grep ovn2.13
ovn2.13-host-20.09.0-17.el8fdp.x86_64
ovn2.13-2.13.0-37.el8fdp.x86_64
ovn2.13-central-2.13.0-37.el8fdp.x86_64
Comment 15 Jianlin Shi 2020-11-24 01:18:20 UTC 
set VERIFIED per above comments
Comment 17 errata-xmlrpc 2020-12-01 15:07:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5308