Bug 1900484 - Updating ovn-controller from ovn2.13-host-2.13.0-37/ovn2.13-20.06-xx to ovn2.13-20.09-xx breaks datapath connectivity
Summary: Updating ovn-controller from ovn2.13-host-2.13.0-37/ovn2.13-20.06-xx to ovn2....
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: ovn2.13
Version: FDP 20.E
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Numan Siddique
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On: 1895220 1899936
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-23 06:53 UTC by Numan Siddique
Modified: 2021-01-06 08:41 UTC (History)
25 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1895220
Environment:
Last Closed: 2020-12-01 15:07:14 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:5308 0 None None None 2020-12-01 15:07:32 UTC

Description Numan Siddique 2020-11-23 06:53:11 UTC
Updating ovn-controller from ovn2.13-host-2.13.0-37.el8fdp.x86_64 to ovn2.13-20.09 breaks datapath connectivity.

When only ovn-controller is updated, it rejects all logical flows with ct_commit action and a warning is seen in the ovn-controller log. Because of which datapath disruption is seen if ACLs with allow-related are configured on logical switches/port groups.

Comment 1 Numan Siddique 2020-11-23 06:55:15 UTC
Steps to reproduce.

Install OVN version ovn2.13-20.06.2-15

# Create OVN resources.
ovn-nbctl ls-add sw0
ovn-nbctl lsp-add sw0 sw0-port1
ovn-nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3"
ovn-nbctl lsp-add sw0 sw0-port2
ovn-nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 10.0.0.4"

ovn-nbctl acl-add sw0 to-lport 1002 "ip" allow-related
ovn-nbctl acl-add sw0 to-lport 1000 "ip" drop

# Create namespace for sw0-port1 and sw0-port2

# Ping from sw0-port1 to sw0-port2. It should work fine.

ovs-ofctl dump-flows br-int table=47 | grep commit

# Stop ovn-controller. Note --restart is important. Otherwise ovn-controller will unregister
# and delete all flows.
ovn-appctl -t ovn-controller exit --restart 


# Update only ovn-controller to ovn2.13-20.09.1-15

# Start ovn-controller.
# ping from sw0-port1 to sw0-port2 should work.

# There should be commit OF flows with ct action.
 ovs-ofctl dump-flows br-int table=47 | grep commit

Comment 2 Jianlin Shi 2020-11-23 08:24:27 UTC
Verified on ovn2.13-host-20.09.0-17.el8fdp.x86_64:

[root@wsfd-advnetlab17 bz1900484]# rpm -ivh ~/2.13.0-37/ovn2.13-*             
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...                               
   1:ovn2.13-2.13.0-37.el8fdp         ################################# [ 33%]                        
Unit ovn-northd.service could not be found.
   2:ovn2.13-central-2.13.0-37.el8fdp ################################# [ 67%]
Unit ovn-controller.service could not be found.        
   3:ovn2.13-host-2.13.0-37.el8fdp    ################################# [100%]

[root@wsfd-advnetlab17 bz1900484]# bash -x rep.sh                                                                                     
+ systemctl start openvswitch                                                       
+ systemctl start ovn-northd    
+ ovn-nbctl set-connection ptcp:6641                        
+ ovn-sbctl set-connection ptcp:6642                            
+ ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.151.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.151.25
+ systemctl restart ovn-controller                               
+ ovn-nbctl ls-add sw0                                                                                                                                                                                      
+ ovn-nbctl lsp-add sw0 sw0-port1                                             
+ ovn-nbctl lsp-set-addresses sw0-port1 '50:54:00:00:00:03 10.0.0.3'                                                                                                                                        
+ ovn-nbctl lsp-add sw0 sw0-port2                                           
+ ovn-nbctl lsp-set-addresses sw0-port2 '50:54:00:00:00:04 10.0.0.4'                                                                 
+ ovn-nbctl acl-add sw0 to-lport 1002 ip allow-related                                                                                                          
+ ovn-nbctl acl-add sw0 to-lport 1000 ip drop                                                                                                                                   
+ ip netns add sw0p1                                                          
+ ovs-vsctl add-port br-int sw0p1 -- set interface sw0p1 type=internal external_ids:iface-id=sw0-port1
+ ip link set sw0p1 netns sw0p1                                    
+ ip netns exec sw0p1 ip link set sw0p1 address 50:54:00:00:00:03             
+ ip netns exec sw0p1 ip link set sw0p1 up                                    
+ ip netns exec sw0p1 ip addr add 10.0.0.3/24 dev sw0p1                       
+ ip netns add sw0p2                                   
+ ovs-vsctl add-port br-int sw0p2 -- set interface sw0p2 type=internal external_ids:iface-id=sw0-port2
+ ip link set sw0p2 netns sw0p2            
+ ip netns exec sw0p2 ip link set sw0p2 address 50:54:00:00:00:04             
+ ip netns exec sw0p2 ip link set sw0p2 up             
+ ip netns exec sw0p2 ip addr add 10.0.0.4/24 dev sw0p2                       
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=1 ttl=64 time=1019 ms            
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=1.79 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.597 ms
                             
--- 10.0.0.4 ping statistics ---                                                                                                                                                                            
3 packets transmitted, 3 received, 0% packet loss, time 22ms
rtt min/avg/max/mdev = 0.597/340.620/1019.469/480.018 ms, pipe 2                                                                                                                                            
+ ovs-ofctl dump-flows br-int table=47                                                                                                                                 
+ grep commit                                            
 cookie=0x75a74162, duration=1.440s, table=47, n_packets=2, n_bytes=196, idle_age=1, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),
resubmit(,48)                    
 cookie=0x75a74162, duration=1.439s, table=47, n_packets=0, n_bytes=0, idle_age=1, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),
resubmit(,48)                     
+ ovn-appctl -t ovn-controller exit --restart                       
+ exit 0                                              
[root@wsfd-advnetlab17 bz1900484]# rpm -Uvh ~/20.09.0-17/ovn2.13-host-20.09.0-17.el8fdp.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]                        
Updating / installing...              
   1:ovn2.13-host-20.09.0-17.el8fdp   ################################# [ 50%]
Cleaning up / removing...                 
   2:ovn2.13-host-2.13.0-37.el8fdp    ################################# [100%]

[root@wsfd-advnetlab17 bz1900484]# bash -x rep_step2.sh
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
64 bytes from 10.0.0.4: icmp_seq=2 ttl=64 time=0.706 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=0.262 ms

--- 10.0.0.4 ping statistics ---      
3 packets transmitted, 2 received, 33.3333% packet loss, time 85ms
rtt min/avg/max/mdev = 0.262/0.484/0.706/0.222 ms

<=== ping works

+ ovs-ofctl dump-flows br-int table=47
+ grep commit
 cookie=0x75a74162, duration=2.102s, table=47, n_packets=1, n_bytes=98, idle_age=2, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)
 cookie=0x75a74162, duration=2.102s, table=47, n_packets=0, n_bytes=0, idle_age=2, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)
[root@wsfd-advnetlab17 bz1900484]# rpm -qa | grep ovn2.13                                                                                                                                                  
ovn2.13-host-20.09.0-17.el8fdp.x86_64
ovn2.13-2.13.0-37.el8fdp.x86_64
ovn2.13-central-2.13.0-37.el8fdp.x86_64

Comment 3 Jianlin Shi 2020-11-23 08:27:56 UTC
Verified on rhel8 version:

[root@wsfd-advnetlab16 bz1900484]# rpm -Uvh ~/20.09.0-17/ovn2.13-host-20.09.0-17.el7fdp.x86_64.rpm    
Preparing...                          ################################# [100%]
Updating / installing...                                                                              
   1:ovn2.13-host-20.09.0-17.el7fdp   ################################# [ 50%]
Cleaning up / removing...                                        
   2:ovn2.13-host-2.13.0-37.el7fdp    ################################# [100%]                        
[root@wsfd-advnetlab16 bz1900484]# bash -x rep_step2.sh                                               
+ systemctl start ovn-controller
+ ip netns exec sw0p1 ping 10.0.0.4 -c 3                                                              
PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.                                                        
64 bytes from 10.0.0.4: icmp_seq=3 ttl=64 time=1.17 ms                                                
                                          
--- 10.0.0.4 ping statistics ---                       
3 packets transmitted, 1 received, 66% packet loss, time 1999ms                                       
rtt min/avg/max/mdev = 1.175/1.175/1.175/0.000 ms        

<=== ping passed
                                             
+ ovs-ofctl dump-flows br-int table=47                
+ grep commit                                         
 cookie=0x714b946, duration=11.012s, table=47, n_packets=1, n_bytes=98, idle_age=10, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)
 cookie=0x714b946, duration=11.012s, table=47, n_packets=0, n_bytes=0, idle_age=11, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)

[root@wsfd-advnetlab16 bz1900484]# rpm -qa | grep -E "openvswitch|ovn"                                                                                                                                     
openvswitch2.13-2.13.0-63.el7fdp.x86_64
ovn2.13-central-2.13.0-37.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-15.el7fdp.noarch
ovn2.13-2.13.0-37.el7fdp.x86_64
ovn2.13-host-20.09.0-17.el7fdp.x86_64

Comment 6 Jianlin Shi 2020-11-24 01:17:58 UTC
set VERIFIED per comment 2 and comment 3

Comment 8 errata-xmlrpc 2020-12-01 15:07:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5308


Note You need to log in before you can comment on or make changes to this bug.