Bug 1780719

Summary: measure the time needed by ovn-controller to resync to a new SB db
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Timothy Redaelli <tredaelli>
Component: openvswitch2.12Assignee: lorenzo bianconi <lorenzo.bianconi>
Status: CLOSED ERRATA QA Contact: ying xu <yinxu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: RHEL 8.0CC: ctrautma, jhsiao, jishi, kfida, lorenzo.bianconi, qding, ralongi, tredaelli
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.12-2.12.0-10.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1720653 Environment:
Last Closed: 2020-01-22 04:03:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nb db file none

Description Timothy Redaelli 2019-12-06 17:39:15 UTC
+++ This bug was initially created as a clone of Bug #1720653 +++

Additional info:
We are interested in the time that it takes ovn-controller to resync to a
different southbound database with substantially the same content.

So for example after creating a lot of logical switches and switch
ports like in the scale tests, copy the southbound database file (or
create an entirely new one with different switches and ports) and start
a new ovsdb-server on a different port on the server.  The change ovn-
remote in the local vswitch and see how long it takes for ovn-
controller to update the flows in OVS based on the new database.

Girish said that the buffers that ovn-controller used to download the
new database were pretty small and when they bumped the size from IIRC
4k -> 64k the download itself went quite a bit faster.

But in the end, it is not clear what the delay from (a) change ovn-
remote to (b) OVS flows are synchronized, is and if we need to look
into optimizing that.

Comment 2 Jianlin Shi 2020-01-09 12:38:37 UTC
steps:

server:

systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642            
cp ~/ovnnb_db.txt /var/lib/ovn/ovnnb_db.db -f                                                         
systemctl restart ovn-northd                                                                          
                                                                                                      

systemctl start openvswitch                                    
ovs-vsctl set open . external_ids:system-id=hv0 external_ids:ovn-remote=tcp:20.0.30.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.30.25
systemctl restart ovn-controller
                                                               
                                                                          
ovn-nbctl clear logical_switch_port 253a3b8b-76c6-4af0-ad12-d1513cf0c6a3 options
ovn-nbctl clear logical_switch_port 4cdaae8b-b472-4ae4-8f26-fd0e2a375bc1 options
ovn-nbctl clear logical_switch_port 9c3ce12f-0373-4004-9a12-7c3b34ce0ae4 options
ovn-nbctl clear logical_switch_port 063a7db6-2f19-45c2-a3b2-8414b4c40d38 options
ovn-nbctl clear logical_switch_port 173224b0-9e29-4c49-8bdf-85a0f4297bff options
ovn-nbctl clear logical_switch_port 2403f99f-9be3-427b-9937-71776bc44b5d options

wait a while for sb generating on server after setup

client:

systemctl start openvswitch
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.30.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.30.26
systemctl restart ovn-controller
sleep 10                                      
                                                      
ovs-vsctl add-port br-int vm0 -- set interface vm0 type=internal     
ovs-vsctl set Interface vm0 external_ids:iface-id=253a3b8b-76c6-4af0-ad12-d1513cf0c6a3
                
ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal
ovs-vsctl set Interface vm1 external_ids:iface-id=4cdaae8b-b472-4ae4-8f26-fd0e2a375bc1
               
ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal
ovs-vsctl set Interface vm2 external_ids:iface-id=9c3ce12f-0373-4004-9a12-7c3b34ce0ae4
ovs-vsctl add-port br-int vm3 -- set interface vm3 type=internal
ovs-vsctl set Interface vm3 external_ids:iface-id=063a7db6-2f19-45c2-a3b2-8414b4c40d38
                                                                
ovs-vsctl add-port br-int vm4 -- set interface vm4 type=internal
ovs-vsctl set Interface vm4 external_ids:iface-id=173224b0-9e29-4c49-8bdf-85a0f4297bff
ovs-vsctl add-port br-int vm5 -- set interface vm5 type=internal
ovs-vsctl set Interface vm5 external_ids:iface-id=2403f99f-9be3-427b-9937-71776bc44b5d

for i in {1..60}                                                                      
do                                                              
        ovs-ofctl dump-flows br-int | wc -l                                           
        sleep 1                 
done                                                
                     
ovs-ofctl dump-flows br-int | wc -l
num_flow=$(ovs-ofctl dump-flows br-int | wc -l)                   
ovs-vsctl set open . external_ids:ovn-remote=tcp:20.0.30.11:6642  
ovs-ofctl del-flows br-int                              
systemctl restart ovn-controller     
ovs-ofctl dump-flows br-int | wc -l          
ovs-vsctl set open . external_ids:ovn-remote=tcp:20.0.30.25:6642
systemctl restart ovn-controller                      
date > $(date +%H-%M).log  
                                
while :                                       
do                                                    
        ovs-ofctl dump-flows br-int | wc -l                          
        num2_flow=$(ovs-ofctl dump-flows br-int | wc -l)             
        date    
        [ $num_flow -le $num2_flow ] && break                  
        sleep 1                                                
done



on openvswitch2.12.0-8:

start time:
Thu Jan  9 07:19:34 EST 2020                                                                          
   
flow finish time:                                                                                                   
Thu Jan  9 07:20:45 EST 2020                                                                          
                                                                                                      
[root@ibm-x3650m4-01 bz1720653]# rpm -qa | grep -E "openvswitch|ovn"
kernel-kernel-networking-openvswitch-ovn-common-1.0-7.noarch
ovn2.12-2.12.0-19.el8fdp.x86_64                                                                       
ovn2.12-host-2.12.0-19.el8fdp.x86_64                                                                  
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
openvswitch2.12-2.12.0-8.el8fdp.x86_64                                                                
ovn2.12-central-2.12.0-19.el8fdp.x86_64

on openvswitch2.12.0-12:

start time:
Thu Jan  9 07:34:24 EST 2020

flow finish time:
Thu Jan  9 07:35:29 EST 2020

it seems that there no much improvement form 8 to 12.

Comment 3 Jianlin Shi 2020-01-09 12:39:26 UTC
Created attachment 1650959 [details]
nb db file

Comment 4 lorenzo bianconi 2020-01-10 09:57:28 UTC
hi Jianlin Shi,

50-100ms delta time is aligned to what is expected since most of the time will be spent parsing/creating the json file

Comment 5 Jianlin Shi 2020-01-10 09:59:51 UTC
set VERIFIED per comment 4

Comment 7 errata-xmlrpc 2020-01-22 04:03:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:0172