Bug 1917533

Summary: [OVN] ovn-controller may crash when deleting container port bindings
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Dumitru Ceara <dceara>
Component: ovn2.13Assignee: Dumitru Ceara <dceara>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: FDP 20.HCC: apevec, camorris, ctrautma, dalvarez, dceara, ekuris, ffernand, jappleii, jishi, lhh, majopela, mamorim, mmichels, pmannidi, ralongi, rhos-maint, scohen, zfang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn2.13-20.12.0-6.el7fdn ovn2.13-20.12.0-6.el8fdn Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1916900 Environment:
Last Closed: 2021-02-11 14:54:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1916900    

Comment 1 Dumitru Ceara 2021-01-18 16:51:48 UTC
Fix sent upstream for review: http://patchwork.ozlabs.org/project/ovn/list/?series=225062

Comment 3 Jianlin Shi 2021-01-29 02:17:05 UTC
the issue can't be reproduced with reproducer in https://github.com/ovn-org/ovn/blob/master/tests/ovn.at#L22364.
but if I added del ls after the test as below, ovn-controller would crash on 20.12.0-1:

systemctl start openvswitch                                                                           
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.38.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.38.25
systemctl restart ovn-controller                                                                      
                                                                                                      
ps aux | grep ovn-controller                                                                          
pid1=$(pidof ovn-controller)                                                                          
                                                                                                      
for i in {1..10}                                                                                      
do                                                                                                    
ovs-vsctl add-port br-int lsp1 -- set interface lsp1 type=internal external_ids:iface-id=lsp1         
ovn-nbctl ls-add ls1 -- ls-add ls2 -- lsp-add ls1 lsp1 -- lsp-add ls2 lsp-cont1 lsp1 1                
ovn-nbctl --wait=hv sync                                                                              
ovn-sbctl list port_binding                                                                           
                                                                                                      
ovn-appctl -t ovn-controller debug/pause                                                              
ovs-vsctl del-port lsp1                                                                               
ovn-nbctl --wait=sb lsp-del lsp-cont1                                                                 
ovn-appctl -t ovn-controller debug/resume                                                             
ovn-nbctl --wait=hv sync                                                                              
if ! ovn-sbctl list port_binding lsp1| grep -E "^chassis.*: \[\]"                                     
then                                                                                                  
        echo $i                                                                                       
        break                                                                                         
fi                                                                                                    
ovn-nbctl lsp-add ls2 lsp-cont1 lsp1 1                                                                
ps aux | grep ovn-controller                                                                          
if ! ps aux | grep ovn-controller | grep $pid1                                                        
then                                                                                                  
        echo $i                                                                                       
        break                                                                                         
fi                                                                                                    
ovn-nbctl ls-del ls1                                                                                  
ovn-nbctl ls-del ls2                                                                                  
ovs-vsctl del-port lsp1                                                                               
ps aux | grep ovn-controller                                                                          
ps aux | grep ovn-controller | grep $pid1                                                             
done

result on 20.12.0-1:

[root@dell-per740-12 bz1917533]# rpm -qa | grep ovn2.13
ovn2.13-20.12.0-1.el8fdp.x86_64
ovn2.13-host-20.12.0-1.el8fdp.x86_64
ovn2.13-central-20.12.0-1.el8fdp.x86_64

+ grep ovn-controller                                     
openvsw+  162207  0.0  0.0 281824  6520 ?        S<sl 21:13   0:00 ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file
=/var/log/ovn/ovn-controller.log --pidfile=/run/ovn/ovn-controller.pid --detach
root      162211  0.0  0.0  12112  1052 pts/0    S+   21:13   0:00 grep ovn-controller
++ pidof ovn-controller         
+ pid1=162207                   
+ for i in {1..10}                                                                                                                                                                                          
+ ovs-vsctl add-port br-int lsp1 -- set interface lsp1 type=internal external_ids:iface-id=lsp1
+ ovn-nbctl ls-add ls1 -- ls-add ls2 -- lsp-add ls1 lsp1 -- lsp-add ls2 lsp-cont1 lsp1 1
+ ovn-nbctl --wait=hv sync                                                                                                                                                                                  
+ ovn-sbctl list port_binding                                                  
_uuid               : e19fd24c-bd58-4df5-9ef6-3b840898a7b7                            
chassis             : e4842fc6-e198-4e35-b3b9-6d46f42e4354
datapath            : 6904b5f6-c059-43e9-bad5-a2ddb16c8284
encap               : []                                  
external_ids        : {}                                                                                                                                                                                    
gateway_chassis     : []
ha_chassis_group    : []
logical_port        : lsp-cont1
mac                 : []
nat_addresses       : []
options             : {}
parent_port         : lsp1
tag                 : 1
tunnel_key          : 1
type                : ""
virtual_parent      : []

_uuid               : 88e740f9-9954-45f4-8b81-fa923e6be73b
chassis             : e4842fc6-e198-4e35-b3b9-6d46f42e4354
datapath            : 70414c81-a3da-407f-9571-52ddcf61af4a
encap               : []
external_ids        : {}
gateway_chassis     : []
ha_chassis_group    : []
logical_port        : lsp1
mac                 : []
nat_addresses       : []
options             : {}
parent_port         : []
tag                 : []
tunnel_key          : 1
type                : ""
virtual_parent      : []
+ ovn-appctl -t ovn-controller debug/pause
+ ovs-vsctl del-port lsp1
+ ovn-nbctl --wait=sb lsp-del lsp-cont1
+ ovn-appctl -t ovn-controller debug/resume
+ ovn-nbctl --wait=hv sync
+ ovn-sbctl list port_binding lsp1
+ grep -E '^chassis.*: \[\]'
chassis             : []
+ ovn-nbctl lsp-add ls2 lsp-cont1 lsp1 1
+ ps aux
+ grep ovn-controller
openvsw+  162207  3.0  0.0 282616  8004 ?        S<sl 21:13   0:00 ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file
=/var/log/ovn/ovn-controller.log --pidfile=/run/ovn/ovn-controller.pid --detach
root      162289  0.0  0.0  12112  1096 pts/0    S+   21:13   0:00 grep ovn-controller
+ ps aux  
+ grep ovn-controller
+ grep 162207
openvsw+  162207  3.0  0.0 282616  8004 ?        S<sl 21:13   0:00 ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file
=/var/log/ovn/ovn-controller.log --pidfile=/run/ovn/ovn-controller.pid --detach
+ ovn-nbctl ls-del ls1
+ ovn-nbctl ls-del ls2
+ ovs-vsctl del-port lsp1
ovs-vsctl: no port named lsp1
+ ps aux
+ grep ovn-controller
openvsw+  162207  4.0  0.0 282616  8004 ?        S<sl 21:13   0:00 ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file
=/var/log/ovn/ovn-controller.log --pidfile=/run/ovn/ovn-controller.pid --detach
root      162297  0.0  0.0  12112   992 pts/0    S+   21:13   0:00 grep ovn-controller
+ ps aux  
+ grep ovn-controller
+ grep 162207
openvsw+  162207  4.0  0.0 282616  8004 ?        S<sl 21:13   0:00 ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file
=/var/log/ovn/ovn-controller.log --pidfile=/run/ovn/ovn-controller.pid --detach
+ for i in {1..10}
+ ovs-vsctl add-port br-int lsp1 -- set interface lsp1 type=internal external_ids:iface-id=lsp1
+ ovn-nbctl ls-add ls1 -- ls-add ls2 -- lsp-add ls1 lsp1 -- lsp-add ls2 lsp-cont1 lsp1 1
+ ovn-nbctl --wait=hv sync
+ ovn-sbctl list port_binding
_uuid               : 7c05fb01-8b76-4e72-8f6d-02c79d277ba8
chassis             : e4842fc6-e198-4e35-b3b9-6d46f42e4354
datapath            : 27cd7fde-96f8-44b9-862d-aa0caf6c0ae0
encap               : []
external_ids        : {}
gateway_chassis     : []
ha_chassis_group    : []
logical_port        : lsp1
mac                 : []
nat_addresses       : []
options             : {}
parent_port         : []
tag                 : []
tunnel_key          : 1
type                : ""
virtual_parent      : []

_uuid               : 67a0c30a-3ac5-4376-a01d-c58fbb22053c
chassis             : e4842fc6-e198-4e35-b3b9-6d46f42e4354
datapath            : ba091e6f-c9cf-4235-8e8e-f3929dec5389
encap               : []
external_ids        : {}
gateway_chassis     : []
ha_chassis_group    : []
logical_port        : lsp-cont1
mac                 : []
nat_addresses       : []
options             : {}
parent_port         : lsp1
tag                 : 1
tunnel_key          : 1
type                : ""
virtual_parent      : []
+ ovn-appctl -t ovn-controller debug/pause
+ ovs-vsctl del-port lsp1
+ ovn-nbctl --wait=sb lsp-del lsp-cont1
+ ovn-appctl -t ovn-controller debug/resume
+ ovn-nbctl --wait=hv sync
+ ovn-sbctl list port_binding lsp1
+ grep -E '^chassis.*: \[\]'
chassis             : []
+ ovn-nbctl lsp-add ls2 lsp-cont1 lsp1 1
+ ps aux
+ grep ovn-controller
openvsw+  162342  0.0  0.0 282616  5708 ?        S<sl 21:13   0:00 ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file

<=== ovn-controller has crashed and rebooted

=/var/log/ovn/ovn-controller.log --pidfile=/run/ovn/ovn-controller.pid --detach
root      162358  0.0  0.0  12112  1060 pts/0    S+   21:13   0:00 grep ovn-controller
+ ps aux  
+ grep ovn-controller
+ grep 162207
+ echo 2
2 
+ break


but the crash may be related to bz1914304

Comment 4 Jianlin Shi 2021-01-29 02:17:59 UTC
ovn-controller didn't crash on ovn2.13-20.12.0-9.

Comment 5 Dumitru Ceara 2021-02-01 20:34:44 UTC
With ovn2.13-20.12.0-1:

$ rpm -qa | grep ovn
ovn2.13-host-20.12.0-1.el8fdp.x86_64
ovn2.13-debuginfo-20.12.0-1.el8fdp.x86_64
ovn2.13-host-debuginfo-20.12.0-1.el8fdp.x86_64
ovn2.13-central-20.12.0-1.el8fdp.x86_64
ovn2.13-20.12.0-1.el8fdp.x86_64

Executing the following commands triggers the crashes fixed in this BZ:

    ovn-nbctl ls-del ls1 -- ls-del ls2

    ovs-vsctl \
        -- add-port br-int vif1 \
        -- set Interface vif1 type=internal \
        -- set Interface vif1 external_ids:iface-id=lsp1 \
        ofport-request=1

    ovn-nbctl ls-add ls1 \
        -- ls-add ls2 \
        -- lsp-add ls1 lsp1 \
        -- lsp-add ls2 lsp-cont1 lsp1 1
    ovn-nbctl --wait=hv sync

    ovn-appctl -t ovn-controller debug/pause
    ovs-vsctl del-port vif1
    ovn-nbctl --wait=sb lsp-del lsp-cont1
    ovn-appctl -t ovn-controller debug/resume

    ovn-nbctl --wait=hv sync
    ovs-vsctl \
        -- add-port br-int vif1 \
        -- set interface vif1 type=internal \
        -- set Interface vif1 external_ids:iface-id=lsp1 \
        ofport-request=1

    ovn-nbctl lsp-add ls2 lsp-cont1 lsp1 1
    ovn-nbctl --wait=hv sync

    ovn-appctl -t ovn-controller debug/pause
    ovn-nbctl lsp-del lsp1 \
        -- lsp-del lsp-cont1
    ovn-nbctl --wait=sb sync
    ovn-appctl -t ovn-controller debug/resume

    ovn-appctl -t ovn-controller debug/pause
    ovn-nbctl lsp-add ls1 lsp1 \
        -- lsp-add ls2 lsp-cont1 lsp1 1
    ovn-nbctl --wait=sb sync
    ovn-appctl -t ovn-controller debug/resume

Core dump:
(gdb) bt
#0  0x00007f333df1070f in raise () from /lib64/libc.so.6
#1  0x00007f333defab25 in abort () from /lib64/libc.so.6
#2  0x000055f21d292b94 in ovs_abort_valist (err_no=err_no@entry=0,
    format=format@entry=0x55f21d372150 "%s: assertion %s failed in %s()", args=args@entry=0x7ffee26304f0)
    at lib/util.c:419
#3  0x000055f21d29a984 in vlog_abort_valist (module_=<optimized out>,
    message=0x55f21d372150 "%s: assertion %s failed in %s()", args=args@entry=0x7ffee26304f0) at lib/vlog.c:1249
#4  0x000055f21d29aa2a in vlog_abort (module=module@entry=0x55f21d62d2e0 <this_module>,
    message=message@entry=0x55f21d372150 "%s: assertion %s failed in %s()") at lib/vlog.c:1263
#5  0x000055f21d2928ab in ovs_assert_failure (where=where@entry=0x55f21d36dae3 "lib/ovsdb-idl.c:4651",
    function=function@entry=0x55f21d36edf0 <__func__.11167> "ovsdb_idl_txn_write__",
    condition=condition@entry=0x55f21d36da59 "row->new_datum != NULL") at lib/util.c:86
#6  0x000055f21d278dfa in ovsdb_idl_txn_write__ (row_=0x55f21edcfab0,
    column=0x55f21d609960 <sbrec_port_binding_columns>, datum=0x7ffee2630630, owns_datum=<optimized out>)
    at lib/ovsdb-idl.c:4693
#7  0x000055f21d27aeeb in ovsdb_idl_txn_write_clone (row=<optimized out>, column=<optimized out>,
    datum=<optimized out>) at lib/ovsdb-idl.c:4755
#8  0x000055f21d209e60 in sbrec_port_binding_set_chassis (row=row@entry=0x55f21edcfab0, chassis=chassis@entry=0x0)
    at lib/ovn-sb-idl.c:23810
#9  0x000055f21d1b457b in release_lport (pb=0x55f21edcfab0, sb_readonly=<optimized out>,
    tracked_datapaths=0x55f21ed8b978) at controller/binding.c:893
#10 0x000055f21d1b4c6b in release_local_binding_children (chassis_rec=0x55f21eddc150, lbinding=0x55f21edd10f0,
    sb_readonly=false, tracked_dp_bindings=0x55f21ed8b978) at controller/binding.c:956
#11 0x000055f21d1b71f2 in handle_deleted_vif_lport (b_ctx_out=0x7ffee26307b0, b_ctx_in=0x7ffee2630800,
--Type <RET> for more, q to quit, c to continue without paging-- 
    mized out>, pb=0x55f21edcf910) at controller/binding.h:108
#12 binding_handle_port_binding_changes (b_ctx_in=b_ctx_in@entry=0x7ffee2630800, b_ctx_out=b_ctx_out@entry=0x7ffee26307b0) at controller/binding.c:2150
#13 0x000055f21d1d8508 in runtime_data_sb_port_binding_handler (node=0x7ffee2631b70, data=0x55f21ed8b890) at controller/ovn-controller.c:1480
#14 0x000055f21d1f1f43 in engine_compute (recompute_allowed=<optimized out>, node=<optimized out>) at lib/inc-proc-eng.c:306
#15 engine_run_node (recompute_allowed=true, node=0x7ffee2631b70) at lib/inc-proc-eng.c:352
#16 engine_run (recompute_allowed=recompute_allowed@entry=true) at lib/inc-proc-eng.c:377
#17 0x000055f21d1b1d1e in main (argc=<optimized out>, argv=<optimized out>) at controller/ovn-controller.c:2794

With ovn2.13-20.12.0-9 the crashes are not seen.

Regards,
Dumitru

Comment 9 Jianlin Shi 2021-02-03 02:30:44 UTC
with reproducer in comment 5, reproduced on 20.12.0-1.el7:

enable_coredump()
{
        ulimit -c unlimited
        ulimit -s unlimited
        sysctl -w fs.suid_dumpable=2                                                                  
        if ! sysctl kernel.core_pattern | grep systemd-coredump                                       
        then                                                                                          
                sysctl -w kernel.core_pattern="|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e"                                                                                                 
        fi                                                                                            
        rm -rf /var/lib/systemd/coredump/*                                                            
        rm -rf /run/log/journal/*                                                                     
        rm -rf /var/log/journal/*                                                                     
        systemctl restart systemd-journald                                                            
}

[root@wsfd-advnetlab20 bz1917533]# bash -x rep.sh                         
+ systemctl start openvswitch                                                                               
+ systemctl start ovn-northd                                                                          
+ ovn-nbctl set-connection ptcp:6641                                    
+ ovn-sbctl set-connection ptcp:6642                                                                  
+ ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.174.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.174.25
+ systemctl restart ovn-controller                                                          
+ ovn-nbctl ls-del ls1 -- ls-del ls2                                     
ovn-nbctl: ls1: switch name not found                                   
+ ovs-vsctl -- add-port br-int vif1 -- set Interface vif1 type=internal -- set Interface vif1 external_ids:iface-id=lsp1 ofport-request=1
+ ovn-nbctl ls-add ls1 -- ls-add ls2 -- lsp-add ls1 lsp1 -- lsp-add ls2 lsp-cont1 lsp1 1              
+ ovn-nbctl --wait=hv sync                                               
+ ovn-appctl -t ovn-controller debug/pause                           
+ ovs-vsctl del-port vif1                                                                             
+ ovn-nbctl --wait=sb lsp-del lsp-cont1                          
+ ovn-appctl -t ovn-controller debug/resume                       
+ ovn-nbctl --wait=hv sync                                              
+ ovs-vsctl -- add-port br-int vif1 -- set interface vif1 type=internal -- set Interface vif1 external_ids:iface-id=lsp1 ofport-request=1
+ ovn-nbctl lsp-add ls2 lsp-cont1 lsp1 1                                   
+ ovn-nbctl --wait=hv sync                                               
+ ovn-appctl -t ovn-controller debug/pause                              
+ ovn-nbctl lsp-del lsp1 -- lsp-del lsp-cont1                                                         
+ ovn-nbctl --wait=sb sync                                                    
+ ovn-appctl -t ovn-controller debug/resume                                                           
+ ovn-appctl -t ovn-controller debug/pause                           
2021-02-03T02:26:07Z|00001|jsonrpc|WARN|unix:/var/run/ovn/ovn-controller.100997.ctl: receive error: Connection reset by peer
2021-02-03T02:26:07Z|00002|unixctl|WARN|error communicating with unix:/var/run/ovn/ovn-controller.100997.ctl: Connection reset by peer
ovn-appctl: ovn-controller: transaction error (Connection reset by peer)                  
+ ovn-nbctl lsp-add ls1 lsp1 -- lsp-add ls2 lsp-cont1 lsp1 1             
+ ovn-nbctl --wait=sb sync                                         
+ ovn-appctl -t ovn-controller debug/resume                               
2021-02-03T02:26:07Z|00001|daemon_unix|WARN|/var/run/ovn/ovn-controller.pid: open: No such file or directory
ovn-appctl: cannot read pidfile "/var/run/ovn/ovn-controller.pid" (No such file or directory)

[root@wsfd-advnetlab20 bz1917533]# coredumpctl list                     
TIME                            PID   UID   GID SIG PRESENT EXE                       
Tue 2021-02-02 21:26:08 EST  100898   996   993  11 * /usr/bin/ovn-controller              
Tue 2021-02-02 21:26:08 EST  100997   996   993   6 * /usr/bin/ovn-controller               
[root@wsfd-advnetlab20 bz1917533]# coredumpctl info                      
           PID: 100898 (ovn-controller)                                 
           UID: 996 (openvswitch)                                   
           GID: 993 (openvswitch)                                      
        Signal: 11 (SEGV)                                                
     Timestamp: Tue 2021-02-02 21:26:06 EST (18s ago)                
  Command Line: ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info -
    Executable: /usr/bin/ovn-controller
 Control Group: /system.slice/ovn-controller.service
          Unit: ovn-controller.service
         Slice: system.slice
       Boot ID: 1da9f7cf38124c07998685886c5a97d5
    Machine ID: 0450469b5cd549449d7843859dd761c7
      Hostname: wsfd-advnetlab20.anl.lab.eng.bos.redhat.com
      Coredump: /var/lib/systemd/coredump/core.ovn-controller.996.1da9f7cf38124c07998685886c5a97d5.100
       Message: Process 100898 (ovn-controller) of user 996 dumped core.
       
                Stack trace of thread 100898:
                #0  0x000055fb20b2768a consider_container_lport (ovn-controller)
                #1  0x000055fb20b29092 binding_handle_ovs_interface_changes (ovn-controller)
                #2  0x000055fb20b49f94 runtime_data_ovs_interface_handler (ovn-controller)
                #3  0x000055fb20b63c00 engine_run (ovn-controller)
                #4  0x000055fb20b23dc3 main (ovn-controller)
                #5  0x00007f40f528c555 __libc_start_main (libc.so.6)
                #6  0x000055fb20b25535 _start (ovn-controller)
                
                Stack trace of thread 100899:
                #0  0x00007f40f5e5154d __lll_lock_wait (libpthread.so.0)
                #1  0x00007f40f5e4ceb6 _L_lock_941 (libpthread.so.0)
                #2  0x00007f40f5e4cdaf pthread_mutex_lock (libpthread.so.0)
                #3  0x000055fb20bdc048 ovs_mutex_lock_at (ovn-controller)
                #4  0x000055fb20b870c1 fatal_signal_run (ovn-controller)
                #5  0x000055fb20bf38da poll_block (ovn-controller)
                #6  0x000055fb20b41bda pinctrl_handler (ovn-controller)
                #7  0x000055fb20bdcd9f ovsthread_wrapper (ovn-controller)
                #8  0x00007f40f5e4aea5 start_thread (libpthread.so.0)
                #9  0x00007f40f536896d __clone (libc.so.6)
Stack trace of thread 100900:
                #0  0x00007f40f5e5154d __lll_lock_wait (libpthread.so.0)
                #1  0x00007f40f5e4ceb6 _L_lock_941 (libpthread.so.0)
                #2  0x00007f40f5e4cdaf pthread_mutex_lock (libpthread.so.0)
                #3  0x000055fb20bdc048 ovs_mutex_lock_at (ovn-controller)
                #4  0x000055fb20b870c1 fatal_signal_run (ovn-controller)
                #5  0x000055fb20bf38da poll_block (ovn-controller)
                #6  0x000055fb20bda9de ovsrcu_postpone_thread (ovn-controller)
                #7  0x000055fb20bdcd9f ovsthread_wrapper (ovn-controller)
                #8  0x00007f40f5e4aea5 start_thread (libpthread.so.0)
                #9  0x00007f40f536896d __clone (libc.so.6)
                
                Stack trace of thread 100901:
                #0  0x00007f40f5e516fd __write (libpthread.so.0)
                #1  0x000055fb20c0a2a1 vlog_valist (ovn-controller)
                #2  0x000055fb20c097af vlog (ovn-controller)
                #3  0x000055fb20b87140 fatal_signal_run (ovn-controller)
                #4  0x000055fb20bf38da poll_block (ovn-controller)
                #5  0x000055fb20bf2bac stopwatch_thread (ovn-controller)
                #6  0x000055fb20bdcd9f ovsthread_wrapper (ovn-controller)
                #7  0x00007f40f5e4aea5 start_thread (libpthread.so.0)
                #8  0x00007f40f536896d __clone (libc.so.6)

[root@wsfd-advnetlab20 bz1917533]# rpm -qa | grep -E ovn2.13
ovn2.13-20.12.0-1.el7fdp.x86_64
ovn2.13-host-20.12.0-1.el7fdp.x86_64
ovn2.13-central-20.12.0-1.el7fdp.x86_64

Verified on 20.12.0-9.el7:

[root@wsfd-advnetlab20 bz1917533]# bash -x rep.sh 
+ systemctl start openvswitch
+ systemctl start ovn-northd
+ ovn-nbctl set-connection ptcp:6641
+ ovn-sbctl set-connection ptcp:6642
+ ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.174.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.174.25
+ systemctl restart ovn-controller
+ ovn-nbctl ls-del ls1 -- ls-del ls2
ovn-nbctl: ls1: switch name not found
+ ovs-vsctl -- add-port br-int vif1 -- set Interface vif1 type=internal -- set Interface vif1 external_ids:iface-id=lsp1 ofport-request=1
+ ovn-nbctl ls-add ls1 -- ls-add ls2 -- lsp-add ls1 lsp1 -- lsp-add ls2 lsp-cont1 lsp1 1
+ ovn-nbctl --wait=hv sync
+ ovn-appctl -t ovn-controller debug/pause
+ ovs-vsctl del-port vif1
+ ovn-nbctl --wait=sb lsp-del lsp-cont1
+ ovn-appctl -t ovn-controller debug/resume
+ ovn-nbctl --wait=hv sync
+ ovs-vsctl -- add-port br-int vif1 -- set interface vif1 type=internal -- set Interface vif1 external_ids:iface-id=lsp1 ofport-request=1
+ ovn-nbctl lsp-add ls2 lsp-cont1 lsp1 1
+ ovn-nbctl --wait=hv sync
+ ovn-appctl -t ovn-controller debug/pause
+ ovn-nbctl lsp-del lsp1 -- lsp-del lsp-cont1
+ ovn-nbctl --wait=sb sync
+ ovn-appctl -t ovn-controller debug/resume
+ ovn-appctl -t ovn-controller debug/pause
+ ovn-nbctl lsp-add ls1 lsp1 -- lsp-add ls2 lsp-cont1 lsp1 1
+ ovn-nbctl --wait=sb sync
+ ovn-appctl -t ovn-controller debug/resume
[root@wsfd-advnetlab20 bz1917533]# coredumpctl list
No coredumps found.
[root@wsfd-advnetlab20 bz1917533]# rpm -qa | grep ovn2.13
ovn2.13-20.12.0-9.el7fdp.x86_64
ovn2.13-central-20.12.0-9.el7fdp.x86_64
ovn2.13-host-20.12.0-9.el7fdp.x86_64

Comment 11 errata-xmlrpc 2021-02-11 14:54:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0496