Bug 1872681

Summary: ovn-controller is crashing when virtual logical port is deleted
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Numan Siddique <nusiddiq>
Component: ovn2.13Assignee: Numan Siddique <nusiddiq>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: FDP 20.ECC: ctrautma, jishi, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-16 16:01:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Numan Siddique 2020-08-26 12:11:21 UTC
Description of problem:

Steps to reproduce
----

ovn-nbctl ls-add sw0
ovn-nbctl lsp-add sw0 sw0-vir
ovn-nbctl lsp-set-type sw0-vir virtual

ovs-vsctl set open . external_ids:ovn-monitor-all=true

ovn-appctl -t ovn-controller version

ovn-nbctl lsp-del sw0-vir

sleep 1

ovn-appctl -t ovn-controller version # This command fails as ovn-controller would have crashed.




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Jianlin Shi 2020-08-27 07:34:10 UTC
test with following script:

#!/bin/bash                                                                                           

systemctl start openvswitch                                                                           
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641                                                                    
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:127.0.0.1:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1
ovs-vsctl set open . external_ids:ovn-monitor-all=true
#systemctl restart ovn-controller                                                                     
/usr/share/ovn/scripts/ovn-ctl start_controller                                                       
                                                                                                      
sleep 3                                                                                               
ovn-nbctl ls-add sw0                                                                                  
ovn-nbctl lsp-add sw0 sw0-vir                                                                         
ovn-nbctl lsp-set-type sw0-vir virtual                                                                
                                                                                                      
ps aux | grep ovn-controller                                                                          
#ovs-vsctl set open . external_ids:ovn-monitor-all=true
ovn-appctl -t ovn-controller version                                                                  
ovn-nbctl lsp-del sw0-vir                                                                             
sleep 1                                                                                               
ovn-appctl -t ovn-controller version                                                                  
ps aux | grep ovn-controller

reproduced on ovn2.13-20.06.2-1.el8fdp.x86_64:

[root@dell-per740-12 bz1872681]# rpm -qa | grep -E "openvswitch|ovn"                                  
ovn2.13-central-20.06.2-1.el8fdp.x86_64                                                               
ovn2.13-20.06.2-1.el8fdp.x86_64
ovn2.13-host-20.06.2-1.el8fdp.x86_64
openvswitch2.13-2.13.0-54.el8fdp.x86_64
kernel-kernel-networking-openvswitch-ovn-common-1.0-7.noarch                                          
python3-openvswitch2.13-2.13.0-54.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch

[root@dell-per740-12 bz1872681]# bash -x rep.sh                                                                                                                                                             
+ systemctl start openvswitch                                     
+ systemctl start ovn-northd                                                          
+ ovn-nbctl set-connection ptcp:6641                
+ ovn-sbctl set-connection ptcp:6642
+ ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:127.0.0.1:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1
+ ovs-vsctl set open . external_ids:ovn-monitor-all=true
+ /usr/share/ovn/scripts/ovn-ctl start_controller
Starting ovn-controller                                    [  OK  ]
+ sleep 3                                                  
+ ovn-nbctl ls-add sw0                                      
+ ovn-nbctl lsp-add sw0 sw0-vir                
+ ovn-nbctl lsp-set-type sw0-vir virtual             
+ ps aux                                   
+ grep ovn-controller
root      587466  0.0  0.0  41972  1088 ?        S<s  03:30   0:00 ovn-controller: monitoring pid 587467 (healthy)
root      587467  0.3  0.0 263836  5060 ?        S<l  03:30   0:00 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/ovn/ovn-controlle
r.log --pidfile=/var/run/ovn/ovn-controller.pid --detach --monitor 
root      587532  0.0  0.0  12112  1092 pts/0    S+   03:30   0:00 grep ovn-controller
+ ovn-appctl -t ovn-controller version
ovn-controller 20.06.2                                                                                                                                                                                     
Open vSwitch Library 2.13.0                           
+ ovn-nbctl lsp-del sw0-vir      
+ sleep 1                                      
+ ovn-appctl -t ovn-controller version
ovn-controller 20.06.2
Open vSwitch Library 2.13.0
+ ps aux                     
+ grep ovn-controller                 
root      587466  0.0  0.0  41972  2644 ?        S<s  03:30   0:00 ovn-controller: monitoring pid 587537 (1 crashes: pid 587467 died, killed (Segmentation fault), core dumped)
root      587537  0.0  0.0 263828  4800 ?        S<l  03:30   0:00 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/ovn/ovn-controlle
r.log --pidfile=/var/run/ovn/ovn-controller.pid --detach --monitor 

<=== ovn-controller crashed once

root      587543  0.0  0.0  12112  1000 pts/0    S+   03:30   0:00 grep ovn-controller

Verified on ovn2.13-20.06.2-2.el8fdp.x86_64:

[root@dell-per740-12 bz1872681]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.13-host-20.06.2-2.el8fdp.x86_64
ovn2.13-central-20.06.2-2.el8fdp.x86_64
openvswitch2.13-2.13.0-54.el8fdp.x86_64
kernel-kernel-networking-openvswitch-ovn-common-1.0-7.noarch
python3-openvswitch2.13-2.13.0-54.el8fdp.x86_64
ovn2.13-20.06.2-2.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-23.el8fdp.noarch

+ ovn-nbctl lsp-set-type sw0-vir virtual
+ ps aux
+ grep ovn-controller
root      589005  0.0  0.0  41972  1088 ?        S<s  03:33   0:00 ovn-controller: monitoring pid 589006 (healthy)
root      589006  0.3  0.0 263832  5140 ?        S<l  03:33   0:00 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/ovn/ovn-controller.log --pidfile=/var/run/ovn/ovn-controller.pid --detach --monitor
root      589070  0.0  0.0  12112  1112 pts/0    S+   03:33   0:00 grep ovn-controller
+ ovn-appctl -t ovn-controller version
ovn-controller 20.06.2
Open vSwitch Library 2.13.0
+ ovn-nbctl lsp-del sw0-vir
+ sleep 1
+ ovn-appctl -t ovn-controller version
ovn-controller 20.06.2
Open vSwitch Library 2.13.0
+ ps aux
+ grep ovn-controller
root      589005  0.0  0.0  41972  1088 ?        S<s  03:33   0:00 ovn-controller: monitoring pid 589006 (healthy)
root      589006  0.2  0.0 263832  5140 ?        S<l  03:33   0:00 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --no-chdir --log-file=/var/log/ovn/ovn-controller.log --pidfile=/var/run/ovn/ovn-controller.pid --detach --monitor

<=== not crashed

root      589076  0.0  0.0  12112  1072 pts/0    S+   03:33   0:00 grep ovn-controller

Comment 5 errata-xmlrpc 2020-09-16 16:01:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3769