Bug 2164058

Summary: Use the libc native backtrace mechanism exposed in OvS
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Dumitru Ceara <dceara>
Component: ovn23.09Assignee: Ales Musil <amusil>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: unspecified Docs Contact:
Priority: high    
Version: FDP 23.ACC: amusil, ctrautma, i.maximets, jiji, mmichels
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn23.09-23.09.0-alpha.157.el9fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-01-24 11:17:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2177760    
Bug Blocks:    

Description Dumitru Ceara 2023-01-24 15:54:00 UTC
Description of problem:

OVS/OVN binaries send out a backtrace to their monitor processes (if any) only if they link with lib-unwind.  Currently downstream (and upstream) RPMs are built without lib-unwind.

The upstream OVS spec file requires lib-unwind:
https://github.com/openvswitch/ovs/blob/master/rhel/openvswitch.spec.in#L42

Comment 6 Jianlin Shi 2023-12-04 06:57:01 UTC
Hi Ales,
what should we do for this bug?
ovn23.09 doesn't depend on lib-unwind:
[root@kvm-01-guest22 23.09-73]# yum install ovn23.09-*
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 1:28:25 ago on Mon 04 Dec 2023 06:27:50 AM CET.
Dependencies resolved.
==================================================================================================================================================================================
 Package                                       Architecture                        Version                                        Repository                                 Size
==================================================================================================================================================================================
Installing:
 ovn23.09                                      x86_64                              23.09.0-73.el9fdp                              @commandline                              1.1 M
 ovn23.09-central                              x86_64                              23.09.0-73.el9fdp                              @commandline                              539 k
 ovn23.09-host                                 x86_64                              23.09.0-73.el9fdp                              @commandline                              683 k

Transaction Summary
==================================================================================================================================================================================
Install  3 Packages

Total size: 2.3 M
Installed size: 7.8 M
Is this ok [y/N]: 

[root@kvm-01-guest22 23.09-73]# yum info lib-unwind
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 1:28:41 ago on Mon 04 Dec 2023 06:27:50 AM CET.
Error: No matching Packages to list

Comment 7 Ales Musil 2023-12-04 11:07:51 UTC
Hi Jianlin,

right we are using the libc native mechanism which was exposed via OvS.
To check if that works you can kill the process with SIGSEGV e.g. "kill -SIGSEGV $(pidof -s ovn-controller)".
After that there should be backtrace in the ovn-controller.log something like the backtrace below:

SIGSEGV detected, backtrace:
ovn-controller[0x56b1da]
ovn-controller[0x49a358]
/lib64/libc.so.6(+0x3e9a0)[0x7f5700a339a0]
/lib64/libc.so.6(__poll+0x4d)[0x7f5700afcb8d]
ovn-controller[0x5126a2]
ovn-controller[0x507afa]
ovn-controller[0x40a860]
/lib64/libc.so.6(+0x2814a)[0x7f5700a1d14a]
/lib64/libc.so.6(__libc_start_main+0x8b)[0x7f5700a1d20b]
ovn-controller[0x40ced5]

Comment 8 Jianlin Shi 2023-12-05 01:54:21 UTC
tested with following script:

systemctl start openvswitch                                                                           
systemctl start ovn-northd                                                                            
ovn-nbctl set-connection ptcp:6641                                                                    
ovn-sbctl set-connection ptcp:6642                                                                    
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:127.0.0.1:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1
systemctl restart ovn-controller

kill -SIGSEGV $(pidof -s ovn-controller)
grep SIGSEGV -A 20 /var/log/ovn/ovn-controller.log

result on ovn23.03-23.03.0-106.el9:

[root@kvm-01-guest22 bz2164058]# rpm -qa | grep -E "ovn23.03|openvswitch3.1"
openvswitch3.1-3.1.0-70.el9fdp.x86_64
ovn23.03-23.03.0-106.el9fdp.x86_64
ovn23.03-central-23.03.0-106.el9fdp.x86_64
ovn23.03-host-23.03.0-106.el9fdp.x86_64

+ kill -SIGSEGV 55394
+ grep SIGSEGV -A 20 /var/log/ovn/ovn-controller.log


result on ovn23.09-23.09.0-73.el9:

[root@kvm-01-guest22 bz2164058]# rpm -qa | grep -E "ovn23.09|openvswitch3.1"
openvswitch3.1-3.1.0-70.el9fdp.x86_64
ovn23.09-23.09.0-73.el9fdp.x86_64
ovn23.09-central-23.09.0-73.el9fdp.x86_64
ovn23.09-host-23.09.0-73.el9fdp.x86_64

+ grep SIGSEGV -A 20 /var/log/ovn/ovn-controller.log
SIGSEGV detected, backtrace:
ovn-controller(+0xa9b71)[0x563d95a68b71]
/lib64/libc.so.6(+0x54db0)[0x7fe377254db0]
/lib64/libc.so.6(__poll+0x4f)[0x7fe3773428bf]
ovn-controller(+0x111c6d)[0x563d95ad0c6d]
ovn-controller(+0xe7d2d)[0x563d95aa6d2d]
ovn-controller(+0x253e4)[0x563d959e43e4]
/lib64/libc.so.6(+0x3feb0)[0x7fe37723feb0]
/lib64/libc.so.6(__libc_start_main+0x80)[0x7fe37723ff60]
ovn-controller(+0x290e5)[0x563d959e80e5]
2023-12-05T01:53:53.522Z|00001|fatal_signal(stopwatch1)|WARN|terminating with signal 11 (Segmentation fault)

set Verified

Comment 10 errata-xmlrpc 2024-01-24 11:17:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn23.09 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0392