Bug 2178866

Summary:	virt-admin coredumped when executing 'virt-admin srv-threadpool-info virtqemud'
Product:	Red Hat Enterprise Linux 9	Reporter:	yafu <yafu>
Component:	libvirt	Assignee:	Ján Tomko <jtomko>
libvirt sub component:	General	QA Contact:	Lili Zhu <lizhu>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	hhan, jdenemar, jtomko, lcheng, lizhu, lmen, virt-maint
Version:	9.3	Keywords:	Automation, Regression, Triaged
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-9.2.0-1.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-11-07 08:31:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:	9.2.0
Embargoed:

Description yafu 2023-03-16 01:30:11 UTC

Description of problem:
virt-admin coredumped when executing 'virt-admin srv-threadpool-info virtqemud'

Version-Release number of selected component (if applicable):
libvirt-9.1.0-1.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1.# virt-admin server-threadpool-info virtqemud
2023-03-16 01:25:00.789+0000: 43672: info : libvirt version: 9.1.0, package: 1.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-03-13-09:31:29, )
2023-03-16 01:25:00.789+0000: 43672: info : hostname: ibm-x3250m6-04.lab.eng.pek2.redhat.com
2023-03-16 01:25:00.789+0000: 43672: warning : virObjectGetLockableObj:411 : Object 0x7ffc29ec7f78 ((null)) is not a virObjectLockable instance
Segmentation fault (core dumped)

2.
3.

Actual results:
virt-admin crashed when executing 'virt-admin srv-threadpool-info virtqemud'

Expected results:
virt-admin works as expected without core dumped.

Additional info:
1.The backtrace:
(gdb) t a a bt

Thread 2 (Thread 0x7f6c91dff640 (LWP 43673)):
#0  0x00007f6c9374296f in __GI___poll (fds=0x7f6c8c004fa0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f6c93f8849c in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7f6c8c004fa0, timeout=<optimized out>, context=0x7f6c8c001ca0) at ../glib/gmain.c:4434
#2  g_main_context_iterate.constprop.0 (context=context@entry=0x7f6c8c001ca0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4126
#3  0x00007f6c93f315f3 in g_main_context_iteration (context=0x7f6c8c001ca0, context@entry=0x0, may_block=may_block@entry=1) at ../glib/gmain.c:4196
#4  0x00007f6c93ad7df4 in virEventGLibRunOnce () at ../src/util/vireventglib.c:515
#5  0x000055ea2973e9b5 in vshEventLoop ()
#6  0x00007f6c93b35e69 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:256
#7  0x00007f6c9369f832 in start_thread (arg=<optimized out>) at pthread_create.c:443
#8  0x00007f6c9363f450 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f6c92127f80 (LWP 43672)):
#0  ___pthread_mutex_lock (mutex=mutex@entry=0x18) at pthread_mutex_lock.c:81
#1  0x00007f6c93b2cff9 in virMutexLock (m=m@entry=0x18) at ../src/util/virthread.c:91
#2  0x00007f6c93b2d01e in virLockGuardLock (m=0x18) at ../src/util/virthread.c:102
#3  0x00007f6c94021d06 in remoteAdminConnectLookupServer (flags=0, name=0x55ea297b32a0 "virtqemud", conn=0x55ea297e5820) at src/admin/admin_client.h:111
#4  virAdmConnectLookupServer (conn=0x55ea297e5820, name=0x55ea297b32a0 "virtqemud", flags=0) at ../src/admin/libvirt-admin.c:782
#5  0x000055ea2973ed5d in cmdSrvThreadpoolInfo ()
#6  0x000055ea2973eb1f in vshCommandRun ()
#7  0x000055ea2973d0cd in main ()
(gdb) 

Thread 2 (Thread 0x7f6c91dff640 (LWP 43673)):
#0  0x00007f6c9374296f in __GI___poll (fds=0x7f6c8c004fa0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f6c93f8849c in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7f6c8c004fa0, timeout=<optimized out>, context=0x7f6c8c001ca0) at ../glib/gmain.c:4434
#2  g_main_context_iterate.constprop.0 (context=context@entry=0x7f6c8c001ca0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4126
#3  0x00007f6c93f315f3 in g_main_context_iteration (context=0x7f6c8c001ca0, context@entry=0x0, may_block=may_block@entry=1) at ../glib/gmain.c:4196
#4  0x00007f6c93ad7df4 in virEventGLibRunOnce () at ../src/util/vireventglib.c:515
#5  0x000055ea2973e9b5 in vshEventLoop ()
#6  0x00007f6c93b35e69 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:256
#7  0x00007f6c9369f832 in start_thread (arg=<optimized out>) at pthread_create.c:443
#8  0x00007f6c9363f450 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f6c92127f80 (LWP 43672)):
#0  ___pthread_mutex_lock (mutex=mutex@entry=0x18) at pthread_mutex_lock.c:81
#1  0x00007f6c93b2cff9 in virMutexLock (m=m@entry=0x18) at ../src/util/virthread.c:91
#2  0x00007f6c93b2d01e in virLockGuardLock (m=0x18) at ../src/util/virthread.c:102
#3  0x00007f6c94021d06 in remoteAdminConnectLookupServer (flags=0, name=0x55ea297b32a0 "virtqemud", conn=0x55ea297e5820) at src/admin/admin_client.h:111
#4  virAdmConnectLookupServer (conn=0x55ea297e5820, name=0x55ea297b32a0 "virtqemud", flags=0) at ../src/admin/libvirt-admin.c:782
#5  0x000055ea2973ed5d in cmdSrvThreadpoolInfo ()
#6  0x000055ea2973eb1f in vshCommandRun ()
#7  0x000055ea2973d0cd in main ()

Comment 1 Han Han 2023-03-17 03:01:24 UTC

Bisection shows the first bad commit is:
commit 778c3004609ede0a9df4cf3e01c031047530efb7
Author: Daniel P. Berrangé <berrange>
Date:   Thu Dec 22 10:28:50 2022 -0500

    rpc: use VIR_LOCK_GUARD in remote client code
    
    Using VIR_LOCK_GUARD helps to simplify the control flow
    logic.
    
    Reviewed-by: Ján Tomko <jtomko>
    Signed-off-by: Daniel P. Berrangé <berrange>

Comment 2 Ján Tomko 2023-03-17 15:01:45 UTC

Upstream patch:
https://listman.redhat.com/archives/libvir-list/2023-March/238847.html

Comment 3 Ján Tomko 2023-03-17 15:44:07 UTC

Pushed upstream as:
commit 50f0e8e7aa32c307e976a09387bccd5e40a285d9
Author:     Ján Tomko <jtomko>
CommitDate: 2023-03-17 16:42:55 +0100

    rpc: fix typo in admin code generation
    
    An extra '&' introduced a crash.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2178866
    
    Fixes: 778c3004609ede0a9df4cf3e01c031047530efb7
    Signed-off-by: Ján Tomko <jtomko>
    Reviewed-by: Peter Krempa <pkrempa>

git describe: v9.1.0-247-g50f0e8e7aa

Comment 4 Han Han 2023-03-20 04:10:17 UTC

Test passes on v9.1.0-248-g27d8bcc337
+ /root/libvirt/build/tools/virt-admin server-threadpool-info virtqemud
error: Server not found: No server named 'virtqemud'

Comment 5 Lili Zhu 2023-04-06 03:21:26 UTC

Tested with:
libvirt-9.2.0-1.el9.x86_64

1. check the threadpool info of each daemon with connecting to the daemon firstly
# for drv in qemu interface network nodedev nwfilter secret storage ; do virt-admin server-threadpool-info virt${drv}d; done
error: Server not found: No server named 'virtqemud'

error: Server not found: No server named 'virtinterfaced'

error: Server not found: No server named 'virtnetworkd'

error: Server not found: No server named 'virtnodedevd'

error: Server not found: No server named 'virtnwfilterd'

error: Server not found: No server named 'virtsecretd'

error: Server not found: No server named 'virtstoraged'

# virt-admin server-threadpool-info virtproxyd
minWorkers     : 5
maxWorkers     : 20
nWorkers       : 5
freeWorkers    : 5
prioWorkers    : 5
jobQueueDepth  : 0


2. check the threadpool info of each daemon
# for drv in qemu interface network nodedev nwfilter secret storage ; do virt-admin -c virt${drv}d:///system client-list virt${drv}d; done
 Id   Transport   Connected since
-----------------------------------

 Id   Transport   Connected since
-----------------------------------

 Id   Transport   Connected since
-----------------------------------

 Id   Transport   Connected since
-----------------------------------

 Id   Transport   Connected since
-----------------------------------

 Id   Transport   Connected since
-----------------------------------

 Id   Transport   Connected since
-----------------------------------


# virt-admin client-list virtproxyd
 Id   Transport   Connected since
-----------------------------------

(Not coredumped)

Comment 9 Lili Zhu 2023-05-19 08:09:27 UTC

Tested with:
libvirt-9.3.0-1.el9.x86_64

The testing steps are the same with those in Comment #5

Mark the bug as verified.

Comment 11 errata-xmlrpc 2023-11-07 08:31:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6409