1739564 – libvirtd crashes non deterministically on shutdown

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1739564 - libvirtd crashes non deterministically on shutdown

Summary: libvirtd crashes non deterministically on shutdown

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	unspecified
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Michal Privoznik
QA Contact:	yafu
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-08-09 14:15 UTC by Katerina Koukiou
Modified:	2022-05-17 13:01 UTC (History)
CC List:	9 users (show)
Fixed In Version:	libvirt-7.7.0-3.el9
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-05-17 12:45:05 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:
Flags:	knoel: mirror+

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2022:2390	0	None	None	None	2022-05-17 12:45:31 UTC

Internal Links: 1728530

Description Katerina Koukiou 2019-08-09 14:15:01 UTC

Description of problem:
The same issue like https://bugzilla.redhat.com/show_bug.cgi?id=1728530
but the stack strace implies that libvirtd crashes in different part of the code, so creating a second bug report for this crash.

Version-Release number of selected component (if applicable):
libvirt-daemon-4.5.0-31

How reproducible:
Non deterministically reproduceble

Steps to Reproduce:
1. The code which the test is triggering is calling virDomainDestroy API, and afterwards virDomainUndefine

I haven't managed to reproduce outside of our CI, but I installed debuginfo packages and got a coredump file, so I guess it's enough for you to debug the crash.

Actual results:
(gdb)  thread apply all bt full

Thread 17 (Thread 0x7f139d2f2700 (LWP 1929)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833fa240a8, 
    m=m@entry=0x55833fa24080) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833fa1f9c0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833fa24040
        cond = 0x55833fa240a8
        priority = false
        curWorkers = 0x55833fa24120
        maxLimit = 0x55833fa24108
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833fa1f9c0}
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 16 (Thread 0x7f139daf3700 (LWP 1928)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833fa240a8, 
    m=m@entry=0x55833fa24080) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9fd390) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833fa24040
        cond = 0x55833fa240a8
        priority = false
        curWorkers = 0x55833fa24120
        maxLimit = 0x55833fa24108
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
--Type <RET> for more, q to quit, c to continue without paging--
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9fd390}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7f139e2f4700 (LWP 1927)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833fa240a8, 
    m=m@entry=0x55833fa24080) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9fa780) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833fa24040
        cond = 0x55833fa240a8
        priority = false
        curWorkers = 0x55833fa24120
--Type <RET> for more, q to quit, c to continue without paging--
        maxLimit = 0x55833fa24108
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9fa780}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7f139eaf5700 (LWP 1926)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833fa240a8, 
    m=m@entry=0x55833fa24080) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833fa24a40) at util/virthreadpool.c:124
        data = 0x0
--Type <RET> for more, q to quit, c to continue without paging--
        pool = 0x55833fa24040
        cond = 0x55833fa240a8
        priority = false
        curWorkers = 0x55833fa24120
        maxLimit = 0x55833fa24108
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833fa24a40}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7f139f2f6700 (LWP 1925)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833fa240a8, 
    m=m@entry=0x55833fa24080) at util/virthread.c:154
--Type <RET> for more, q to quit, c to continue without paging--
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9fc7f0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833fa24040
        cond = 0x55833fa240a8
        priority = false
        curWorkers = 0x55833fa24120
        maxLimit = 0x55833fa24108
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9fc7f0}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7f13d1940700 (LWP 1924)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdc68, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295af4 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9a4be0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdc68
        priority = true
        curWorkers = 0x55833f9bdc58
        maxLimit = 0x55833f9bdc50
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9a4be0}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
No symbol table info available.

Thread 11 (Thread 0x7f13d2141700 (LWP 1923)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdc68, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295af4 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9a4d80) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdc68
        priority = true
        curWorkers = 0x55833f9bdc58
        maxLimit = 0x55833f9bdc50
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
--Type <RET> for more, q to quit, c to continue without paging--
          opaque = 0x55833f9a4d80}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7f13d2942700 (LWP 1922)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdc68, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295af4 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9a47c0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdc68
        priority = true
        curWorkers = 0x55833f9bdc58
        maxLimit = 0x55833f9bdc50
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9a47c0}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7f13d3143700 (LWP 1921)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdc68, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295af4 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9a4230) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdc68
        priority = true
--Type <RET> for more, q to quit, c to continue without paging--
        curWorkers = 0x55833f9bdc58
        maxLimit = 0x55833f9bdc50
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9a4230}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7f13d3944700 (LWP 1920)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdc68, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295af4 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9a43b0) at util/virthreadpool.c:124
--Type <RET> for more, q to quit, c to continue without paging--
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdc68
        priority = true
        curWorkers = 0x55833f9bdc58
        maxLimit = 0x55833f9bdc50
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9a43b0}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f13d4145700 (LWP 1919)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdbc8, 
--Type <RET> for more, q to quit, c to continue without paging--
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9a33c0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdbc8
        priority = false
        curWorkers = 0x55833f9bdc40
        maxLimit = 0x55833f9bdc28
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9a33c0}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f13d4946700 (LWP 1918)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdbc8, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9aa250) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdbc8
        priority = false
        curWorkers = 0x55833f9bdc40
        maxLimit = 0x55833f9bdc28
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9aa250}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f13d5147700 (LWP 1917)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdbc8, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9aa310) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdbc8
        priority = false
        curWorkers = 0x55833f9bdc40
        maxLimit = 0x55833f9bdc28
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
--Type <RET> for more, q to quit, c to continue without paging--
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9aa310}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f13d5948700 (LWP 1916)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdbc8, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9aa3d0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdbc8
        priority = false
        curWorkers = 0x55833f9bdc40
        maxLimit = 0x55833f9bdc28
        job = 0x0
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9aa3d0}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f13d6149700 (LWP 1915)):
#0  0x00007f13dd0f547c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e029500a in virCondWait (c=c@entry=0x55833f9bdbc8, 
    m=m@entry=0x55833f9bdba0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f13e0295b43 in virThreadPoolWorker (
    opaque=opaque@entry=0x55833f9a9dd0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55833f9bdb60
        cond = 0x55833f9bdbc8
--Type <RET> for more, q to quit, c to continue without paging--
        priority = false
        curWorkers = 0x55833f9bdc40
        maxLimit = 0x55833f9bdc28
        job = 0x0
#3  0x00007f13e0294d6c in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f13e0295960 <virThreadPoolWorker>, 
          funcName = 0x7f13e04af6ab "virNetServerHandleJob", worker = true, 
          opaque = 0x55833f9a9dd0}
#4  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f13e0d8fb80 (LWP 1914)):
#0  0x00007f139fa0c3c0 in boost::aligned_storage<16ul, 8ul>::~aligned_storage()
    () from /usr/lib64/ceph/libceph-common.so.0
No symbol table info available.
#1  0x00007f13dcd5e06c in __run_exit_handlers () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f13dcd5e1a0 in exit () from /lib64/libc.so.6
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x00007f13dcd4787a in __libc_start_main () from /lib64/libc.so.6
No symbol table info available.
#4  0x000055833ea86a4e in _start () at remote/remote_daemon.c:396
No symbol table info available.

Thread 1 (Thread 0x7f139caf1700 (LWP 8444)):
#0  0x00007f13dd0f1aa4 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f13e0294f19 in virMutexLock (m=<optimized out>)
    at util/virthread.c:89
No locals.
#2  0x00007f13ac1100a2 in qemuDriverLock (driver=0x0) at qemu/qemu_conf.c:1065
No locals.
#3  virQEMUDriverGetConfig (driver=0x0) at qemu/qemu_conf.c:1065
        conf = <optimized out>
#4  0x00007f13ac154ac0 in qemuStateStop () at qemu/qemu_driver.c:986
        ret = -1
        conn = <optimized out>
        numDomains = 0
        i = <optimized out>
        state = 0
        domains = 0x0
        flags = 0x0
--Type <RET> for more, q to quit, c to continue without paging--
        cfg = <optimized out>
        __FUNCTION__ = "qemuStateStop"
#5  0x00007f13e041656f in virStateStop () at libvirt.c:730
        i = 7
        ret = 0
#6  0x000055833ea86d31 in daemonStopWorker (opaque=0x55833f9bcf00)
    at remote/remote_daemon.c:732
        __x = <optimized out>
        dmn = 0x55833f9bcf00
        __func__ = "daemonStopWorker"
#7  0x00007f13e0294d9a in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x55833ea86cf0 <daemonStopWorker>, 
          funcName = 0x55833eab9404 "daemonStopWorker", worker = false, 
          opaque = 0x55833f9bcf00}
#8  0x00007f13dd0ef2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007f13dce20133 in clone () from /lib64/libc.so.6
No symbol table info available.

Comment 1 Ján Tomko 2019-08-09 15:39:37 UTC

daemonStopWorker is called after we receive a signal that the system is going to be shut down:

commit b88b171731b6c00cd04c7ffd79b04ccdaf9502c9
Author:     Alexander Larsson <alexl>
CommitDate: 2012-12-04 12:14:04 +0000

    Shut down session libvirtd cleanly on host shutdown/user logout
    
    When the session dies or when the system is going to be shut down
    we issue a virStateStop() call to instruct drivers to prepare to
    be stopped. This will remove any previously acquire inhibitions.
    
    Signed-off-by: Daniel P. Berrange <berrange>

git describe: v1.0.0-225-gb88b171731 contains: v1.0.1-rc1~92

However there is a chance that the worker gets executed after we already cleaned up the driver object

Comment 6 Michal Privoznik 2020-11-24 16:55:39 UTC

Fix pushed upstream:

commit a42b46dd7db2cafe77010bfae55f2e4631a26844
Author:     Michal Prívozník <mprivozn>
AuthorDate: Fri Nov 13 10:56:59 2020 +0100
Commit:     Michal Prívozník <mprivozn>
CommitDate: Tue Nov 24 17:52:54 2020 +0100

    virnetdaemon: Wait for "daemon-stop" thread to finish before quitting
    
    When the host is shutting down then we get PrepareForShutdown
    signal on DBus to which we react by creating a thread which
    runs virStateStop() and thus qemuStateStop(). But if scheduling
    the thread is delayed just a but it may happen that we receive
    SIGTERM (sent by systemd) to which we respond by quitting our
    event loop and cleaning up everything (including drivers). And
    only after that the thread gets to run only to find qemu_driver
    being NULL.
    
    What we can do is to delay exiting event loop and join the thread
    that's executing virStateStop(). If the join doesn't happen in
    given timeout (currently 30 seconds) then libvirtd shuts down
    forcefully anyways (see virNetDaemonRun()).
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1895359
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1739564
    
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Ján Tomko <jtomko>

v6.9.0-373-ga42b46dd7d

Comment 7 Jaroslav Suchanek 2020-12-04 20:58:59 UTC

Katerina, do you require this to be backported to rhel-8.4? Otherwise it land in rhel-8.6.

Comment 10 John Ferlan 2021-09-14 19:31:07 UTC

Bulk update: Move RHEL8 bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

Comment 13 yafu 2021-10-14 03:34:36 UTC

Reproduced with libvirt-daemon-6.10.0-1.el8.x86_64. 

Steps are the same as https://bugzilla.redhat.com/show_bug.cgi?id=1895359#c18.



Verified with libvirt-7.8.0-1.el9.x86_64.

Steps are the same as https://bugzilla.redhat.com/show_bug.cgi?id=1895359#c19 - https://bugzilla.redhat.com/show_bug.cgi?id=1895359#c21.

Comment 16 errata-xmlrpc 2022-05-17 12:45:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390

Note You need to log in before you can comment on or make changes to this bug.