RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1845468 - [backport] libvirt crashes when stopping daemon after virsh command
Summary: [backport] libvirt crashes when stopping daemon after virsh command
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: libvirt
Version: 8.3
Hardware: All
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 8.0
Assignee: Michal Privoznik
QA Contact: yafu
URL:
Whiteboard:
Depends On: 1836865 1949342
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-09 10:24 UTC by smitterl
Modified: 2022-05-10 13:23 UTC (History)
10 users (show)

Fixed In Version: libvirt-7.10.0-1.module+el8.6.0+13502+4f24a11d
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1836865
Environment:
Last Closed: 2022-05-10 13:18:39 UTC
Type: Bug
Target Upstream Version: 7.10.0
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2022:1759 0 None None None 2022-05-10 13:19:39 UTC

Comment 2 Peter Krempa 2020-07-13 08:05:35 UTC 
Please note that even if libvirtd crashes when it's being stopped, this has no impact on it being started back up and working afterwards, so the crashes can be ignored until they are fixed.
Comment 3 Jaroslav Suchanek 2020-09-11 09:16:08 UTC 
There were number of changes/improvements in the deamon stopping code,

commit 9b648cb83eeb86293ed3471098fa21f6c44b5e31
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 11:13:44 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:34:00 2020 +0300

    util: remove unused virThreadPoolNew macro
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 61845fbf4226118a0cf868974cc100b852e0a282
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 11:13:12 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    rpc: cleanup virNetDaemonClose method
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 399039a6b1d07610294cc810ac7a01b51ff6b5cf
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 11:12:26 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    qemu: implement driver's shutdown/shutdown wait methods
    
    On shutdown we just stop accepting new jobs for worker thread so that on
    shutdown wait we can exit worker thread faster. Yes we basically stop
    processing of events for VMs but we are going to do so anyway in case of daemon
    shutdown.
    
    At the same time synchronous event processing that some API calls may require
    are still possible as per VM event loop is still running and we don't need
    worker thread for synchronous event processing.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 860a999802d3c82538373bb3f314f92a2e258754
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 11:02:59 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    qemu: avoid deadlock in qemuDomainObjStopWorker
    
    We are dropping the only reference here so that the event loop thread
    is going to be exited synchronously. In order to avoid deadlocks we
    need to unlock the VM so that any handler being called can finish
    execution and thus even loop thread be finished too.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>
    Reviewed-by: Daniel P. Berrangé <berrange>

commit f4fc3db9204407874181117085756c9ced78adad
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 10:23:00 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    vireventthread: exit thread synchronously on finalize
    
    It it useful to be sure no thread is running after we drop all references to
    virEventThread. Otherwise in order to avoid crashes we need to synchronize some
[jarda@jsrh libvirt]$ git log --author=nshirokovskiy
commit 9b648cb83eeb86293ed3471098fa21f6c44b5e31
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 11:13:44 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:34:00 2020 +0300

    util: remove unused virThreadPoolNew macro
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 61845fbf4226118a0cf868974cc100b852e0a282
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 11:13:12 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    rpc: cleanup virNetDaemonClose method
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 399039a6b1d07610294cc810ac7a01b51ff6b5cf
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 11:12:26 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    qemu: implement driver's shutdown/shutdown wait methods
    
    On shutdown we just stop accepting new jobs for worker thread so that on
    shutdown wait we can exit worker thread faster. Yes we basically stop
    processing of events for VMs but we are going to do so anyway in case of daemon
    shutdown.
    
    At the same time synchronous event processing that some API calls may require
    are still possible as per VM event loop is still running and we don't need
    worker thread for synchronous event processing.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 860a999802d3c82538373bb3f314f92a2e258754
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 11:02:59 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    qemu: avoid deadlock in qemuDomainObjStopWorker
    
    We are dropping the only reference here so that the event loop thread
    is going to be exited synchronously. In order to avoid deadlocks we
    need to unlock the VM so that any handler being called can finish
    execution and thus even loop thread be finished too.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>
    Reviewed-by: Daniel P. Berrangé <berrange>

commit f4fc3db9204407874181117085756c9ced78adad
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 10:23:00 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    vireventthread: exit thread synchronously on finalize
    
    It it useful to be sure no thread is running after we drop all references to
    virEventThread. Otherwise in order to avoid crashes we need to synchronize some
    other way or we make extra references in event handler callbacks to all the
    object in use. And some of them are not prepared to be refcounted.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>
    Reviewed-by: Daniel P. Berrangé <berrange>

commit 5c0cd375d1d1659ae3a5db0ce3e26e5570123dff
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 10:10:26 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    qemu: don't shutdown event thread in monitor EOF callback
    
    This hunk was introduced in [1] in order to avoid loosing
    events from monitor on stopping qemu process. But as explained
    in [2] on destroy we won't get neither EOF nor any other
    events as monitor is just closed. In case of crash/shutdown
    we won't get any more events as well and qemuDomainObjStopWorker
    will be called by qemuProcessStop eventually. Thus let's
    remove qemuDomainObjStopWorker from qemuProcessHandleMonitorEOF
    as it is not useful anymore.
    
    [1] e6afacb0f: qemu: start/stop an event loop thread for domains
    [2] d2954c072: qemu: ensure domain event thread is always stopped
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>
    Reviewed-by: Daniel P. Berrangé <berrange>

commit 94e45d1042e21e03a15ce993f90fbef626f1ae41
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 09:53:04 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    rpc: finish all threads before exiting main loop
    
    Currently we have issues like [1] on libvirtd shutdown as we cleanup while RPC
    and other threads are still running. Let's finish all threads other then main
    before cleanup.
    
    The approach to finish threads is suggested in [2]. In order to finish RPC
    threads serving API calls we let the event loop run but stop accepting new API
    calls and block processing any pending API calls. We also inform all drivers of
    shutdown so they can prepare for shutdown too. Then we wait for all RPC threads
    and driver's background thread to finish. If finishing takes more then 15s we
    just exit as we can't safely cleanup in time.
    
    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1828207
    [2] https://www.redhat.com/archives/libvir-list/2020-April/msg01328.html
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>
    Reviewed-by: Daniel P. Berrangé <berrange>

commit b776dfa8e881c868dc554c5c245f15c49332ce80
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 09:50:25 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    rpc: add shutdown facilities to netserver
    
    virNetServerClose and virNetServerShutdownWait are used to start net server
    threads shutdown and wait net server threads to actually finish respectively
    during net daemon shutdown procedure.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 0f38dedd8929dcb1473fc64773be4b941526ee1d
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 23 09:43:46 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    rpc: add virNetDaemonSetShutdownCallbacks
    
    The function is used to set shutdown prepare and wait callbacks. Prepare
    callback is used to inform other threads of the daemon that the daemon will be
    closed soon so that they can start to shutdown. Wait callback is used to wait
    for other threads to actually finish.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 1eae52b9f1f2c0232d14e0effa47e8e6e5cce28d
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 10:59:33 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:58 2020 +0300

    rpc: don't unref service ref on socket behalf twice
    
    Second unref was added in [1]. We don't need it actually as
    we pass free callback to virNetSocketAddIOCallback thus
    when we call virNetSocketRemoveIOCallback the extra ref for
    callback will be dropped without extra efforts.
    
    [1] 355d8f470f9: virNetServerServiceClose: Don't leak sockets
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 255437eeb710d8135136af11b37ceae674d483ce
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 10:58:02 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:58 2020 +0300

    util: add stop/drain functions to thread pool
    
    Stop just send signal for threads to exit when they finish with
    current task. Drain waits when all threads will finish.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit 018e213f5d1bbf5a68b7b7d46c8bacec06d97d49
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Fri Jul 10 14:36:54 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:58 2020 +0300

    util: always initialize priority condition
    
    Even if we have no priority threads on pool creation we can add them thru
    virThreadPoolSetParameters later.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit c5bf40bfa6afa4ec27052120e268a55213709ca4
Author:     Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

commit c5bf40bfa6afa4ec27052120e268a55213709ca4
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 10:46:34 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:58 2020 +0300

    libvirt: add stateShutdownPrepare/stateShutdownWait to drivers
    
    stateShutdownPrepare is supposed to inform driver that it will be closed soon
    so that the driver can prepare and finish all background threads quickly on
    stateShutdownWait call.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

Would you be able to re-test this with current master? Thanks.
Comment 4 smitterl 2020-09-14 12:01:25 UTC 
Couldn't reproduce with latest master.

Reproduced with:
libvirt-daemon-6.6.0-4.module+el8.3.0+7883+3d717aa8.s390x

How often: 8/10
Command: for i in $(seq 10); do dmesg -C; systemctl start libvirtd; virsh list; systemctl stop libvirtd; dmesg; sleep 2; done

Verified with:
libvirt@master HEAD@8c16f81eb97bbd576a009f64f13773200171704b (Thu Sep 10 15:24:02 2020 +0200)
#virsh --version
6.8.0

How often: 0/50
Command: for i in $(seq 50); do dmesg -C; systemctl start libvirtd; virsh list; systemctl stop libvirtd; dmesg; sleep 2; done
Comment 6 Michal Privoznik 2021-08-26 12:31:27 UTC 
Moving to POST per comments 3 and 4.
Comment 9 yafu 2021-10-26 08:09:32 UTC 
Hi Michal,

I met another libvirtd crash while trying to verify this bug. Could you help to check it please? Thanks in advance.

Test steps:
1.# for i in {1..10}; do systemctl restart libvirtd; virsh list ; systemctl restart libvirtd ; sleep 3; done

2.# coredumpctl debug
           PID: 1656719 (libvirtd)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2021-10-26 03:59:46 EDT (4min 49s ago)
  Command Line: /usr/sbin/libvirtd --timeout 120
    Executable: /usr/sbin/libvirtd
 Control Group: /
         Slice: -.slice
       Boot ID: 914f188bc9df437ca6ca52fcf196857d
    Machine ID: a7d52dfbef0b4c86a4823ec28ac04cf1
      Hostname: dell-per730-30.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.libvirtd.0.914f188bc9df437ca6ca52fcf196857d.1656719.1635235186000000.lz4
       Message: Process 1656719 (libvirtd) of user 0 dumped core.
                
                Stack trace of thread 1656794:
                #0  0x00007f836a26ac24 virEventThreadGetContext (libvirt.so.0)
                #1  0x00007f832090e547 qemuConnectAgent (libvirt_driver_qemu.so)
                #2  0x00007f832091aa7f qemuProcessReconnect (libvirt_driver_qemu.so)
                #3  0x00007f836a2be31b virThreadHelper (libvirt.so.0)
                #4  0x00007f83665b117a start_thread (libpthread.so.0)
                #5  0x00007f8368d5fdc3 __clone (libc.so.6)
                
                Stack trace of thread 1656719:
                #0  0x00007f8368cb3167 vfprintf (libc.so.6)
                #1  0x00007f8368d6fc8c __vasprintf_chk (libc.so.6)
                #2  0x00007f8369b03d7d g_vasprintf (libglib-2.0.so.0)
                #3  0x00007f8369adcf51 g_strdup_vprintf (libglib-2.0.so.0)
                #4  0x00007f836a2485ed vir_g_strdup_vprintf (libvirt.so.0)
                #5  0x00007f836a288717 virLogMessage (libvirt.so.0)
                #6  0x00007f836a2a1d8a virObjectUnref (libvirt.so.0)
                #7  0x00007f836a3e9f4c virSecurityStackClose (libvirt.so.0)
                #8  0x00007f836a3e64f7 virSecurityManagerDispose (libvirt.so.0)
                #9  0x00007f836a2a18de vir_object_finalize (libvirt.so.0)
                #10 0x00007f83698328a9 g_object_unref (libgobject-2.0.so.0)
                #11 0x00007f836a2a1d58 virObjectUnref (libvirt.so.0)
                #12 0x00007f832088b5dc qemuStateCleanup (libvirt_driver_qemu.so)
                #13 0x00007f836a4721d0 virStateCleanup (libvirt.so.0)
                #14 0x000055b37089866d main (libvirtd)
                #15 0x00007f8368c86493 __libc_start_main (libc.so.6)
                #16 0x000055b370898e7e _start (libvirtd)
                
                Stack trace of thread 1656736:
                #0  0x00007f8368d54a41 __poll (libc.so.6)
                #1  0x00007f8369abdc86 g_main_context_iterate.isra.21 (libglib-2.0.so.0)
                #2  0x00007f8369abddb0 g_main_context_iteration (libglib-2.0.so.0)
                #3  0x00007f8369abde01 glib_worker_main (libglib-2.0.so.0)
                #4  0x00007f8369ae5fca g_thread_proxy (libglib-2.0.so.0)
                #5  0x00007f83665b117a start_thread (libpthread.so.0)
                #6  0x00007f8368d5fdc3 __clone (libc.so.6)
                
                Stack trace of thread 1656788:
                #0  0x00007f8368d506eb llseek.5 (libc.so.6)
                #1  0x00007f836a2a26a4 virPCIDeviceRead.isra.1 (libvirt.so.0)
                #2  0x00007f836a2a2790 virPCIDeviceFindCapabilityOffset (libvirt.so.0)
                #3  0x00007f836a2a44ed virPCIDeviceInit (libvirt.so.0)
                #4  0x00007f836a2a6e90 virPCIDeviceIsPCIExpress (libvirt.so.0)
                #5  0x00007f8321231f6d udevAddOneDevice (libvirt_driver_nodedev.so)
                #6  0x00007f83212326ee nodeStateInitializeEnumerate (libvirt_driver_nodedev.so)
                #7  0x00007f836a2be31b virThreadHelper (libvirt.so.0)
                #8  0x00007f83665b117a start_thread (libpthread.so.0)
                #9  0x00007f8368d5fdc3 __clone (libc.so.6)
                
                Stack trace of thread 1656738:
                #0  0x00007f8368d54a41 __poll (libc.so.6)
                #1  0x00007f8369abdc86 g_main_context_iterate.isra.21 (libglib-2.0.so.0)
                #2  0x00007f8369abe042 g_main_loop_run (libglib-2.0.so.0)
                #3  0x00007f836954a5da gdbus_shared_thread_func (libgio-2.0.so.0)
                #4  0x00007f8369ae5fca g_thread_proxy (libglib-2.0.so.0)
                #5  0x00007f83665b117a start_thread (libpthread.so.0)
                #6  0x00007f8368d5fdc3 __clone (libc.so.6)
                
                Stack trace of thread 1656787:
                #0  0x00007f83665b73fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f836a2be1ca virCondWait (libvirt.so.0)
                #2  0x00007f83212328ac udevEventHandleThread (libvirt_driver_nodedev.so)
                #3  0x00007f836a2be31b virThreadHelper (libvirt.so.0)
                #4  0x00007f83665b117a start_thread (libpthread.so.0)
                #5  0x00007f8368d5fdc3 __clone (libc.so.6)
                
                Stack trace of thread 1656793:
                #0  0x00007f83665b73fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f836a2be1ca virCondWait (libvirt.so.0)
                #2  0x00007f83208e5654 qemuMonitorSend (libvirt_driver_qemu.so)
                #3  0x00007f83208f4c35 qemuMonitorJSONCommandWithFd (libvirt_driver_qemu.so)
                #4  0x00007f83208ff658 qemuMonitorJSONGetObjectListPaths (libvirt_driver_qemu.so)
                #5  0x00007f8320901d4b qemuMonitorJSONGetDeviceAliases (libvirt_driver_qemu.so)
                #6  0x00007f832087a3e3 qemuDomainUpdateDeviceList (libvirt_driver_qemu.so)
                #7  0x00007f832091a6c7 qemuProcessReconnect (libvirt_driver_qemu.so)
                #8  0x00007f836a2be31b virThreadHelper (libvirt.so.0)
                #9  0x00007f83665b117a start_thread (libpthread.so.0)
                #10 0x00007f8368d5fdc3 __clone (libc.so.6)
                
                Stack trace of thread 1656792:
                #0  0x00007f83665b73fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f836a2be1ca virCondWait (libvirt.so.0)
                #2  0x00007f83208e5654 qemuMonitorSend (libvirt_driver_qemu.so)
                #3  0x00007f83208f4c35 qemuMonitorJSONCommandWithFd (libvirt_driver_qemu.so)
                #4  0x00007f83208ff658 qemuMonitorJSONGetObjectListPaths (libvirt_driver_qemu.so)
                #5  0x00007f8320901d4b qemuMonitorJSONGetDeviceAliases (libvirt_driver_qemu.so)
                #6  0x00007f832087a3e3 qemuDomainUpdateDeviceList (libvirt_driver_qemu.so)
                #7  0x00007f832091a6c7 qemuProcessReconnect (libvirt_driver_qemu.so)
                #8  0x00007f836a2be31b virThreadHelper (libvirt.so.0)
                #9  0x00007f83665b117a start_thread (libpthread.so.0)
                #10 0x00007f8368d5fdc3 __clone (libc.so.6)
Comment 10 yafu 2021-10-26 08:10:57 UTC 
I did the test with libvirt-7.8.0-1.module+el8.6.0+12982+5e169f40.x86_64 for the issue in comment #9.
Comment 11 Michal Privoznik 2021-10-26 12:38:33 UTC 
I'm sorry but from the stack trace it's unclear to me what is wrong. I suspect the second thread is the one that crashed, but it's not obvious why. Do you have coredump available please?
Comment 12 yafu 2021-10-27 05:45:57 UTC 
(In reply to Michal Privoznik from comment #11)
> I'm sorry but from the stack trace it's unclear to me what is wrong. I
> suspect the second thread is the one that crashed, but it's not obvious why.
> Do you have coredump available please?

Sorry, i did not install the debug pkgs for the backtrace in comment #9. How about this backtrace?

(gdb) t a a bt

Thread 8 (Thread 0x7f8305ffb700 (LWP 1656792)):
#0  0x00007f83665b73fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f836a2be1ca in virCondWait (c=c@entry=0x7f82e4005630, m=m@entry=0x7f82e4005608) at ../src/util/virthread.c:156
#2  0x00007f83208e5654 in qemuMonitorSend (mon=mon@entry=0x7f82e40055f0, msg=msg@entry=0x7f8305ffa780) at ../src/qemu/qemu_monitor.c:961
#3  0x00007f83208f4c35 in qemuMonitorJSONCommandWithFd (mon=0x7f82e40055f0, cmd=0x7f82f0000c60, scm_fd=-1, reply=0x7f8305ffa810) at ../src/qemu/qemu_monitor_json.c:327
#4  0x00007f83208ff658 in qemuMonitorJSONCommand (reply=0x7f8305ffa810, cmd=0x7f82f0000c60, mon=0x7f82e40055f0) at ../src/qemu/qemu_monitor_json.c:6283
#5  qemuMonitorJSONGetObjectListPaths (mon=0x7f82e40055f0, path=path@entry=0x7f832095c873 "/machine/peripheral", paths=paths@entry=0x7f8305ffa860) at ../src/qemu/qemu_monitor_json.c:6283
#6  0x00007f8320901d4b in qemuMonitorJSONGetDeviceAliases (mon=<optimized out>, aliases=aliases@entry=0x7f8305ffa890) at ../src/qemu/qemu_monitor_json.c:7421
#7  0x00007f83208eefc6 in qemuMonitorGetDeviceAliases (mon=<optimized out>, aliases=aliases@entry=0x7f8305ffa890) at ../src/qemu/qemu_monitor.c:3920
#8  0x00007f832087a3e3 in qemuDomainUpdateDeviceList (driver=driver@entry=0x7f8300147890, vm=vm@entry=0x7f83001d4020, asyncJob=asyncJob@entry=0) at ../src/qemu/qemu_domain.c:8163
#9  0x00007f832091a6c7 in qemuProcessUpdateDevices (vm=0x7f83001d4020, driver=0x7f8300147890) at ../src/qemu/qemu_process.c:3726
#10 qemuProcessReconnect (opaque=<optimized out>) at ../src/qemu/qemu_process.c:8719
#11 0x00007f836a2be31b in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#12 0x00007f83665b117a in start_thread () from /lib64/libpthread.so.0
#13 0x00007f8368d5fdc3 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f83057fa700 (LWP 1656793)):
#0  0x00007f83665b73fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f836a2be1ca in virCondWait (c=c@entry=0x7f82e40050b0, m=m@entry=0x7f82e4005088) at ../src/util/virthread.c:156
#2  0x00007f83208e5654 in qemuMonitorSend (mon=mon@entry=0x7f82e4005070, msg=msg@entry=0x7f83057f9780) at ../src/qemu/qemu_monitor.c:961
#3  0x00007f83208f4c35 in qemuMonitorJSONCommandWithFd (mon=0x7f82e4005070, cmd=0x7f82e4010340, scm_fd=-1, reply=0x7f83057f9810) at ../src/qemu/qemu_monitor_json.c:327
#4  0x00007f83208ff658 in qemuMonitorJSONCommand (reply=0x7f83057f9810, cmd=0x7f82e4010340, mon=0x7f82e4005070) at ../src/qemu/qemu_monitor_json.c:6283
#5  qemuMonitorJSONGetObjectListPaths (mon=0x7f82e4005070, path=path@entry=0x7f832095c873 "/machine/peripheral", paths=paths@entry=0x7f83057f9860) at ../src/qemu/qemu_monitor_json.c:6283
#6  0x00007f8320901d4b in qemuMonitorJSONGetDeviceAliases (mon=<optimized out>, aliases=aliases@entry=0x7f83057f9890) at ../src/qemu/qemu_monitor_json.c:7421
#7  0x00007f83208eefc6 in qemuMonitorGetDeviceAliases (mon=<optimized out>, aliases=aliases@entry=0x7f83057f9890) at ../src/qemu/qemu_monitor.c:3920
#8  0x00007f832087a3e3 in qemuDomainUpdateDeviceList (driver=driver@entry=0x7f8300147890, vm=vm@entry=0x7f83001d4110, asyncJob=asyncJob@entry=0) at ../src/qemu/qemu_domain.c:8163
#9  0x00007f832091a6c7 in qemuProcessUpdateDevices (vm=0x7f83001d4110, driver=0x7f8300147890) at ../src/qemu/qemu_process.c:3726
#10 qemuProcessReconnect (opaque=<optimized out>) at ../src/qemu/qemu_process.c:8719
#11 0x00007f836a2be31b in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#12 0x00007f83665b117a in start_thread () from /lib64/libpthread.so.0
#13 0x00007f8368d5fdc3 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f8306ffd700 (LWP 1656787)):
#0  0x00007f83665b73fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f836a2be1ca in virCondWait (c=c@entry=0x55b370dd26f8, m=m@entry=0x55b370dd26b8) at ../src/util/virthread.c:156
#2  0x00007f83212328ac in udevEventHandleThread (opaque=<optimized out>) at ../src/node_device/node_device_udev.c:1802
#3  0x00007f836a2be31b in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#4  0x00007f83665b117a in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8368d5fdc3 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f83077fe700 (LWP 1656738)):
#0  0x00007f8368d54a41 in poll () from /lib64/libc.so.6
#1  0x00007f8369abdc86 in g_main_context_iterate.isra () from /lib64/libglib-2.0.so.0
#2  0x00007f8369abe042 in g_main_loop_run () from /lib64/libglib-2.0.so.0
#3  0x00007f836954a5da in gdbus_shared_thread_func () from /lib64/libgio-2.0.so.0
#4  0x00007f8369ae5fca in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5  0x00007f83665b117a in start_thread () from /lib64/libpthread.so.0
#6  0x00007f8368d5fdc3 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f83067fc700 (LWP 1656788)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f8368d506eb in lseek64 () from /lib64/libc.so.6
#1  0x00007f836a2a26a4 in virPCIDeviceRead (cfgfd=cfgfd@entry=3, pos=pos@entry=6, buf=buf@entry=0x7f83067fb936 "", buflen=buflen@entry=2, dev=<optimized out>) at ../src/util/virpci.c:341
#2  0x00007f836a2a2790 in virPCIDeviceRead16 (pos=6, cfgfd=3, dev=<optimized out>) at ../src/util/virpci.c:540
#3  virPCIDeviceFindCapabilityOffset (dev=dev@entry=0x7f82ec081160, cfgfd=cfgfd@entry=3, capability=capability@entry=1, offset=offset@entry=0x7f82ec0811bc) at ../src/util/virpci.c:540
#4  0x00007f836a2a44ed in virPCIDeviceInit (dev=0x7f82ec081160, cfgfd=3) at ../src/util/virpci.c:995
#5  0x00007f836a2a6e90 in virPCIDeviceIsPCIExpress (dev=dev@entry=0x7f82ec081160) at ../src/util/virpci.c:2727
#6  0x00007f8321231f6d in udevProcessPCI (def=0x7f82ec04eca0, device=<optimized out>) at ../src/node_device/node_device_udev.c:434
#7  udevGetDeviceDetails (def=0x7f82ec04eca0, device=<optimized out>) at ../src/node_device/node_device_udev.c:1366
#8  udevAddOneDevice (device=<optimized out>) at ../src/node_device/node_device_udev.c:1524

#9  0x00007f83212326ee in udevProcessDeviceListEntry (list_entry=0x7f82ec01e6b0, udev=0x7f830010dda0) at ../src/node_device/node_device_udev.c:1594

#10 udevEnumerateDevices (udev=0x7f830010dda0) at ../src/node_device/node_device_udev.c:1648
#11 nodeStateInitializeEnumerate (opaque=0x7f830010dda0) at ../src/node_device/node_device_udev.c:1979
#12 0x00007f836a2be31b in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#13 0x00007f83665b117a in start_thread () from /lib64/libpthread.so.0
#14 0x00007f8368d5fdc3 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f8307fff700 (LWP 1656736)):
#0  0x00007f8368d54a41 in poll () from /lib64/libc.so.6
#1  0x00007f8369abdc86 in g_main_context_iterate.isra () from /lib64/libglib-2.0.so.0
#2  0x00007f8369abddb0 in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#3  0x00007f8369abde01 in glib_worker_main () from /lib64/libglib-2.0.so.0
#4  0x00007f8369ae5fca in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5  0x00007f83665b117a in start_thread () from /lib64/libpthread.so.0
#6  0x00007f8368d5fdc3 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f836aa08b00 (LWP 1656719)):
#0  0x00007f8368cb3167 in vfprintf () from /lib64/libc.so.6
#1  0x00007f8368d6fc8c in __vasprintf_chk () from /lib64/libc.so.6
#2  0x00007f8369b03d7d in g_vasprintf () from /lib64/libglib-2.0.so.0
#3  0x00007f8369adcf51 in g_strdup_vprintf () from /lib64/libglib-2.0.so.0
#4  0x00007f836a2485ed in vir_g_strdup_vprintf (msg=msg@entry=0x7f836a4c8ba9 "OBJECT_UNREF: obj=%p", args=args@entry=0x7ffefc860cf8) at ../src/util/glibcompat.c:209
#5  0x00007f836a288717 in virLogVMessage (vargs=0x7ffefc860cf8, fmt=0x7f836a4c8ba9 "OBJECT_UNREF: obj=%p", metadata=0x0, funcname=0x7f836a4c8d68 <__func__.19256> "virObjectUnref", linenr=380, 
    filename=0x7f836a4c8b2f "../src/util/virobject.c", priority=VIR_LOG_INFO, source=0x7f836a7ff4b0 <virLogSelf>) at ../src/util/virlog.c:536
#6  virLogMessage (source=source@entry=0x7f836a7ff4b0 <virLogSelf>, priority=priority@entry=VIR_LOG_INFO, filename=filename@entry=0x7f836a4c8b2f "../src/util/virobject.c", linenr=linenr@entry=380, 
    funcname=funcname@entry=0x7f836a4c8d68 <__func__.19256> "virObjectUnref", metadata=metadata@entry=0x0, fmt=0x7f836a4c8ba9 "OBJECT_UNREF: obj=%p") at ../src/util/virlog.c:635
#7  0x00007f836a2a1d8a in virObjectUnref (anyobj=0x7f8300128420) at ../src/util/virobject.c:380
#8  0x00007f836a3e9f4c in virSecurityStackClose (mgr=<optimized out>) at ../src/security/security_stack.c:96
#9  0x00007f836a3e64f7 in virSecurityManagerDispose (obj=0x7f8300128490) at ../src/security/security_manager.c:57
#10 0x00007f836a2a18de in vir_object_finalize (gobj=0x7f8300128490) at ../src/util/virobject.c:325
#11 0x00007f83698328a9 in g_object_unref () from /lib64/libgobject-2.0.so.0
#12 0x00007f836a2a1d58 in virObjectUnref (anyobj=0x7f8300128490) at ../src/util/virobject.c:379
#13 0x00007f832088b5dc in qemuStateCleanup () at ../src/qemu/qemu_driver.c:1085
#14 0x00007f836a4721d0 in virStateCleanup () at ../src/libvirt.c:733
#15 0x000055b37089866d in main (argc=<optimized out>, argv=<optimized out>) at ../src/remote/remote_daemon.c:1227

Thread 1 (Thread 0x7f8304ff9700 (LWP 1656794)):
#0  virEventThreadGetContext (evt=0x0) at ../src/util/vireventthread.c:194
#1  0x00007f832090e547 in qemuConnectAgent (driver=driver@entry=0x7f8300147890, vm=0x7f83001d4200) at ../src/qemu/qemu_process.c:243
#2  0x00007f832091aa7f in qemuProcessReconnect (opaque=<optimized out>) at ../src/qemu/qemu_process.c:8727
#3  0x00007f836a2be31b in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#4  0x00007f83665b117a in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007f8368d5fdc3 in clone () from /lib64/libc.so.6
Comment 13 Michal Privoznik 2021-10-29 08:28:49 UTC 
Thanks, I've posted patches here:

https://listman.redhat.com/archives/libvir-list/2021-October/msg01114.html
Comment 14 Michal Privoznik 2021-10-29 08:31:21 UTC 
Let me move back to ASSIGNED so that the patch can be picked up by rebase (once pushed upstream).
Comment 18 Michal Privoznik 2021-11-12 13:22:55 UTC 
Patches from comment 13 pushed upstream:

3640731ed5 qemuMonitorOpen: Rework domain object refcounting
e812213bc1 qemu_agent: Drop destroy callback
0a9cb29ba2 qemuAgentOpen: Rework domain object refcounting
108e131a3d qemu_agent: Rework domain object locking when opening agent

v7.9.0-130-g3640731ed5
Comment 19 yafu 2021-12-07 01:51:25 UTC 
Verified with libvirt-7.10.0-1.module+el8.6.0+13502+4f24a11d.x86_64.

Test steps:
1.#for i in {1..50}; do systemctl restart libvirtd; virsh list ; systemctl restart libvirtd ; sleep 5; done

2.Check coredump info:
#coredumpctl list
No coredumps found.
Comment 21 errata-xmlrpc 2022-05-10 13:18:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1759
Note You need to log in before you can comment on or make changes to this bug.