1728530 – libvirtd crashes non deterministically when trying to destroy a guest

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1728530 - libvirtd crashes non deterministically when trying to destroy a guest

Summary: libvirtd crashes non deterministically when trying to destroy a guest

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	8.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	8.0
Assignee:	Ján Tomko
QA Contact:	yafu
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-07-10 06:33 UTC by Katerina Koukiou
Modified:	2020-11-14 11:39 UTC (History)
CC List:	4 users (show)
Fixed In Version:	libvirt-4.5.0-31.el8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-11-05 20:51:02 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
coredump-libvirtd.lz4 (4.39 MB, application/x-lz4) 2019-07-10 06:33 UTC, Katerina Koukiou	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1739564	0	unspecified	CLOSED	libvirtd crashes non deterministically on shutdown	2022-05-17 13:28:37 UTC
Red Hat Product Errata	RHSA-2019:3345	0	None	None	None	2019-11-05 20:51:32 UTC

Description Katerina Koukiou 2019-07-10 06:33:53 UTC

Created attachment 1589001 [details]
coredump-libvirtd.lz4

Description of problem:

libvirtd rarely crashes when trying to destroy a domain.
I have noticed it only in RHEL8-1 when running CI tests for cockpit-machines.

Version-Release number of selected component (if applicable):
libvirt-daemon-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64

How reproducible:
Rarely

Steps to Reproduce:
1. The code which the test is triggering is calling virDomainDestroy API, and afterwards virDomainUndefine

I haven't managed to reproduce outside of our CI, but I installed debuginfo packages and got a coredump file, so I guess it's enough for you to debug the crash.

Find in attachment the core dump file.

Bellow the bt from gdb:

(gdb) thread apply all bt full

Thread 17 (Thread 0x7f8615454380 (LWP 1649)):
#0  0x00007f86114e11e1 in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f86148f7ff5 in poll (__timeout=4990, __nfds=11,
    __fds=<optimized out>) at /usr/include/bits/poll2.h:46
No locals.
#2  virEventPollRunOnce () at util/vireventpoll.c:641
        fds = 0x55d4de1b0ba0
        ret = <optimized out>
        timeout = <optimized out>
        nfds = 11
        __func__ = "virEventPollRunOnce"
        __FUNCTION__ = "virEventPollRunOnce"
#3  0x00007f86148f6b75 in virEventRunDefaultImpl () at util/virevent.c:327
        __func__ = "virEventRunDefaultImpl"
#4  0x00007f8614a27e8d in virNetDaemonRun (dmn=0x55d4ddffb760)
    at rpc/virnetdaemon.c:850
        timerid = -1
        timerActive = false
        __FUNCTION__ = "virNetDaemonRun"
        __func__ = "virNetDaemonRun"
#5  0x000055d4dc1a7773 in main (argc=<optimized out>, argv=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
    at remote/remote_daemon.c:1461
        dmn = 0x55d4ddffb760
        srv = 0x55d4ddffbaa0
        srvAdm = 0x55d4de062510
        adminProgram = 0x55d4de034ce0
        lxcProgram = 0x55d4de039fa0
        remote_config_file = 0x55d4ddfefcd0 "/etc/libvirt/libvirtd.conf"
        statuswrite = -1
        ret = 1
        pid_file_fd = 3
        pid_file = 0x55d4de007100 "/var/run/libvirtd.pid"
        sock_file = 0x55d4ddffad70 "/var/run/libvirt/libvirt-sock"
        sock_file_ro = 0x55d4ddff0820 "/var/run/libvirt/libvirt-sock-ro"
        sock_file_adm = 0x55d4ddffb310 "/var/run/libvirt/libvirt-admin-sock"
        timeout = -1
        verbose = 0
        godaemon = 0
        ipsock = 0
        config = 0x55d4ddff6620
        privileged = <optimized out>
        implicit_conf = <optimized out>
        run_dir = 0x55d4de007150 "/var/run/libvirt"
        old_umask = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        opts = {{name = 0x55d4dc1db538 "verbose", has_arg = 0,
            flag = 0x7ffcda2979e8, val = 118}, {
            name = 0x55d4dc1db540 "daemon", has_arg = 0,
            flag = 0x7ffcda2979ec, val = 100}, {
            name = 0x55d4dc1db547 "listen", has_arg = 0,
            flag = 0x7ffcda2979f0, val = 108}, {
            name = 0x55d4dc1db621 "config", has_arg = 1, flag = 0x0,
            val = 102}, {name = 0x55d4dc1db59d "timeout", has_arg = 1,
            flag = 0x0, val = 116}, {name = 0x55d4dc1db54e "pid-file",
            has_arg = 1, flag = 0x0, val = 112}, {
            name = 0x55d4dc1db557 "version", has_arg = 0, flag = 0x0,
            val = 86}, {name = 0x55d4dc1db55f "help", has_arg = 0, flag = 0x0,
            val = 104}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
        __func__ = "main"

Thread 16 (Thread 0x7f85fca6b700 (LWP 1653)):
#0  0x00007f86117c0475 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f86149559f9 in virRWLockWrite (m=<optimized out>)
    at util/virthread.c:122
No locals.
#2  0x00007f861493009f in virObjectRWLockWrite (anyobj=<optimized out>)
    at util/virobject.c:487
--Type <RET> for more, q to quit, c to continue without paging--
        obj = <optimized out>
#3  0x00007f86149b88cc in virDomainObjListRemove (doms=0x7f85d00f7250,
    dom=0x55d4de073620) at conf/virdomainobjlist.c:398
No locals.
#4  0x00007f85cb4605a2 in qemuDomainRemoveInactive (vm=0x55d4de073620,
    driver=0x7f85d00f36b0) at qemu/qemu_domain.c:8298
        snapDir = 0x0
        cfg = 0x7f85d00f7850
        snapDir = <optimized out>
        cfg = <optimized out>
        __func__ = "qemuDomainRemoveInactive"
        __FUNCTION__ = "qemuDomainRemoveInactive"
#5  qemuDomainRemoveInactive (driver=driver@entry=0x7f85d00f36b0,
    vm=0x55d4de073620) at qemu/qemu_domain.c:8269
        snapDir = <optimized out>
        cfg = <optimized out>
        __func__ = "qemuDomainRemoveInactive"
        __FUNCTION__ = "qemuDomainRemoveInactive"
#6  0x00007f85cb4e3f7a in qemuDomainDestroyFlags (dom=<optimized out>,
    flags=<optimized out>) at qemu/qemu_driver.c:2215
        driver = 0x7f85d00f36b0
        vm = 0x55d4de073620
        ret = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        event = <optimized out>
        priv = <optimized out>
        stopFlags = <optimized out>
        state = <optimized out>
        reason = 1
        starting = false
        __FUNCTION__ = "qemuDomainDestroyFlags"
        __func__ = "qemuDomainDestroyFlags"
#7  0x00007f8614ad9289 in virDomainDestroyFlags (
    domain=domain@entry=0x7f85d8007190, flags=0) at libvirt-domain.c:535
        ret = <optimized out>
        conn = 0x55d4de063120
        __func__ = "virDomainDestroyFlags"
        __FUNCTION__ = "virDomainDestroyFlags"
#8  0x000055d4dc1bf4e6 in remoteDispatchDomainDestroyFlags (
    server=0x55d4ddffbaa0, msg=0x55d4de0563e0, args=0x7f85d80039f0,
    rerr=0x7f85fca6a960, client=<optimized out>)
    at remote/remote_daemon_dispatch_stubs.h:4883
        rv = -1
        dom = 0x7f85d8007190
        priv = <optimized out>
        rv = <optimized out>
        dom = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        priv = <optimized out>
        __FUNCTION__ = "remoteDispatchDomainDestroyFlags"
#9  remoteDispatchDomainDestroyFlagsHelper (server=0x55d4ddffbaa0,
    client=<optimized out>, msg=0x55d4de0563e0, rerr=0x7f85fca6a960,
    args=0x7f85d80039f0, ret=0x7f85d8004c00)
    at remote/remote_daemon_dispatch_stubs.h:4859
        rv = <optimized out>
        __func__ = "remoteDispatchDomainDestroyFlagsHelper"
#10 0x00007f8614a22174 in virNetServerProgramDispatchCall (msg=0x55d4de0563e0,
    client=0x55d4de05f830, server=0x55d4ddffbaa0, prog=0x55d4de037cd0)
    at rpc/virnetserverprogram.c:437
        ret = 0x7f85d8004c00 ""
        rv = -1
        i = <optimized out>
        identity = 0x7f85e0000db0
        arg = 0x7f85d80039f0 "\340\036"
        dispatcher = 0x55d4dc402de0 <remoteProcs+11232>
        rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0,
          str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}
        arg = <optimized out>
        ret = <optimized out>
        rv = <optimized out>
        dispatcher = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        rerr = <optimized out>
        i = <optimized out>
        identity = <optimized out>
        __FUNCTION__ = "virNetServerProgramDispatchCall"
#11 virNetServerProgramDispatch (prog=0x55d4de037cd0,
    server=server@entry=0x55d4ddffbaa0, client=0x55d4de05f830,
    msg=0x55d4de0563e0) at rpc/virnetserverprogram.c:304
        rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0,
          str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}
        __func__ = "virNetServerProgramDispatch"
        __FUNCTION__ = "virNetServerProgramDispatch"
#12 0x00007f8614a2862c in virNetServerProcessMsg (msg=<optimized out>,
    prog=<optimized out>, client=<optimized out>, srv=0x55d4ddffbaa0)
    at rpc/virnetserver.c:143
        ret = -1
        ret = <optimized out>
        __func__ = "virNetServerProcessMsg"
#13 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x55d4ddffbaa0)
    at rpc/virnetserver.c:164
        srv = 0x55d4ddffbaa0
        job = 0x55d4de1a9a20
        __func__ = "virNetServerHandleJob"
#14 0x00007f86149564d0 in virThreadPoolWorker (
--Type <RET> for more, q to quit, c to continue without paging--
    opaque=opaque@entry=0x55d4de005f50) at util/virthreadpool.c:167
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbbf8
        priority = false
        curWorkers = 0x55d4ddffbc70
        maxLimit = 0x55d4ddffbc58
        job = 0x55d4de091ed0
#15 0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de005f50}
#16 0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#17 0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7f85c93d2700 (LWP 1664)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4de05f288,
    m=m@entry=0x55d4de05f260) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de037af0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4de05f220
        cond = 0x55d4de05f288
        priority = false
        curWorkers = 0x55d4de05f300
        maxLimit = 0x55d4de05f2e8
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de037af0}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

--Type <RET> for more, q to quit, c to continue without paging--
Thread 14 (Thread 0x7f85c9bd3700 (LWP 1663)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4de05f288,
    m=m@entry=0x55d4de05f260) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de0381b0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4de05f220
        cond = 0x55d4de05f288
        priority = false
        curWorkers = 0x55d4de05f300
        maxLimit = 0x55d4de05f2e8
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de0381b0}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7f85ca3d4700 (LWP 1662)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4de05f288,
    m=m@entry=0x55d4de05f260) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de037c40) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4de05f220
        cond = 0x55d4de05f288
        priority = false
        curWorkers = 0x55d4de05f300
        maxLimit = 0x55d4de05f2e8
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
--Type <RET> for more, q to quit, c to continue without paging--
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de037c40}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7f85cabd5700 (LWP 1661)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4de05f288,
    m=m@entry=0x55d4de05f260) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de0389e0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4de05f220
        cond = 0x55d4de05f288
        priority = false
        curWorkers = 0x55d4de05f300
        maxLimit = 0x55d4de05f2e8
--Type <RET> for more, q to quit, c to continue without paging--
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de0389e0}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7f85cb3d6700 (LWP 1660)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4de05f288,
    m=m@entry=0x55d4de05f260) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de040c40) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4de05f220
--Type <RET> for more, q to quit, c to continue without paging--
        cond = 0x55d4de05f288
        priority = false
        curWorkers = 0x55d4de05f300
        maxLimit = 0x55d4de05f2e8
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de040c40}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7f85f9a65700 (LWP 1659)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbc98,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
        ret = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x00007f8614956564 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de005a90) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbc98
        priority = true
        curWorkers = 0x55d4ddffbc88
        maxLimit = 0x55d4ddffbc80
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de005a90}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7f85fa266700 (LWP 1658)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbc98,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f8614956564 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de005c60) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbc98
        priority = true
        curWorkers = 0x55d4ddffbc88
        maxLimit = 0x55d4ddffbc80
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de005c60}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--

Thread 8 (Thread 0x7f85faa67700 (LWP 1657)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbc98,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f8614956564 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de005cb0) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbc98
        priority = true
        curWorkers = 0x55d4ddffbc88
        maxLimit = 0x55d4ddffbc80
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de005cb0}
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f85fba69700 (LWP 1655)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbc98,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f8614956564 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de005e30) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbc98
        priority = true
        curWorkers = 0x55d4ddffbc88
        maxLimit = 0x55d4ddffbc80
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
--Type <RET> for more, q to quit, c to continue without paging--
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de005e30}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f85fc26a700 (LWP 1654)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbbf8,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de005e80) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbbf8
        priority = false
        curWorkers = 0x55d4ddffbc70
--Type <RET> for more, q to quit, c to continue without paging--
        maxLimit = 0x55d4ddffbc58
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de005e80}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f85fd26c700 (LWP 1652)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbbf8,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de006030) at util/virthreadpool.c:124
        data = 0x0
--Type <RET> for more, q to quit, c to continue without paging--
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbbf8
        priority = false
        curWorkers = 0x55d4ddffbc70
        maxLimit = 0x55d4ddffbc58
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de006030}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f85fb268700 (LWP 1656)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbc98,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
--Type <RET> for more, q to quit, c to continue without paging--
        ret = <optimized out>
#2  0x00007f8614956564 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de005d00) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbc98
        priority = true
        curWorkers = 0x55d4ddffbc88
        maxLimit = 0x55d4ddffbc80
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de005d00}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f85fda6d700 (LWP 1651)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x55d4ddffbbf8,
    m=m@entry=0x55d4ddffbbd0) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f86149565b3 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de006170) at util/virthreadpool.c:124
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbbf8
        priority = false
        curWorkers = 0x55d4ddffbc70
        maxLimit = 0x55d4ddffbc58
        job = 0x0
#3  0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de006170}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
No symbol table info available.

Thread 2 (Thread 0x7f85c2fcb700 (LWP 1747)):
#0  0x00007f86117c147c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f8614955a7a in virCondWait (c=c@entry=0x7f85d00f0d40,
    m=m@entry=0x7f85d00f0d00) at util/virthread.c:154
        ret = <optimized out>
#2  0x00007f85cbdf20e4 in udevEventHandleThread (opaque=<optimized out>)
    at node_device/node_device_udev.c:1604
        priv = 0x7f85d00f0cf0
        device = <optimized out>
        __FUNCTION__ = "udevEventHandleThread"
#3  0x00007f861495580a in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
        args = 0x0
        local = {func = 0x7f85cbdf2090 <udevEventHandleThread>,
          funcName = 0x7f85cbdf979e "udevEventHandleThread", worker = false,
          opaque = 0x0}
#4  0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f86114ec463 in clone () from /lib64/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
No symbol table info available.

Thread 1 (Thread 0x7f85fe26e700 (LWP 1650)):
#0  0x00007f86149958bf in virDomainDefFree (def=<optimized out>,
    def@entry=0x7f85c4015a50) at conf/domain_conf.c:3007
        i = 0
#1  0x00007f85cb4df21e in qemuDomainDefineXMLFlags (conn=0x7f85d804cdf0,
    xml=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:7427
        driver = 0x7f85d00f36b0
        def = 0x7f85c4015a50
        oldDef = 0x0
        vm = 0x0
        dom = 0x0




        event = 0x0
        cfg = 0x7f85d00f7850
        caps = 0x7f85c4009d50
        parse_flags = <optimized out>
        __FUNCTION__ = "qemuDomainDefineXMLFlags"
        __func__ = "qemuDomainDefineXMLFlags"
#2  0x00007f8614ae97af in virDomainDefineXML (conn=0x7f85d804cdf0,
    xml=0x7f85c4025c30 "<domain type=\"qemu\">\n  <name>subVmTestCreate20</name>\n  <uuid>11067c4a-5370-42e0-a44d-e34b019cf9e2</uuid>\n  <metadata>\n    <libosinfo:libosinfo xmlns:libosinfo=\"http://libosinfo.org/xmlns/libvirt/doma"...)
--Type <RET> for more, q to quit, c to continue without paging--
    at libvirt-domain.c:6153
        ret = <optimized out>
        __func__ = "virDomainDefineXML"
        __FUNCTION__ = "virDomainDefineXML"

#3  0x000055d4dc1d4340 in remoteDispatchDomainDefineXML (
    server=0x55d4ddffbaa0, msg=0x55d4de0f5d30, args=0x7f85c401ce30,

    ret=0x7f85c401ce70, rerr=0x7f85fe26d960, client=0x55d4de080af0)
    at remote/remote_daemon_dispatch_stubs.h:4658
        rv = -1
        dom = 0x0
        priv = <optimized out>
        rv = <optimized out>
        dom = <optimized out>
        priv = <optimized out>
        __FUNCTION__ = "remoteDispatchDomainDefineXML"


#4  remoteDispatchDomainDefineXMLHelper (server=0x55d4ddffbaa0,
    client=0x55d4de080af0, msg=0x55d4de0f5d30, rerr=0x7f85fe26d960,
    args=0x7f85c401ce30, ret=0x7f85c401ce70)
    at remote/remote_daemon_dispatch_stubs.h:4636
        rv = <optimized out>
        __func__ = "remoteDispatchDomainDefineXMLHelper"
#5  0x00007f8614a22174 in virNetServerProgramDispatchCall (msg=0x55d4de0f5d30,


    client=0x55d4de080af0, server=0x55d4ddffbaa0, prog=0x55d4de037cd0)
--Type <RET> for more, q to quit, c to continue without paging--
    at rpc/virnetserverprogram.c:437

        ret = 0x7f85c401ce70 ""
        rv = -1
        i = <optimized out>

        identity = 0x55d4de094b50
        arg = 0x7f85c401ce30 "0\\\002ą\177"
        dispatcher = 0x55d4dc400410 <remoteProcs+528>
        rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0,
          str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}
        arg = <optimized out>
        ret = <optimized out>
        rv = <optimized out>
        dispatcher = <optimized out>
        rerr = <optimized out>
        i = <optimized out>
        identity = <optimized out>
        __FUNCTION__ = "virNetServerProgramDispatchCall"

#6  virNetServerProgramDispatch (prog=0x55d4de037cd0,

    server=server@entry=0x55d4ddffbaa0, client=0x55d4de080af0,
    msg=0x55d4de0f5d30) at rpc/virnetserverprogram.c:304
        rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0,
          str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}
        __func__ = "virNetServerProgramDispatch"
--Type <RET> for more, q to quit, c to continue without paging--
        __FUNCTION__ = "virNetServerProgramDispatch"
#7  0x00007f8614a2862c in virNetServerProcessMsg (msg=<optimized out>,
    prog=<optimized out>, client=<optimized out>, srv=0x55d4ddffbaa0)
    at rpc/virnetserver.c:143
        ret = -1
        ret = <optimized out>
        __func__ = "virNetServerProcessMsg"
#8  virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x55d4ddffbaa0)
    at rpc/virnetserver.c:164
        srv = 0x55d4ddffbaa0
        job = 0x55d4de1a5fb0
        __func__ = "virNetServerHandleJob"
#9  0x00007f86149564d0 in virThreadPoolWorker (
    opaque=opaque@entry=0x55d4de006eb0) at util/virthreadpool.c:167
        data = 0x0
        pool = 0x55d4ddffbb90
        cond = 0x55d4ddffbbf8
        priority = false
        curWorkers = 0x55d4ddffbc70
        maxLimit = 0x55d4ddffbc58
        job = 0x55d4de038d10
#10 0x00007f86149557dc in virThreadHelper (data=<optimized out>)
    at util/virthread.c:206
--Type <RET> for more, q to quit, c to continue without paging--
        args = 0x0

        local = {func = 0x7f86149563d0 <virThreadPoolWorker>,
          funcName = 0x7f8614b7000b "virNetServerHandleJob", worker = true,
          opaque = 0x55d4de006eb0}
#11 0x00007f86117bb2de in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#12 0x00007f86114ec463 in clone () from /lib64/libc.so.6
No symbol table info available.

Comment 1 Ján Tomko 2019-07-10 09:13:32 UTC

~30 % chance of reproducing this with:

$ cat <<EOF >xml
> <domain type='qemu'>
>   <name>test_domain</name>
>   <memory unit='KiB'>2097152</memory>
>   <uuid>e6e546d3-9642-4ecb-908e-f3cc84e66a39</uuid>
>   <currentMemory unit='KiB'>2097152</currentMemory>
>   <vcpu>1</vcpu>
>   <os>
>     <type arch='x86_64' machine='pc'>hvm</type>
>   </os>
>   <devices>
>     <emulator>/usr/bin/qemu-system-x86_64</emulator>
>   </devices>
> </domain>
> EOF

$ virsh create xml; virsh destroy test_domain& virsh define xml; sleep 1; virsh undefine test_domain

The define call crashes while trying to free def, which it should only do when adding it to the list failed:
#2  0x00007ffff7cbb900 in virDomainDefFree (def=<optimized out>, def@entry=0x7fffd4001fe0) at conf/domain_conf.c:3004
#3  0x00007fffc31b9f2e in qemuDomainDefineXMLFlags (conn=0x7fffd80019e0, xml=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:7427

Comment 2 Ján Tomko 2019-07-10 09:48:28 UTC

Fixed upstream by:
commit 7e760f61577e6c4adbb0b015f8f7ac1796570cdd
Author:     Marc Hartmayer <mhartmay.com>
CommitDate: 2018-08-29 10:02:03 +0200

    virDomainObjListAddLocked: fix double free
    
    If @vm has flagged as "to be removed" virDomainObjListFindByNameLocked
    returns NULL (although the definition actually exists). Therefore, the
    possibility exits that "virHashAddEntry" will raise the error
    "Duplicate key" => virDomainObjListAddObjLocked fails =>
    virDomainObjEndAPI(&vm) is called and this leads to a freeing of @def
    since @def is already assigned to vm->def. But actually this leads to
    a double free since the common usage pattern is that the caller of
    virDomainObjListAdd(Locked) is responsible for freeing @def in case of
    an error.
    
    Let's fix this by setting vm->def to NULL in case of an error.
    
    Backtrace:
    
       ➤  bt
       #0  virFree (ptrptr=0x7575757575757575)
       #1  0x000003ffb5b25b3e in virDomainResourceDefFree
       #2  0x000003ffb5b37c34 in virDomainDefFree
       #3  0x000003ff9123f734 in qemuDomainDefineXMLFlags
       #4  0x000003ff9123f7f4 in qemuDomainDefineXML
       #5  0x000003ffb5cd2c84 in virDomainDefineXML
       #6  0x000000011745aa82 in remoteDispatchDomainDefineXML
       ...
    
    Reviewed-by: Bjoern Walk <bwalk.com>
    Signed-off-by: Marc Hartmayer <mhartmay.com>

git describe: v4.7.0-rc1-30-g7e760f6157 contains: v4.7.0-rc2~2

Comment 5 yafu 2019-08-06 07:04:46 UTC

Hi,Ján,

It reports error "error: internal error: Duplicate key" when i try to verify the bug with libvirt-4.5.0-31.module+el8.1.0+3808+3325c1a3.x86_64. 
Would you help to check it please?

# virsh create vm1.xml; virsh destroy vm1& virsh define vm1.xml;sleep 1; virsh undefine vm1;
Domain vm1 created from vm1.xml

[1] 26608
Domain vm1 destroyed
error: 
Failed to define domain from vm1.xml
error: internal error: Duplicate key

[1]+  Done                    virsh destroy vm1
error: failed to get domain 'vm1'
error: Domain not found: no domain with matching name 'vm1'

Comment 6 Ján Tomko 2019-08-06 08:08:00 UTC

Hi,

the error message is not as helpful as it could be.
As long as libvirt does not crash, I consider this bug fixed.

The error message is already fixed upstream by:
commit a5c71129bf2c12a827f1bc00149acd1c572ffe9c
    virDomainObjListAddLocked: Produce better error message than 'Duplicate key'
However I do consider that a separate bug.

Comment 7 yafu 2019-08-06 09:15:31 UTC

(In reply to Ján Tomko from comment #6)
> Hi,
> 
> the error message is not as helpful as it could be.
> As long as libvirt does not crash, I consider this bug fixed.
> 
> The error message is already fixed upstream by:
> commit a5c71129bf2c12a827f1bc00149acd1c572ffe9c
>     virDomainObjListAddLocked: Produce better error message than 'Duplicate
> key'
> However I do consider that a separate bug.

Thanks. File a bug to track it:
https://bugzilla.redhat.com/show_bug.cgi?id=1737790

Comment 8 yafu 2019-08-06 09:19:34 UTC

Reproduced with libvirt-4.5.0-25.x86_64.

Verified with libvirt-4.5.0-31.module+el8.1.0+3808+3325c1a3.x86_64.
Test steps:
1.Check the libvirtd pid:
#pidof libvirtd
23812

2.#for i in {1..100}; do virsh create vm1.xml; virsh destroy vm1& virsh define vm1.xml; sleep 1; virsh undefine vm1; done

3.Check libvirtd pid again, the libvirtd not crash during step2:
#pidof libvirtd
23812

Comment 9 Katerina Koukiou 2019-08-09 13:47:42 UTC

It's still crashing for me with the same test that caused the initial issue, just the stack trace is different now.
That's on the libvirt version that this bug was verified. libvirt-daemon-4.5.0-31
However this stopped being reproduceable for me after I installed the debuginfo packages, so can't provide yet the coredump with debuginfo.

However this is the relevant section from the journal, does it give you some hint what's going on?

Aug 09 09:05:39 localhost.localdomain systemd-coredump[8447]: Process 1914 (libvirtd) of user 1001 dumped core.

Stack trace of thread 8444:
#0  0x00007f13dd0f1aa4 __pthread_mutex_lock (libpthread.so.0)
#1  0x00007f13ac1100a2 virQEMUDriverGetConfig (libvirt_driver_qemu.so)
#2  0x00007f13ac154ac0 qemuStateStop (libvirt_driver_qemu.so)
#3  0x00007f13e041656f virStateStop (libvirt.so.0)
#4  0x000055833ea86d31 daemonStopWorker (libvirtd)
#5  0x00007f13e0294d9a virThreadHelper (libvirt.so.0)
#6  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#7  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1914:
#0  0x00007f139fa0c3c0 _ZN5boost15aligned_storageILm16ELm8EED1Ev (libceph-common.so.0)
#1  0x00007f13dcd5e06c __run_exit_handlers (libc.so.6)
#2  0x00007f13dcd5e1a0 exit (libc.so.6)
#3  0x00007f13dcd4787a __libc_start_main (libc.so.6)
#4  0x000055833ea86a4e _start (libvirtd)

Stack trace of thread 1915:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1916:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1917:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1918:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1919:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1920:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295af4 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1921:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295af4 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1922:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295af4 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1923:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295af4 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1924:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295af4 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1925:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1926:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1927:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1928:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)

Stack trace of thread 1929:
#0  0x00007f13dd0f547c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1  0x00007f13e029500a virCondWait (libvirt.so.0)
#2  0x00007f13e0295b43 virThreadPoolWorker (libvirt.so.0)
#3  0x00007f13e0294d6c virThreadHelper (libvirt.so.0)
#4  0x00007f13dd0ef2de start_thread (libpthread.so.0)
#5  0x00007f13dce20133 __clone (libc.so.6)



The logs from the domain that caused the crash are here:

[root@localhost ~]# cat /var/log/libvirt/qemu/subVmTestCreate21.log 
2019-08-09 13:04:38.763+0000: starting up libvirt version: 4.5.0, package: 31.module+el8.1.0+3808+3325c1a3 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2019-07-30-15:19:59, ), qemu version: 2.12.0qemu-kvm-2.12.0-83.module+el8.1.0+3852+0ba8aef0, kernel: 4.18.0-128.el8.x86_64, hostname: localhost.localdomain
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=subVmTestCreate21,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-subVmTestCreate21/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=tcg,usb=off,vmport=off,dump-guest-core=off -m 256 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 8582a929-7854-4256-9656-da8ccf415580 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=30,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/pools/tmpPool/vmTmpDestination.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/var/lib/libvirt/novell.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=32,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fb:73:f0,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=33,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -vnc 127.0.0.1:1 -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x9 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2019-08-09T13:04:39.130770Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/1 (label charserial0)
2019-08-09T13:04:39.978852Z qemu-kvm: terminating on signal 15 from pid 1641 (<unknown process>)
2019-08-09 13:04:40.184+0000: shutting down, reason=destroyed

Comment 11 errata-xmlrpc 2019-11-05 20:51:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345

Note You need to log in before you can comment on or make changes to this bug.