Bug 1832498

According to the backtrace it seems like it happens on libvirtd shutdown code path. Do you by any chance restart libvirtd in the tests?

Yes, we restart it a *lot*. Essentially, between each test we stop it, restore the original files in /var/lib/libvirt/, and the next test restarts it.

We have a "similar" report about nondeterministic crashes in daemon shutdown for example in bug 1739564. Is it session deamon too? Does it crashes with newer libvirt too? From rhel-av, Fedora, upstream?

Thanks.

bug 1739564 was reported by our college, so likely the same cause (and I just missed there was one already opened).
On fedora-32 we have https://bugzilla.redhat.com/show_bug.cgi?id=1828207 this related problem. So if it is the same issue, than yeah, happens also on the newest fedora.
And it is happening for users with uid > 1000, so yes, it is session deamon.

Moving to advanced virtualization as it should be target there first.

Btw. Daniel Berrange started discussion about this and similar problems on the list:
https://www.redhat.com/archives/libvir-list/2020-April/msg01328.html

I think its' fixed by this commit:

commit 399039a6b1d07610294cc810ac7a01b51ff6b5cf
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Jul 9 11:12:26 2020 +0300
Commit:     Nikolay Shirokovskiy <nshirokovskiy>
CommitDate: Mon Sep 7 09:33:59 2020 +0300

    qemu: implement driver's shutdown/shutdown wait methods
    
    On shutdown we just stop accepting new jobs for worker thread so that on
    shutdown wait we can exit worker thread faster. Yes we basically stop
    processing of events for VMs but we are going to do so anyway in case of daemon
    shutdown.
    
    At the same time synchronous event processing that some API calls may require
    are still possible as per VM event loop is still running and we don't need
    worker thread for synchronous event processing.
    
    Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy>
    Reviewed-by: Daniel P. Berrangé <berrange>
    Reviewed-by: Daniel Henrique Barboza <danielhb413>

v6.8.0-rc1~275

Matej, can you try libvirt 6.8.0 or newer to see if the problem persists?

Our CI does not have 6.8.0 available. But we don't see this issue in rhel 8.5 anymore, only in 8.4 so I would say it got fixed somewhere along the way.

In that case, is RHEL-AV 8.5.0 good enough for you? I'd rather avoid backporting this to 8.4 because it was a lot of changes.

(In reply to Matej Marušák from comment #8)
> Our CI does not have 6.8.0 available. But we don't see this issue in rhel
> 8.5 anymore, only in 8.4 so I would say it got fixed somewhere along the way.

So what is your CI based on, Fedora, CentOS Stream or RHEL? Which versions? Libvirt in rhel-8.4 and rhel-8.5 will be the same - version 6.0.0 (with backports). I assume you do not pull in rhel-av-8 versions of virt module, do you?

> In that case, is RHEL-AV 8.5.0 good enough for you? I'd rather avoid backporting this to 8.4 because it was a lot of changes.

Yes, we don't require backport.


> So what is your CI based on, Fedora, CentOS Stream or RHEL? Which versions? Libvirt in rhel-8.4 and rhel-8.5 will be the same - version 6.0.0 (with backports). I assume you do not pull in rhel-av-8 versions of virt module, do you?

We are using the latest composes for rhel 8.4 and 8.5 and install what is available from normal repos.

From our side, this can be closed as current release.

In that case, I'm moving this to RHEL-AV-8.5.0 and setting TestOnly keyword so it gets properly tested.

I cannot reproduce the crash as the bug report.
But I can reproduce an crash as the followings on qemu-kvm-2.12.0-99.module+el8.2.0+5827+8c39933c.x86_64 libvirt-4.5.0-42.module+el8.2.0+6024+15a2423f.x86_64:
Script for terminal 1:
while true;do
    systemctl restart libvirtd virtlogd
    sleep 2
done

Script for terminal 2:
while true;do
    virsh iface-list --all
    sleep 1
done

Script for terminal 3:
TIME_FROM_START_TO_LOGIN=64
TIME_TO_SHUTDOWN=7
VM=pc
while true;do
        virsh -k0 start $VM
        sleep $TIME_FROM_START_TO_LOGIN
        virsh -k0 shutdown $VM
        sleep $TIME_TO_SHUTDOWN
done


The prepare an VM named pc. Run the 3 scripts. It will get the coredump:
# coredumpctl
Tue 2021-08-03 00:13:59 EDT   44862     0     0  11 present   /usr/sbin/libvirtd

Backtrace:
(gdb) bt
#0  0x00007f283eebf086 in virBitmapIsBitSet (bitmap=0x0, b=b@entry=13) at util/virbitmap.c:316
#1  0x00007f2801ca9ca4 in virQEMUCapsGet (qemuCaps=qemuCaps@entry=0x7f27e8107020, flag=flag@entry=QEMU_CAPS_KVM) at qemu/qemu_capabilities.c:1759
#2  0x00007f2801caa412 in virQEMUCapsIsValid (data=0x7f27e8107020, privData=0x7f27e80f43e0) at qemu/qemu_capabilities.c:4090
#3  0x00007f283ef5254c in virFileCacheValidate (cache=cache@entry=0x7f27e80f4250, name=name@entry=0x7f27e82b9ef0 "/usr/libexec/qemu-kvm", data=data@entry=0x7f2828074380)
    at util/virfilecache.c:280
#4  0x00007f283ef52b3b in virFileCacheLookup (cache=cache@entry=0x7f27e80f4250, name=name@entry=0x7f27e82b9ef0 "/usr/libexec/qemu-kvm") at util/virfilecache.c:323
#5  0x00007f2801caf69b in virQEMUCapsCacheLookup (cache=0x7f27e80f4250, binary=0x7f27e82b9ef0 "/usr/libexec/qemu-kvm") at qemu/qemu_capabilities.c:4957
#6  0x00007f2801ce422b in qemuDomainDeviceDefValidate (dev=0x7f2828074440, def=0x7f27e82b9890, opaque=<optimized out>) at qemu/qemu_domain.c:5731
#7  0x00007f283ef6f72c in virDomainDeviceDefValidate (dev=0x7f2828074440, def=0x7f27e82b9890, parseFlags=<optimized out>, xmlopt=<optimized out>, xmlopt=<optimized out>)
    at conf/domain_conf.c:6012
#8  0x00007f283ef5d82c in virDomainDeviceInfoIterateInternal (def=def@entry=0x7f27e82b9890, cb=cb@entry=0x7f283ef6ff10 <virDomainDefValidateDeviceIterator>, all=all@entry=true, 
    opaque=opaque@entry=0x7f28280744b0) at conf/domain_conf.c:3878
#9  0x00007f283ef9b701 in virDomainDefValidate (def=0x7f27e82b9890, caps=caps@entry=0x7f27e80f4460, parseFlags=<optimized out>, parseFlags@entry=0, xmlopt=<optimized out>)
    at conf/domain_conf.c:6360
#10 0x00007f2801d0ed02 in qemuProcessStartValidateXML (driver=0x7f27e81229d0, vm=0x7f27e82bc980, flags=<optimized out>, caps=0x7f27e80f4460, qemuCaps=0x7f281c002340)
    at qemu/qemu_process.c:3901
#11 qemuProcessStartValidate (flags=<optimized out>, caps=0x7f27e80f4460, qemuCaps=0x7f281c002340, vm=0x7f27e82bc980, driver=0x7f27e81229d0) at qemu/qemu_process.c:5216
#12 qemuProcessInit (driver=0x7f27e81229d0, vm=0x7f27e82bc980, updatedCPU=<optimized out>, asyncJob=QEMU_ASYNC_JOB_START, migration=<optimized out>, flags=<optimized out>)
    at qemu/qemu_process.c:5332
#13 0x00007f2801d0fe95 in qemuProcessStart (conn=conn@entry=0x7f27e4001f60, driver=driver@entry=0x7f27e81229d0, vm=vm@entry=0x7f27e82bc980, updatedCPU=updatedCPU@entry=0x0, 
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_START, migrateFrom=migrateFrom@entry=0x0, migrateFd=-1, migratePath=0x0, snapshot=0x0, vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=17)
    at qemu/qemu_process.c:6787
#14 0x00007f2801d7013d in qemuDomainObjStart (conn=0x7f27e4001f60, driver=driver@entry=0x7f27e81229d0, vm=0x7f27e82bc980, flags=flags@entry=0, asyncJob=QEMU_ASYNC_JOB_START)
    at qemu/qemu_driver.c:7271
#15 0x00007f2801d70789 in qemuDomainCreateWithFlags (dom=0x7f281c002d00, flags=0) at qemu/qemu_driver.c:7324
#16 0x00007f283f0d8cc7 in virDomainCreate (domain=domain@entry=0x7f281c002d00) at libvirt-domain.c:6524
#17 0x00005560f6ec45e3 in remoteDispatchDomainCreate (server=0x5560f8f3ed80, msg=0x5560f8f85830, args=0x7f281c0034a0, rerr=0x7f28280748e0, client=<optimized out>)
    at remote/remote_daemon_dispatch_stubs.h:4434
#18 remoteDispatchDomainCreateHelper (server=0x5560f8f3ed80, client=<optimized out>, msg=0x5560f8f85830, rerr=0x7f28280748e0, args=0x7f281c0034a0, ret=0x7f281c002e10)
    at remote/remote_daemon_dispatch_stubs.h:4410
#19 0x00007f283f010a94 in virNetServerProgramDispatchCall (msg=0x5560f8f85830, client=0x5560f8fa8c40, server=0x5560f8f3ed80, prog=0x5560f8f7c560) at rpc/virnetserverprogram.c:437
#20 virNetServerProgramDispatch (prog=0x5560f8f7c560, server=server@entry=0x5560f8f3ed80, client=0x5560f8fa8c40, msg=0x5560f8f85830) at rpc/virnetserverprogram.c:304
#21 0x00007f283f016f4c in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x5560f8f3ed80) at rpc/virnetserver.c:143
#22 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x5560f8f3ed80) at rpc/virnetserver.c:164
#23 0x00007f283ef44dc0 in virThreadPoolWorker (opaque=opaque@entry=0x5560f8f2a1f0) at util/virthreadpool.c:167
#24 0x00007f283ef440cc in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#25 0x00007f283bd8e2de in start_thread () from /lib64/libpthread.so.0
#26 0x00007f283babfe83 in clone () from /lib64/libc.so.6


I also run these above on libvirt-7.6.0-1.module+el8.5.0+12097+2c77910b.x86_64 qemu-kvm-6.0.0-26.module+el8.5.0+12044+525f0ebc.x86_64. It cannot get any coredump.

Michel, could you please help to check if the coredump of this comment is the same issue of this bug?

Matej, could you have a test on the latest RHEL8.5 on cockpit CI as what you did in the bug report?

(In reply to Han Han from comment #13)
> 
> Michal, could you please help to check if the coredump of this comment is
> the same issue of this bug?

It's not exactly the same problem, but it's also fixed by commit from comment 7. The commit fixes the following problem: when shutting the daemon down, parts of code clean up runtime information (e.g. clear QEMU capabilities cache) even though there might be some other thread that are still using it. Therefore, the problem can demonstrate itself under different forms - depending on what are the other threads doing. In case of cockpit it was related to libpciaccess, in your case it is starting domain. But both cases are trying to access something that was freed by shutdown thread.

(In reply to Michal Privoznik from comment #15)
> (In reply to Han Han from comment #13)
> > 
> > Michal, could you please help to check if the coredump of this comment is
> > the same issue of this bug?
> 
> It's not exactly the same problem, but it's also fixed by commit from
> comment 7. The commit fixes the following problem: when shutting the daemon
> down, parts of code clean up runtime information (e.g. clear QEMU
> capabilities cache) even though there might be some other thread that are
> still using it. Therefore, the problem can demonstrate itself under
> different forms - depending on what are the other threads doing. In case of
> cockpit it was related to libpciaccess, in your case it is starting domain.
> But both cases are trying to access something that was freed by shutdown
> thread.

OK. Thank for your explanation.

Clearing up needsinfo - if I understand correctly my reproducing is not needed anymore. Please correct me if I am wrong.