Description of problem: In our CI we see a lot of libvirtd crashes in our test that checks responding to libvirtd.service starting and stopping [1]. It first stops libvirtd (which works fine), and then 6 seconds after startup it crashes, see the journal log [2]. I installed debug symbols and did a symbolic stack trace [3].


Version-Release number of selected component (if applicable):

libvirt-daemon-6.1.0-2.fc32.x86_64


How reproducible: Pretty much always happens for me locally, and happens in ~ ¾ of runs in our CI as well.


Steps to Reproduce:
Originally:

1. Run `test/verify/check-machines -stv TestMachines.testLibvirt` with TEST_OS=fedora-31 or TEST_OS=fedora-32.

I tried to find a CLI reproducer, with something like

  while systemctl stop libvirtd; systemctl reset-failed libvirtd; systemctl start libvirtd; sleep 1; virsh list; virsh domifaddr 1; do true; done

but no success yet. That may well be not what virConnectListAllInterfaces() is doing, maybe you have a suggestion?


Additional info:

[1] https://github.com/cockpit-project/cockpit/blob/feb83028ec03dd286c85a06094bef14efb0a4352/test/verify/check-machines#L378

[2]
systemd[1]: Started Virtualization daemon.
dnsmasq-dhcp[3117]: DHCPDISCOVER(virbr0) 52:54:00:2a:75:c2
dnsmasq-dhcp[3117]: DHCPOFFER(virbr0) 192.168.122.77 52:54:00:2a:75:c2
dnsmasq[3117]: read /etc/hosts - 2 addresses
dnsmasq[3117]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
dnsmasq-dhcp[3117]: read /var/lib/libvirt/dnsmasq/default.hostsfile
dnsmasq-dhcp[3117]: DHCPREQUEST(virbr0) 192.168.122.77 52:54:00:2a:75:c2
dnsmasq-dhcp[3117]: DHCPACK(virbr0) 192.168.122.77 52:54:00:2a:75:c2 cirros
[6 seconds after the above]
systemd-coredump[16466]: Process 15608 (libvirtd) of user 0 dumped core.
                         
                         Stack trace of thread 15636:
                         #0  0x00007f4f59881839 compile_exp (libaugeas.so.0 + 0x1c839)
                         #1  0x00007f4f5987f9b2 load_module_file (libaugeas.so.0 + 0x1a9b2)
                         #2  0x00007f4f59880132 lookup_internal (libaugeas.so.0 + 0x1b132)
                         #3  0x00007f4f59880352 ctx_lookup_bnd (libaugeas.so.0 + 0x1b352)
                         #4  0x00007f4f59880709 check_exp (libaugeas.so.0 + 0x1b709)
                         #5  0x00007f4f5987fac5 load_module_file (libaugeas.so.0 + 0x1aac5)
                         #6  0x00007f4f59880132 lookup_internal (libaugeas.so.0 + 0x1b132)
                         #7  0x00007f4f5988024d lens_lookup (libaugeas.so.0 + 0x1b24d)
                         #8  0x00007f4f59891265 lens_from_name (libaugeas.so.0 + 0x2c265)
                         #9  0x00007f4f59891bdc transform_validate (libaugeas.so.0 + 0x2cbdc)
                         #10 0x00007f4f5986fbd3 aug_load (libaugeas.so.0 + 0xabd3)
                         #11 0x00007f4f598d2e45 get_augeas (libnetcf.so.1 + 0x7e45)
                         #12 0x00007f4f598d58e5 list_interface_ids.constprop.0 (libnetcf.so.1 + 0xa8e5)
                         #13 0x00007f4f598e69d0 netcfConnectListAllInterfaces (libvirt_driver_interface.so + 0x59d0)
                         #14 0x00007f4f6d8ce38f virConnectListAllInterfaces (libvirt.so.0 + 0x32838f)
                         #15 0x000055feee04d588 remoteDispatchConnectListAllInterfacesHelper (libvirtd + 0x2b588)
                         #16 0x00007f4f6d7d43e0 virNetServerProgramDispatch (libvirt.so.0 + 0x22e3e0)
                         #17 0x00007f4f6d7d949c virNetServerHandleJob (libvirt.so.0 + 0x23349c)
                         #18 0x00007f4f6d6f075f virThreadPoolWorker (libvirt.so.0 + 0x14a75f)
                         #19 0x00007f4f6d6efa7c virThreadHelper (libvirt.so.0 + 0x149a7c)
                         #20 0x00007f4f6d37f432 start_thread (libpthread.so.0 + 0x9432)
                         #21 0x00007f4f6d2929d3 __clone (libc.so.6 + 0x1019d3)
                         
                         Stack trace of thread 15608:
                         #0  0x00007f4f6d385e92 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xfe92)
                         #1  0x00007f4f6d6efcfb virCondWait (libvirt.so.0 + 0x149cfb)
                         #2  0x00007f4f6d6f09e8 virThreadPoolFree (libvirt.so.0 + 0x14a9e8)
                         #3  0x00007f4f6d7d903f virNetServerDispose (libvirt.so.0 + 0x23303f)
                         #4  0x00007f4f6d6cac1b virObjectUnref (libvirt.so.0 + 0x124c1b)
                         #5  0x00007f4f6d6a2bad virHashFree (libvirt.so.0 + 0xfcbad)
                         #6  0x00007f4f6d7d78c6 virNetDaemonDispose (libvirt.so.0 + 0x2318c6)
                         #7  0x00007f4f6d6cac1b virObjectUnref (libvirt.so.0 + 0x124c1b)
                         #8  0x000055feee048850 main (libvirtd + 0x26850)
                         #9  0x00007f4f6d1b8042 __libc_start_main (libc.so.6 + 0x27042)
                         #10 0x000055feee04964e _start (libvirtd + 0x2764e)
systemd[1]: systemd-coredump: Succeeded.

[3]
#0  0x00007f4f59881839 in apply (ctx=<optimized out>, app=0x7f4f34055f70) at syntax.c:1586
        f = 0x7f4f344927a0
        result = 0x0
        lctx = {name = 0x7f4f340126a0 "Shellvars", aug = 0x7f4f34005030, local = 0x7f4f34496a30}
        arg = 0x7f4f34055e10
        f = <optimized out>
        result = <optimized out>
        lctx = <optimized out>
        arg = <optimized out>
        __PRETTY_FUNCTION__ = "apply"
#1  compile_exp (info=<optimized out>, exp=<optimized out>, ctx=<optimized out>) at syntax.c:1669
        v = 0x0
        __PRETTY_FUNCTION__ = "compile_exp"
#2  0x00007f4f5987f9b2 in compile_decl (ctx=0x7f4f5e9183d0, term=0x7f4f34055ff0) at syntax.c:1757
        result = <optimized out>
        v = <optimized out>
        __PRETTY_FUNCTION__ = "compile_decl"
        result = <optimized out>
        v = <optimized out>
        error = <optimized out>
        ms = <optimized out>
#3  compile (aug=0x7f4f34005030, term=<optimized out>) at syntax.c:1795
        dcl = 0x7f4f34055ff0
        ctx = {name = 0x7f4f340126a0 "Shellvars", aug = 0x7f4f34005030, local = 0x7f4f34492890}
        autoload = 0x0
        module = <optimized out>
        ctx = <optimized out>
        autoload = <optimized out>
        __PRETTY_FUNCTION__ = "compile"
        module = <optimized out>
        dcl = <optimized out>
        bnd = <optimized out>
#4  load_module_file (aug=aug@entry=0x7f4f34005030, filename=filename@entry=0x55feef660490 "/usr/share/augeas/lenses/dist/shellvars.aug", 
    name=name@entry=0x7f4f3400cdd0 "Shellvars") at syntax.c:1986
        term = 0x7f4f34057870
        result = -1
        module = <optimized out>
        bad_module = <optimized out>
        __PRETTY_FUNCTION__ = "load_module_file"
#5  0x00007f4f59880132 in load_module (name=0x7f4f3400cdd0 "Shellvars", aug=0x7f4f34005030) at syntax.c:2021
        filename = 0x55feef660490 "/usr/share/augeas/lenses/dist/shellvars.aug"
        filename = <optimized out>
#6  load_module (name=<optimized out>, aug=<optimized out>) at syntax.c:2012
        filename = <optimized out>
        filename = <optimized out>
#7  lookup_internal (aug=aug@entry=0x7f4f34005030, ctx_modname=ctx_modname@entry=0x7f4f3400d470 "Sysconfig", 
    name=name@entry=0x7f4f3400cac0 "Shellvars.eol", bnd=bnd@entry=0x7f4f5e9184a0) at syntax.c:516
        modname = 0x7f4f3400cdd0 "Shellvars"
        __PRETTY_FUNCTION__ = "lookup_internal"
        loaded = <optimized out>
#8  0x00007f4f59880352 in ctx_lookup_bnd (info=0x7f4f3400c920, ctx=0x7f4f5e918570, name=0x7f4f3400cac0 "Shellvars.eol") at syntax.c:548
        r = <optimized out>
        modname = <optimized out>
        b = 0x0
        nlen = <optimized out>
#9  0x00007f4f59880709 in ctx_lookup_type (ident=<optimized out>, ident=<optimized out>, ctx=0x7f4f5e918570, info=<optimized out>) at syntax.c:1239
        b = <optimized out>
#10 check_exp (ctx=0x7f4f5e918570, term=0x7f4f3400d890) at syntax.c:1239
--Type <RET> for more, q to quit, c to continue without paging--c
        t = <optimized out>
        result = <optimized out>
        result = <optimized out>
        __PRETTY_FUNCTION__ = "check_exp"
        func = <optimized out>
        t = <optimized out>
#11 check_exp (term=0x7f4f3400d890, ctx=0x7f4f5e918570) at syntax.c:1174
        __PRETTY_FUNCTION__ = "check_exp"
        func = <optimized out>
        t = <optimized out>
#12 0x00007f4f5987fac5 in check_decl (ctx=0x7f4f5e918570, term=0x7f4f34012590) at syntax.c:1299
        __PRETTY_FUNCTION__ = "check_decl"
#13 typecheck (aug=0x7f4f34005030, term=<optimized out>) at syntax.c:1362
        dcl = 0x7f4f34012590
        ok = 1
        ctx = {name = 0x7f4f3400d470 "Sysconfig", aug = 0x7f4f34005030, local = 0x0}
        fname = <optimized out>
        basenam = <optimized out>
        ok = <optimized out>
        ctx = <optimized out>
        fname = <optimized out>
        basenam = <optimized out>
        __PRETTY_FUNCTION__ = "typecheck"
        dcl = <optimized out>
#14 load_module_file (aug=aug@entry=0x7f4f34005030, filename=filename@entry=0x7f4f3400c950 "/usr/share/augeas/lenses/dist/sysconfig.aug", name=name@entry=0x7f4f3400d450 "Sysconfig") at syntax.c:1983
        term = 0x7f4f34012610
        result = -1
        module = <optimized out>
        bad_module = <optimized out>
        __PRETTY_FUNCTION__ = "load_module_file"
#15 0x00007f4f59880132 in load_module (name=0x7f4f3400d450 "Sysconfig", aug=0x7f4f34005030) at syntax.c:2021
        filename = 0x7f4f3400c950 "/usr/share/augeas/lenses/dist/sysconfig.aug"
        filename = <optimized out>
#16 load_module (name=<optimized out>, aug=<optimized out>) at syntax.c:2012
        filename = <optimized out>
        filename = <optimized out>
#17 lookup_internal (aug=0x7f4f34005030, ctx_modname=ctx_modname@entry=0x0, name=0x7f4f3400cca0 "\320", <incomplete sequence \312>, bnd=bnd@entry=0x7f4f5e918620) at syntax.c:516
        modname = 0x7f4f3400d450 "Sysconfig"
        __PRETTY_FUNCTION__ = "lookup_internal"
        loaded = <optimized out>
#18 0x00007f4f5988024d in lens_lookup (aug=<optimized out>, qname=<optimized out>) at syntax.c:527
        bnd = 0x0
#19 0x00007f4f59891265 in lens_from_name (aug=aug@entry=0x7f4f34005030, name=0x7f4f3400cca0 "\320", <incomplete sequence \312>) at transform.c:689
        result = 0x0
#20 0x00007f4f59891bdc in transform_validate (aug=aug@entry=0x7f4f34005030, xfm=xfm@entry=0x7f4f3400cd70) at transform.c:796
        l = 0x7f4f3400ce10
#21 0x00007f4f5986fbd3 in aug_load (aug=0x7f4f34005030) at augeas.c:766
        xfm = 0x7f4f3400cd70
        option = 0x7f4f34006a50 "\210"
        meta = <optimized out>
        meta_files = 0x7f4f3400c880
        files = 0x7f4f34005d20
        load = <optimized out>
        vars = 0x7f4f34006040
#22 0x00007f4f598d2e45 in get_augeas () from /lib64/libnetcf.so.1
No symbol table info available.
#23 0x00007f4f598d58e5 in list_interface_ids.constprop () from /lib64/libnetcf.so.1
No symbol table info available.
#24 0x00007f4f598e69d0 in netcfConnectListAllInterfaces (conn=0x55feef65f220, ifaces=0x7f4f5e9188d0, flags=<optimized out>) at ../../src/interface/interface_backend_netcf.c:643
        count = <optimized out>
        i = <optimized out>
        ncf_flags = 3
        iface = 0x0
        tmp_iface_objs = 0x0
        iface_obj = 0x0
        niface_objs = 0
        ret = -1
        names = 0x0
        __FUNCTION__ = "netcfConnectListAllInterfaces"
        __func__ = "netcfConnectListAllInterfaces"
#25 0x00007f4f6d8ce38f in virConnectListAllInterfaces (conn=0x55feef65f220, ifaces=0x7f4f5e9188d0, flags=3) at ../../src/libvirt-interface.c:97
        ret = <optimized out>
        __func__ = "virConnectListAllInterfaces"
        __FUNCTION__ = "virConnectListAllInterfaces"
#26 0x000055feee04d588 in remoteDispatchConnectListAllInterfaces (server=<optimized out>, msg=0x55feef633e10, ret=0x7f4f34004f30, args=0x7f4f34004f10, rerr=0x7f4f5e9189a0, client=<optimized out>) at ./remote/remote_daemon_dispatch_stubs.h:1450
        rv = -1
        i = <optimized out>
        nresults = 0
        conn = <optimized out>
        result = 0x0
        rv = <optimized out>
        i = <optimized out>
        result = <optimized out>
        nresults = <optimized out>
        conn = <optimized out>
        __FUNCTION__ = "remoteDispatchConnectListAllInterfaces"
#27 remoteDispatchConnectListAllInterfacesHelper (server=<optimized out>, client=<optimized out>, msg=0x55feef633e10, rerr=0x7f4f5e9189a0, args=0x7f4f34004f10, ret=0x7f4f34004f30) at ./remote/remote_daemon_dispatch_stubs.h:1429
        rv = <optimized out>
        __func__ = "remoteDispatchConnectListAllInterfacesHelper"
#28 0x00007f4f6d7d43e0 in virNetServerProgramDispatchCall (msg=0x55feef633e10, client=0x55feef633c10, server=0x55feef5c5eb0, prog=0x55feef62cfb0) at ../../src/rpc/virnetserverprogram.c:430
        ret = 0x7f4f34004f30 ""
        rv = -1
        i = <optimized out>
        identity = 0x55feef62ef60
        arg = 0x7f4f34004f10 "\001"
        dispatcher = 0x55feee0a2740 <remoteProcs+13632>
        rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0, str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}
        arg = <optimized out>
        ret = <optimized out>
        rv = <optimized out>
        dispatcher = <optimized out>
        rerr = <optimized out>
        i = <optimized out>
        identity = <optimized out>
        __FUNCTION__ = "virNetServerProgramDispatchCall"
#29 virNetServerProgramDispatch (prog=0x55feef62cfb0, server=server@entry=0x55feef5c5eb0, client=0x55feef633c10, msg=0x55feef633e10) at ../../src/rpc/virnetserverprogram.c:302
        ret = -1
        rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0, str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0, int2 = 0, net = 0x0}
        __func__ = "virNetServerProgramDispatch"
        __FUNCTION__ = "virNetServerProgramDispatch"
#30 0x00007f4f6d7d949c in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x55feef5c5eb0) at ../../src/rpc/virnetserver.c:136
        __func__ = "virNetServerProcessMsg"
#31 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x55feef5c5eb0) at ../../src/rpc/virnetserver.c:153
        srv = 0x55feef5c5eb0
        job = 0x55feef660dc0
        __func__ = "virNetServerHandleJob"
#32 0x00007f4f6d6f075f in virThreadPoolWorker (opaque=opaque@entry=0x55feef5d56e0) at ../../src/util/virthreadpool.c:163
        data = 0x0
        pool = 0x55feef5c7610
        cond = 0x55feef5c7678
        priority = false
        curWorkers = 0x55feef5c76f0
        maxLimit = 0x55feef5c76d8
        job = 0x55feef660460
#33 0x00007f4f6d6efa7c in virThreadHelper (data=<optimized out>) at ../../src/util/virthread.c:196
        args = 0x0
        local = {func = 0x7f4f6d6f0660 <virThreadPoolWorker>, funcName = 0x7f4f6d93b8c5 "virNetServerHandleJob", worker = true, opaque = 0x55feef5d56e0}
#34 0x00007f4f6d37f432 in start_thread (arg=<optimized out>) at pthread_create.c:477
        ret = <optimized out>
        pd = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139978865743616, 5653074933165692965, 140732928762158, 140732928762159, 140732928762336, 139978865743616, -5697760994462164955, -5697867318766102491}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#35 0x00007f4f6d2929d3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95