I see "Aborted (core dump)" so I wonder why there is no backtrace (generated by "thread apply all backtrace") attached to this bug. Anyway, it looks like dbus is another example of badly written library which calls abort() instead of reporting an error to the caller. But let's wait for the backtrace to confirm it.

# cat core_backtrace 
43d6bacbc58692509e92f13623e2ae60b43568f5 0x359a9 raise /lib64/libc.so.6 f33186a4c862fb0751bca60701f553b829210477
43d6bacbc58692509e92f13623e2ae60b43568f5 0x370b8 abort /lib64/libc.so.6 352bc5cf09233cea84fd089527a146cfb5b7d8dc
a35ed5186147d43260c0bb494bb2e4c874ac169d 0x2f465 _dbus_abort /lib64/libdbus-1.so.3 3b3c43ddc7e5feb2f7e89cb6fbe6b288434981d0
a35ed5186147d43260c0bb494bb2e4c874ac169d 0x26411 _dbus_warn_check_failed /lib64/libdbus-1.so.3 -
1ad6cc22a02cab97102bab868ea32f68d6c924c7 0x7476 nwfilterStateCleanup /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so 68f55a6bbfd2bf2fe40996ee63329fc6b2f7c33e
7c948901a4fa38c179277ac9f0074ffa2cfbfd73 0xf015f virStateCleanup /usr/lib64/libvirt.so.0.1000.5 8ca95ac1c229c244f1f87b8466754487e6c65e3e
8679d8b73ae194aa908b9a353c84fe3983a768cd 0x1230d main /usr/sbin/libvirtd 8ca111e224cc7fa0ba03bd03fb3624f002557020


# gdb /usr/sbin/libvirtd coredump 
GNU gdb (GDB) Red Hat Enterprise Linux (7.6-25.el7)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/libvirtd...Reading symbols from /usr/lib/debug/usr/sbin/libvirtd.debug...done.
done.
[New LWP 10038]
[New LWP 10039]
[New LWP 10043]
[New LWP 10040]
[New LWP 10041]
[New LWP 10042]
[New LWP 10049]
[New LWP 10044]
[New LWP 10048]
[New LWP 10046]
[New LWP 10047]
[New LWP 10045]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `libvirtd'.
Program terminated with signal 6, Aborted.
#0  0x00007f9d964149a9 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install audit-libs-2.3-2.el7.x86_64 augeas-libs-1.0.0-2.el7.x86_64 avahi-libs-0.6.31-11.el7.x86_64 cyrus-sasl-lib-2.1.26-6.el7.x86_64 cyrus-sasl-md5-2.1.26-6.el7.x86_64 cyrus-sasl-plain-2.1.26-6.el7.x86_64 dbus-libs-1.6.8-5.el7.x86_64 device-mapper-libs-1.02.77-5.el7.x86_64 fuse-libs-2.9.2-2.el7.x86_64 glibc-2.17-5.el7.x86_64 gmp-5.1.1-2.el7.x86_64 gnome-keyring-3.8.2-1.el7.x86_64 gnutls-3.1.10-1.el7.x86_64 keyutils-libs-1.5.5-4.el7.x86_64 krb5-libs-1.11.2-2.el7.x86_64 libblkid-2.23-1.el7.x86_64 libcap-ng-0.7.3-3.el7.x86_64 libcom_err-1.42.7-2.el7.x86_64 libcurl-7.29.0-6.el7.x86_64 libdb-5.3.21-9.el7.x86_64 libgcc-4.8.0-5.el7.x86_64 libgcrypt-1.5.2-2.el7.x86_64 libgpg-error-1.11-1.el7.x86_64 libidn-1.26-2.el7.x86_64 libnl3-3.2.21-1.el7.x86_64 libpcap-1.3.0-4.el7.x86_64 libpciaccess-0.13.1-3.el7.x86_64 libselinux-2.1.13-13.el7.x86_64 libsepol-2.1.9-1.el7.x86_64 libssh2-1.4.3-4.el7.x86_64 libtasn1-3.3-1.el7.x86_64 libuuid-2.23-1.el7.x86_64 libxml2-2.9.1-1.el7.x86_64 libxslt-1.1.28-2.el7.x86_64 netcf-libs-0.2.3-4.el7.x86_64 nettle-2.6-2.el7.x86_64 nspr-4.9.5-2.el7.x86_64 nss-3.14.3-12.0.el7.x86_64 nss-softokn-freebl-3.14.3-1.el7.x86_64 nss-util-3.14.3-1.el7.x86_64 numactl-libs-2.0.8-4.el7.x86_64 openldap-2.4.35-3.2.el7.x86_64 openssl-libs-1.0.1e-5.el7.x86_64 p11-kit-0.18.0-1.el7.x86_64 p11-kit-trust-0.18.0-1.el7.x86_64 pcre-8.32-4.el7.x86_64 systemd-libs-202-3.el7.x86_64 xz-libs-5.1.2-5alpha.el7.x86_64 yajl-2.0.4-2.el7.x86_64 zlib-1.2.7-10.el7.x86_64
(gdb) thread apply all backtrace

Thread 12 (Thread 0x7f9d8782b700 (LWP 10045)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e348, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e9b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1240) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f9d86829700 (LWP 10047)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e348, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e9b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1240) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f9d8702a700 (LWP 10046)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e348, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e9b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1520) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f9d86028700 (LWP 10048)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e348, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e9b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1520) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9d8802c700 (LWP 10044)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e348, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e9b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1520) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f9d820ef700 (LWP 10049)):
#0  0x00007f9d964c9a9d in poll () from /lib64/libc.so.6
#1  0x00007f9d992e6427 in poll (__timeout=-1, __nfds=1, __fds=0x7f9d820ee6f0) at /usr/include/bits/poll2.h:41
#2  virCommandProcessIO (cmd=cmd@entry=0x7f9d7c010a50) at util/vircommand.c:1888
#3  0x00007f9d992e9f15 in virCommandRun (cmd=cmd@entry=0x7f9d7c010a50, exitstatus=exitstatus@entry=0x0) at util/vircommand.c:2106
#4  0x00007f9d8474d282 in ebiptablesExecCLI (buf=0x7f9d820eecc0, status=0x0, outbuf=0x7f9d820eecb8) at nwfilter/nwfilter_ebiptables_driver.c:2816
#5  0x00007f9d8474ea12 in ebiptablesDriverTestCLITools () at nwfilter/nwfilter_ebiptables_driver.c:4260
#6  ebiptablesDriverInit (privileged=240) at nwfilter/nwfilter_ebiptables_driver.c:4327
#7  0x00007f9d84747567 in nwfilterStateInitialize (privileged=<optimized out>, callback=<optimized out>, opaque=<optimized out>) at nwfilter/nwfilter_driver.c:200
#8  0x00007f9d99381080 in virStateInitialize (privileged=true, callback=callback@entry=0x7f9d99d4c7f0 <daemonInhibitCallback>, opaque=opaque@entry=0x7f9d9c01e0d0) at libvirt.c:833
#9  0x00007f9d99d4c837 in daemonRunStateInit (opaque=opaque@entry=0x7f9d9c01e0d0) at libvirtd.c:876
#10 0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#11 0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f9d8902e700 (LWP 10042)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e2b0, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e7b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa13a0) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f9d8982f700 (LWP 10041)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e2b0, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e7b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1520) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f9d8a030700 (LWP 10040)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e2b0, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e7b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1680) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f9d8882d700 (LWP 10043)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e2b0, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e7b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa1240) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f9d8a831700 (LWP 10039)):
#0  0x00007f9d96bb1575 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9d99316a36 in virCondWait (c=c@entry=0x7f9d9c01e2b0, m=m@entry=0x7f9d9c01e288) at util/virthreadpthread.c:117
#2  0x00007f9d99316e7b in virThreadPoolWorker (opaque=opaque@entry=0x7f9d9bfa17e0) at util/virthreadpool.c:103
#3  0x00007f9d99316871 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161
#4  0x00007f9d96badc53 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9d964d406d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f9d99d0d880 (LWP 10038)):
#0  0x00007f9d964149a9 in raise () from /lib64/libc.so.6
#1  0x00007f9d964160b8 in abort () from /lib64/libc.so.6
#2  0x00007f9d97a13465 in _dbus_abort () from /lib64/libdbus-1.so.3
#3  0x00007f9d97a0a411 in _dbus_warn_check_failed () from /lib64/libdbus-1.so.3
#4  0x00007f9d84747476 in nwfilterStateCleanup () at nwfilter/nwfilter_driver.c:349
#5  0x00007f9d9938115f in virStateCleanup () at libvirt.c:857
#6  0x00007f9d99d4b30d in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1516
(gdb)

Thread 7 is still running nwFilterStateInitialize(), and has only gotten as far as virNWFilterTechDriversInit(), which means that it hasn't called (nwfilterDriverInstallDBusMatches() yet. Also it has initialized the nwfilterDriver lock, but doesn't yet hold it.

Thread 1 is responding to the kill signal, is inside nwfilterStateCleanup(), and in particular has acquired the nwfilterDriver lock and called nwfilterDriverRemoveDBusMatches() (called on line 349), which I guess is what results in the _dbus_warn_check_failed().

So the problem is caused by nwfilterDriverRemoveDBusMatches() being called before nwfilterDriverInstallDBusMatches() has been called.

Beyond the simple situation above, it seems generally problematic that a separate thread can be woken up to tear down the nwfilter driver when another thread is still in the process of setting it up. For example, what if thread 7 hadn't yet initialized the driver lock, but thread 1 tried to acquire it as part of cleanup?

Stefan - do you have any good ideas for making this more robust? moving the lock aquisition earlier in nwfilterStateInitialize will help, but won't eliminate the problem entirely.

Laine,

  is it stuck in thread 7 talking to firewalld ?

  My guess would be to try to solve this by adding more locking to the code. The initialization part needs to grab an nwfilter lock that all other code also already grabs when for example instantiating filters. Similarly the un-initialization code would have to grab that same lock. However, if thread 7 was stalled for some reason we would dead-lock in thread 1.

   Stefan

PS:  I am trying to see this problem on f18 but having problems to compile on f18... docs don't compile.

Generating cgroups.html.tmp
I/O error : Attempt to load network entity http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd
cgroups.html.in:2: warning: failed to load external entity "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"

A wget on that URL works, though.

Below is a possible patch addressing the coredump.

Thread 1 must be calling  nwfilterDriverRemoveDBusMatches(). It does so with nwfilterDriverLock held. In the patch below I am now moving the nwfilterDriverLock(driverState) further up so that the initialization, which seems to either take a long time or is entirely stuck, occurs with the lock held and the shutdown cannot occur at the same time. 

To avoid having to make the nwfilterDriverLock lockable multiple times / recursive I changed the virNWFilterDriverIsWatchingFirewallD to take as an argument whether it has to grab that lock. There's only a single caller at the moment that calls this function during initialization. We could remove this lock entirely and maybe append to the name of the function NoLock (?).

I could not see the same problem that you are seeing.

---
 src/nwfilter/nwfilter_driver.c            |   14 +++++++++-----
 src/nwfilter/nwfilter_driver.h            |    2 +-
 src/nwfilter/nwfilter_ebiptables_driver.c |    7 ++++++-
 3 files changed, 16 insertions(+), 7 deletions(-)

Index: libvirt/src/nwfilter/nwfilter_driver.c
===================================================================
--- libvirt.orig/src/nwfilter/nwfilter_driver.c
+++ libvirt/src/nwfilter/nwfilter_driver.c
@@ -191,6 +191,8 @@ nwfilterStateInitialize(bool privileged,
     if (!privileged)
         return 0;
 
+    nwfilterDriverLock(driverState);
+
     if (virNWFilterIPAddrMapInit() < 0)
         goto err_free_driverstate;
     if (virNWFilterLearnInit() < 0)
@@ -203,8 +205,6 @@ nwfilterStateInitialize(bool privileged,
     if (virNWFilterConfLayerInit(virNWFilterDomainFWUpdateCB) < 0)
         goto err_techdrivers_shutdown;
 
-    nwfilterDriverLock(driverState);
-
     /*
      * startup the DBus late so we don't get a reload signal while
      * initializing
@@ -314,16 +314,20 @@ nwfilterStateReload(void) {
  * Returns true if it is watching firewalld, false otherwise
  */
 bool
-virNWFilterDriverIsWatchingFirewallD(void)
+virNWFilterDriverIsWatchingFirewallD(bool needDriverLock)
 {
     bool ret;
 
     if (!driverState)
         return false;
 
-    nwfilterDriverLock(driverState);
+    if (needDriverLock)
+        nwfilterDriverLock(driverState);
+
     ret = driverState->watchingFirewallD;
-    nwfilterDriverUnlock(driverState);
+
+    if (needDriverLock)
+        nwfilterDriverUnlock(driverState);
 
     return ret;
 }
Index: libvirt/src/nwfilter/nwfilter_driver.h
===================================================================
--- libvirt.orig/src/nwfilter/nwfilter_driver.h
+++ libvirt/src/nwfilter/nwfilter_driver.h
@@ -33,6 +33,6 @@
 
 int nwfilterRegister(void);
 
-bool virNWFilterDriverIsWatchingFirewallD(void);
+bool virNWFilterDriverIsWatchingFirewallD(bool needDriverLock);
 
 #endif /* __VIR_NWFILTER_DRIVER_H__ */
Index: libvirt/src/nwfilter/nwfilter_ebiptables_driver.c
===================================================================
--- libvirt.orig/src/nwfilter/nwfilter_ebiptables_driver.c
+++ libvirt/src/nwfilter/nwfilter_ebiptables_driver.c
@@ -4191,7 +4191,12 @@ ebiptablesDriverInitWithFirewallD(void)
     int status;
     int ret = -1;
 
-    if (!virNWFilterDriverIsWatchingFirewallD())
+    /*
+     * check wheter we are watching firewalld
+     * Since we call this function during initialization we won't need
+     * to have it get the lock, so we pass 'false'.
+     */
+    if (!virNWFilterDriverIsWatchingFirewallD(false))
         return -1;
 
     firewall_cmd_path = virFindFileInPath("firewall-cmd");

There's one more problem that I found, though I could not pin-point the origin:

When for example adding this to a domain's interface XML

      <filterref filter='clean-traffic'>
        <parameter name='IP' value='10.0.0.2'/>
      </filterref>

the domain starts successfully (virsh returns exit code 0) but then the domain gets a signal 15 from libvirt through this call path below. I am not sure what is causing this.

(gdb) bt
#0  kill () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff7829172 in virProcessKill (pid=<optimized out>, 
    sig=<optimized out>) at util/virprocess.c:251
#2  0x00007ffff782922b in virProcessKillPainfully (pid=2480, 
    force=force@entry=true) at util/virprocess.c:299
#3  0x00007fffeb03052b in qemuProcessKill (vm=vm@entry=0x7fffe40e3830, 
    flags=flags@entry=1) at qemu/qemu_process.c:3934
#4  0x00007fffeb074840 in qemuDomainDestroyFlags (dom=<optimized out>, 
    flags=<optimized out>) at qemu/qemu_driver.c:1996
#5  0x00007ffff78ab777 in virDomainDestroy (domain=domain@entry=0x7fffcc0008c0)
    at libvirt.c:2225
#6  0x000000000042c53e in remoteDispatchDomainDestroy (server=<optimized out>, 
    msg=<optimized out>, args=<optimized out>, rerr=0x7fffed773cd0, 
    client=0x682e20) at remote_dispatch.h:3119
#7  remoteDispatchDomainDestroyHelper (server=<optimized out>, 
    client=0x682e20, msg=<optimized out>, rerr=0x7fffed773cd0, 
    args=<optimized out>, ret=<optimized out>) at remote_dispatch.h:3097
#8  0x00007ffff7919736 in virNetServerProgramDispatchCall (msg=0x684940, 
    client=0x682e20, server=0x67f090, prog=0x66a060)
    at rpc/virnetserverprogram.c:439
#9  virNetServerProgramDispatch (prog=0x66a060, server=server@entry=0x67f090, 
    client=0x682e20, msg=0x684940) at rpc/virnetserverprogram.c:305
#10 0x00007ffff7914828 in virNetServerProcessMsg (msg=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    prog=<optimized out>, client=<optimized out>, srv=0x67f090)
    at rpc/virnetserver.c:163
#11 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x67f090)
    at rpc/virnetserver.c:184
#12 0x00007ffff783396e in virThreadPoolWorker (opaque=opaque@entry=0x663060)
    at util/virthreadpool.c:144
#13 0x00007ffff7832fe6 in virThreadHelper (data=<optimized out>)
    at util/virthreadpthread.c:161
#14 0x0000003845a07d15 in start_thread (arg=0x7fffed774700)
    at pthread_create.c:308
#15 0x00000038456f248d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

   Stefan

Further testing again today with either libvirt in the background or in the foreground did NOT expose this timeout issue as reported in comment 7 above.

This isn't a problem specific to the nwfilter code. The virStateInitialize, virStateCleanup and virStateReload methods must all be serialized wrt each other. We probably ought to just put a mutex in all 3 to ensure this serialization

*** This bug has been marked as a duplicate of bug 1033061 ***