Bug 1530346

Summary: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMemoryStats)
Product: [Community] Virtualization Tools Reporter: John <jss>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: bugzilla49, germano.massullo, jss, libvirt-maint, mprivozn, pkrempa, pmarciniak, rbalakri, redhat, syzop
Target Milestone: ---Keywords: Reopened
Target Release: ---Flags: jss: needinfo-
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1569861 (view as bug list) Environment:
Last Closed: 2020-09-02 11:33:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1569861    

Description John 2018-01-02 16:57:08 UTC 
Description of problem:

After requesting an operation on a VM, virt-manager hangs, and has to be killed.  When re-launched, virt-manager cannot connect to qemu, and libvirt status shows timeout when trying to acquire state change lock.

After recent updates to Fedora 27, I'm now hitting this bug on a pretty regular basis. Sometimes all that's required to trigger it, is asking a machine to shutdown via virt-manager. Other times, it seems to be triggered by doing several operations in quick succession.

Version-Release number of selected component (if applicable):
libvirt-3.7.0-3.fc27.x86_64
qemu-2.10.1-2.fc27.x86_64

How reproducible:
Intermittent, but pretty easy to trigger *accidentally* after recent upgrade to F27.

Steps to Reproduce:
1. Ask machine to shutdown using virt-manager (sometimes this is sufficient)
2. Ask another machine to start or shutdown.
3. It will probably be hard to trigger deliberately, but seems to be happening quite frequently now, when I'm *not* trying to trigger it...

Actual results:
virt-manager hangs, and has to be TERMed or KILLed, when re-launched, it cannot connect to Qemu/KVM, virsh list reports machine stuck "in shutdown" although qemu process has exited (in some fashion), libvirt service reports errors:

Jan 03 02:40:55 Il-Duce libvirtd[1240]: 2018-01-02 16:10:55.366+0000: 1350: warning : qemuGetProcessInfo:1434 : cannot parse process status data
Jan 03 02:41:25 Il-Duce libvirtd[1240]: 2018-01-02 16:11:25.370+0000: 1349: warning : qemuDomainObjBeginJobInternal:4115 : Cannot start job (modify, none) for domain jss_Voltaire_c74; current job is (query, none) owned by (1352 remoteDispatchDomainMemoryStats, 0 <null>) for (277s, 0s)
Jan 03 02:41:25 Il-Duce libvirtd[1240]: 2018-01-02 16:11:25.370+0000: 1349: error : qemuDomainObjBeginJobInternal:4127 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMemoryStats)
Jan 03 02:41:50 Il-Duce libvirtd[1240]: 2018-01-02 16:11:50.028+0000: 1349: warning : qemuGetProcessInfo:1434 : cannot parse process status data
Jan 03 02:41:55 Il-Duce libvirtd[1240]: 2018-01-02 16:11:55.371+0000: 1353: warning : qemuDomainObjBeginJobInternal:4115 : Cannot start job (query, none) for domain jss_Voltaire_c74; current job is (query, none) owned by (1352 remoteDispatchDomainMemoryStats, 0 <null>) for (307s, 0s)
Jan 03 02:41:55 Il-Duce libvirtd[1240]: 2018-01-02 16:11:55.371+0000: 1353: error : qemuDomainObjBeginJobInternal:4127 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMemoryStats)
Jan 03 02:42:13 Il-Duce libvirtd[1240]: 2018-01-02 16:12:13.729+0000: 1240: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error
...
Jan 03 03:00:50 Il-Duce libvirtd[1240]: 2018-01-02 16:30:50.029+0000: 1349: warning : qemuGetProcessInfo:1434 : cannot parse process status data
Jan 03 03:01:50 Il-Duce libvirtd[1240]: 2018-01-02 16:31:50.028+0000: 1350: warning : qemuGetProcessInfo:1434 : cannot parse process status data
Jan 03 03:02:50 Il-Duce libvirtd[1240]: 2018-01-02 16:32:50.028+0000: 1351: warning : qemuGetProcessInfo:1434 : cannot parse process status data

Stopping libvirtd fails and systemd is forced to kill the unit.

Restarting libvirtd tends to succeed, although service status shows this:
Jan 03 03:19:20 Il-Duce libvirtd[21222]: 2018-01-02 16:49:20.500+0000: 21473: error : qemuMonitorOpenUnix:376 : failed to connect to monitor socket: No such process

VM that was shutdown can be restarted.

I don't think it is always so easy to recover, I'm pretty sure i had to reboot to bring libvirt back, a week or so ago.

Expected results:
libvirt & virt-manager behave properly. 

Additional info:
Comment 1 Peter Krempa 2018-01-04 08:38:55 UTC 
Could you please post the stack trace of the libvirtd process when that happens? (e.g. using gstack, or gdb).
Comment 2 Carlos Guidugli 2018-03-14 03:23:19 UTC 
+1. Having this problem occasionally too. Trying just to power off the machine do not work. Need to restart libvirtd.


 Id    Name                           State
----------------------------------------------------
 1     PURGATORY                      running
 3     BATMAN                         in shutdown


$ sudo virsh destroy BATMAN
error: Failed to destroy domain BATMAN
error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMemoryStats)
Comment 3 John 2018-03-19 12:33:49 UTC 
Sorry for the delay, I have been relocating to Sydney and starting new job. I don't have time for this.

But it has happened again, as soon as I have to use libvirt:

Here is the stack trace:

[root@Il-Duce 03-19 22:56:29 ~]# ps -ef | grep libvirtd
root        1163       1  0 21:02 ?        00:00:32 /usr/sbin/libvirtd
root       46035   31093  0 22:56 pts/0    00:00:00 grep --color=auto libvirtd
[root@Il-Duce 03-19 22:56:35 ~]# gstack 1163
Thread 17 (Thread 0x7f02bcc44700 (LWP 31346)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f02bd445700 (LWP 1634)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f02bdc46700 (LWP 1633)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f02be447700 (LWP 1632)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f02bec48700 (LWP 1631)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f02bf449700 (LWP 1630)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f02f3fff700 (LWP 1308)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec934e8 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f02f8916700 (LWP 1307)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec934e8 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f02f9117700 (LWP 1306)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec934e8 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f02f9918700 (LWP 1305)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec934e8 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f02fa119700 (LWP 1304)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec934e8 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f02fa91a700 (LWP 1303)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f02fb11b700 (LWP 1302)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f02c080665c in qemuMonitorSend () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#3  0x00007f02c081a219 in qemuMonitorJSONCommandWithFd () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f02c081c560 in qemuMonitorJSONGetBalloonInfo () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f02c081c6d5 in qemuMonitorJSONGetMemoryStats () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f02c082d5d3 in qemuDomainMemoryStatsInternal () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f02c083ac59 in qemuDomainMemoryStats () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f030ed4e668 in virDomainMemoryStats () from /lib64/libvirt.so.0
#9  0x000056067677cbbc in remoteDispatchDomainMemoryStatsHelper ()
#10 0x00007f030edb80bc in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#11 0x000056067679ac58 in virNetServerHandleJob ()
#12 0x00007f030ec934a1 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#15 0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f02f311b700 (LWP 1301)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f02fb91c700 (LWP 1300)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f02fc11d700 (LWP 1299)):
#0  0x00007f030ae6ecbb in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f030ec92a76 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f030ec935b3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f030ec92818 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f030ae6861b in start_thread () from /lib64/libpthread.so.0
#5  0x00007f030ab95c2f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f030f991980 (LWP 1163)):
#0  0x00007f030ab8967b in poll () from /lib64/libc.so.6
#1  0x00007f030ec3c5d1 in virEventPollRunOnce () from /lib64/libvirt.so.0
#2  0x00007f030ec3b151 in virEventRunDefaultImpl () from /lib64/libvirt.so.0
#3  0x00007f030edb23c5 in virNetDaemonRun () from /lib64/libvirt.so.0
#4  0x0000560676761058 in main ()
[root@Il-Duce 03-19 22:56:41 ~]#


[root@Il-Duce 03-19 21:50:27 ~]# systemctl status libvirtd
? libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-03-19 21:01:44 ACDT; 1h 53min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1163 (libvirtd)
    Tasks: 24 (limit: 32768)
   Memory: 69.6M
      CPU: 34.716s
   CGroup: /system.slice/libvirtd.service
           ??1163 /usr/sbin/libvirtd
           ??2505 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/soe.vorpal.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ??2506 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/soe.vorpal.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ??2676 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/dns.vorpal.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ??2868 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ??2869 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ??3034 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/ovirt.vorpal.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ??3035 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/ovirt.vorpal.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

Mar 19 22:47:38 Il-Duce libvirtd[1163]: 2018-03-19 12:17:38.155+0000: 1299: error : qemuDomainObjBeginJobInternal:4127 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMemoryStats)
Mar 19 22:47:38 Il-Duce libvirtd[1163]: 2018-03-19 12:17:38.161+0000: 1163: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error
Mar 19 22:47:38 Il-Duce libvirtd[1163]: 2018-03-19 12:17:38.395+0000: 1299: warning : qemuGetProcessInfo:1434 : cannot parse process status data

And yes, it is the same error that Carlos is seeing. Extremely frustrating. As soon as I have to use libvirt again, it hits me.
Comment 4 John 2018-03-19 12:35:22 UTC 
This time, all I did was use virt-manager to tell a VM to shutdown. And... game over.
Comment 5 John 2018-03-19 12:40:17 UTC 
I updated my system a week ago, so I am now seeing this with:

Linux Il-Duce 4.15.6-300.fc27.x86_64 #1 SMP Mon Feb 26 18:43:03 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

libvirt-3.7.0-4.fc27.x86_64
qemu-2.10.1-2.fc27.x86_64
Comment 6 Michael Barker 2018-04-03 03:17:03 UTC 
I am seeing the same error on a Ubuntu 17.10 host as well, so it's definitely not only to do with RedHat/Fedora builds.

Linux S1 4.13.0-37-generic #42-Ubuntu SMP Wed Mar 7 14:13:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ virsh --version
3.6.0
$ libvirtd --version
libvirtd (libvirt) 3.6.0
$ /usr/bin/qemu-system-x86_64 --version
QEMU emulator version 2.10.1(Debian 1:2.10+dfsg-0ubuntu3.5)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

All installed from repository.

I should also note that I only encounter this issue with VMs that have PCIe devices passed through (one VM with a GPU and USB controller, another with an Ethernet controller and a SAS controller). Other VMs with no PCIe passthrough have never encountered this issue. It also occurs with VMs of any OS (I have encountered with FreeBSD 11, Ubuntu 17.04 and 17.10, Windows 7 and 10).

I've found that the best way to reproduce this error is to either force a PCIe device to crash within the guest (causing the guest OS to kernel panic/BSOD), or to force-shutdown the guest from virt-manager while PCIe devices are under significant load (I have yet to quantify "significant load"). I have been unable to reproduce by changing states from command line via the virsh command.
Comment 7 Bram Matthys 2018-04-13 12:45:51 UTC 
Same here, VM crashes within minutes when it starts running windows update (w10 64 bit). Then virt-manager freezes up, and often virsh becomes unresponsive as well, even 'virsh destroy' no longer works.
This is:
* Ubuntu 17.10
* Linux 4.13.0-38-generic #43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
* QEMU emulator version 2.10.1(Debian 1:2.10+dfsg-0ubuntu3.5)

Although I think the previous comment from Michael Barker is likely more helpful than this "me too!" post.
Comment 8 John 2018-04-13 13:55:01 UTC 
I don't think I've got pcie passthrough enabled on any of my vms, unless that is the default now, with the latest kernel/libvirt/qemu.
Comment 9 John 2018-04-13 14:00:48 UTC 
The stack trace I provided shows the problem pretty clearly, but if there is any further info i can collect to narrow it down further, let me know. I might have some time this weekend to do some further investigation.

BTW, even if i don't have PCIe passthrough, I am using host CPU passthrough on many or most of my VMs, as I'm doing some nested virtualisation. Perhaps that is a factor.
Comment 10 Michal Privoznik 2018-04-25 08:41:03 UTC 
I suspect this is a race somewhere between one thread shutting down the domain (and clearing out the monitor) while the other thread executing virDomainMemoryStats() is still talking to it.

So I am able to reproduce with 3.7.0 but unable to reproduce with current master (v4.2.0-457-gda613819e9). Important thing is that the host itself is under heavy load so that the race can occur. I'm running bisect now to see which commit fixed this.

Anyway, if somebody (John, Michael, Bram, Tomas?) could try to reproduce with latest git that'd be awesome.
Comment 11 John 2018-04-25 11:49:05 UTC 
Hi Michael, 

It definitely has not required heavy load for me to reproduce. Although load may well make it more likely to happen, it still happens quite easily without load, for me.

I will try to give it a test with latest git (of libvirt i assume) this weekend.

Cheers.
Comment 12 John 2018-04-25 11:50:00 UTC 
Michal. Woops sorry i misspelled your name there!
Comment 13 Michal Privoznik 2018-04-25 12:02:52 UTC 
Yeah, maybe on some machines the host doesn't need to be under load. However, my machine has 8 cores, so if this really is a race condition it helps to reproduce the bug. At least for me. Anyway, I'll wait for your testing.
Comment 14 Michal Privoznik 2018-04-25 14:01:43 UTC 
Shoot. My bisect points at this commit 72adaf2f10509c3682f2c65ffad4176e00e5a2fb  (v4.1.0-rc1~337) which is supposed to fix this bug 1536461 which follows my investigation and suspicion. So basically, 4.1.0 is fixed, the problematic commit that introduced the bug is v3.4.0-rc1~119.
Comment 15 John 2018-04-26 13:10:04 UTC 
I've got 8 cores on my machine too, but seriously, they are rarely under much load, and i've still seen this bug crop up easily. But of course, if *trying* to reproduce the bug on demand, it wouldn't surprise me if a heavy load helps.

I will definitely test on the weekened with new git if you think it's fixed already, i will try to give my vms some use and see if i can confirm.

Cheers :-)
Comment 16 John 2018-04-28 15:07:22 UTC 
Well I've compiled libvirt from git, but I don't want it blatting any of my official rpm files, so i installed it to /usr/local/libvirt to take a look at what it installs. There were not too many libs and binaries, so i moved my distro libs and sbin out of the way, and copy those into place.

libvirtd reports:
[root@Il-Duce 04-28 23:57:18 lib64]# libvirtd --version
libvirtd (libvirt) 4.3.0

And libvirtd service starts ok.
But when i try to start virt-manager, it says no connection driver available for qemu:///system

So it looks like I need a new libvirt qemu driver but the build from git has only given me libvirt_driver_qemu.la - no .so shared lib.

Why no shared libs for the qemu connection driver... arrgh. Weak.
Comment 17 John 2018-04-28 15:15:34 UTC 
Same result if I just modify the systemd service file to start the libvirtd under /usr/local/libvirt

Libvirtd starts, but virt-manager won't connect.
Comment 18 John 2018-04-28 15:16:31 UTC 
disappointing
Comment 19 John 2018-04-29 05:56:51 UTC 
I mean, seriously, i've run configure with:

--prefix=/usr/local/libvirt --sysconfdir=/etc --localstatedir=/var --with-driver-modules --enable-shared

So I've used --enable-shared, and still I get no shared library for libvirt_driver_qemu.la

What the?
Comment 20 John 2018-04-29 14:28:24 UTC 
Bah. Ok I found some inctructions re use of "--system" with autogen, so now virt-manager connects and I have VMs running on libvirt git 4.3.0.

As it is now late Sunday night, I've mostly missed the window to do some intensive testing over the weekend, so it will probably take a few more days to give them a thorough testing and be confident that I cannot reproduce the issue on libvirt 4.3.0.
Comment 21 Michal Privoznik 2018-05-15 11:54:15 UTC 
John, any update on this? I think this is fixed by commit mentioned in comment 14. I'd like to close this as such.
Comment 22 Michal Privoznik 2018-05-29 12:08:38 UTC 
Okay, no update in two weeks. As I say, this should be fixed now therefore I'm closing this one. However, if you disagree please feel free to repoen.
Comment 23 Germano Massullo (Thetra) 2020-09-01 23:54:02 UTC 
libvirt-6.1.0-4.fc32.x86_64
virt-manager-2.2.1-3.fc32.noarch

virt-manager was not responding properly so I tried to run
# virsh shutdown vm
but after some minutes of not being responsive I got error

errore: Timed out during operation: cannot acquire state change lock (held by monitor=qemuProcessReconnect)

this is the GDB trace of libvirtd process that spawned when I ran the virsh command

Thread 18 (Thread 0x7fd6d2cf5700 (LWP 1290844)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fd6c4002aa0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fd6c4002a50, cond=0x7fd6c4002a78) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7fd6c4002a78, mutex=0x7fd6c4002a50) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd721a9cc24 in qemuMonitorSend () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007fd721aabba8 in qemuMonitorJSONCommandWithFd () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007fd721aae435 in qemuMonitorJSONSetCapabilities () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007fd721a7bf7c in qemuConnectMonitor () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007fd721a880c9 in qemuProcessReconnect () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#9  0x00007fd73ab4aa9a in virThreadHelper () from /lib64/libvirt.so.0
#10 0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7fd6f0c02700 (LWP 1290834)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fd6d41046e8) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fd6d4104680, cond=0x7fd6d41046c0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7fd6d41046c0, mutex=0x7fd6d4104680) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd722d9cc94 in udevEventHandleThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so
#5  0x00007fd73ab4aa9a in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7fd6f2b77700 (LWP 1290783)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1eda11890) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1eda11840, cond=0x55c1eda11868) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1eda11868, mutex=0x55c1eda11840) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7fd6f3378700 (LWP 1290782)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1eda11890) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1eda11840, cond=0x55c1eda11868) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1eda11868, mutex=0x55c1eda11840) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7fd7208ed700 (LWP 1290781)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1eda11890) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1eda11840, cond=0x55c1eda11868) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1eda11868, mutex=0x55c1eda11840) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7fd7210ee700 (LWP 1290780)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1eda11890) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1eda11840, cond=0x55c1eda11868) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1eda11868, mutex=0x55c1eda11840) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7fd7218ef700 (LWP 1290779)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1eda11890) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1eda11840, cond=0x55c1eda11868) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1eda11868, mutex=0x55c1eda11840) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7fd7237fe700 (LWP 1290778)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0f94) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0f68) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0f68, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b7e4 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fd723fff700 (LWP 1290777)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0f94) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0f68) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0f68, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b7e4 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fd728d67700 (LWP 1290776)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0f90) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0f68) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0f68, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b7e4 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fd729568700 (LWP 1290775)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0f94) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0f68) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0f68, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b7e4 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fd729d69700 (LWP 1290774)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0f94) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0f68) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0f68, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b7e4 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fd72a56a700 (LWP 1290773)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0ef4) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0ec8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0ec8, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fd72ad6b700 (LWP 1290772)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0ef0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0ec8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0ec8, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fd722d6b700 (LWP 1290771)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0ef4) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0ec8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0ec8, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fd72b56c700 (LWP 1290770)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55c1ed9a0ef0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c1ed9a0ea0, cond=0x55c1ed9a0ec8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x55c1ed9a0ec8, mutex=0x55c1ed9a0ea0) at pthread_cond_wait.c:638
#3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
#4  0x00007fd73ab4b833 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fd72bd6d700 (LWP 1290769)):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fd72bd6c6f0, clockid=735495808, expected=0, futex_word=0x7fd6d41bef30) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x7fd72bd6c6f0, clockid=735495808, mutex=0x7fd6d41bee40, cond=0x7fd6d41bef08) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7fd6d41bef08, mutex=0x7fd6d41bee40, abstime=0x7fd72bd6c6f0) at pthread_cond_wait.c:656
#3  0x00007fd73ab4ad67 in virCondWaitUntil () from /lib64/libvirt.so.0
#4  0x00007fd721a430dc in qemuDomainObjBeginJobInternal () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007fd721a47605 in qemuDomainObjBeginJob () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007fd721ada74d in qemuDomainShutdownFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007fd73ad03df7 in virDomainShutdown () from /lib64/libvirt.so.0
#8  0x000055c1ed6ad781 in remoteDispatchDomainShutdownHelper ()
#9  0x00007fd73ac2f4d0 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#10 0x00007fd73ac3458c in virNetServerHandleJob () from /lib64/libvirt.so.0
#11 0x00007fd73ab4b74f in virThreadPoolWorker () from /lib64/libvirt.so.0
#12 0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
#13 0x00007fd73a7da432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#14 0x00007fd73a6ed913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fd738d61b40 (LWP 1290768)):
#0  0x00007fd73a6e2aaf in __GI___poll (fds=0x55c1eda353d0, nfds=11, timeout=5000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fd73a878aae in g_main_context_poll (priority=<optimized out>, n_fds=11, fds=0x55c1eda353d0, timeout=<optimized out>, context=0x55c1eda34c90) at ../glib/gmain.c:4346
#2  g_main_context_iterate (context=context@entry=0x55c1eda34c90, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4042
#3  0x00007fd73a878be3 in g_main_context_iteration (context=0x55c1eda34c90, may_block=1) at ../glib/gmain.c:4108
#4  0x00007fd73aaf0e44 in virEventGLibRunOnce () from /lib64/libvirt.so.0
#5  0x00007fd73ac33f35 in virNetDaemonRun () from /lib64/libvirt.so.0
#6  0x000055c1ed69e7bc in main ()
Comment 24 Michal Privoznik 2020-09-02 11:33:04 UTC 
(In reply to Germano Massullo from comment #23)
> libvirt-6.1.0-4.fc32.x86_64
> virt-manager-2.2.1-3.fc32.noarch
> 
> virt-manager was not responding properly so I tried to run
> # virsh shutdown vm
> but after some minutes of not being responsive I got error
> 
> errore: Timed out during operation: cannot acquire state change lock (held
> by monitor=qemuProcessReconnect)
> 
> this is the GDB trace of libvirtd process that spawned when I ran the virsh
> command
> 
> Thread 18 (Thread 0x7fd6d2cf5700 (LWP 1290844)):
> #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fd6c4002aa0)
> at ../sysdeps/nptl/futex-internal.h:183
> #1  __pthread_cond_wait_common (abstime=0x0, clockid=0,
> mutex=0x7fd6c4002a50, cond=0x7fd6c4002a78) at pthread_cond_wait.c:508
> #2  __pthread_cond_wait (cond=0x7fd6c4002a78, mutex=0x7fd6c4002a50) at
> pthread_cond_wait.c:638
> #3  0x00007fd73ab4aceb in virCondWait () from /lib64/libvirt.so.0
> #4  0x00007fd721a9cc24 in qemuMonitorSend () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #5  0x00007fd721aabba8 in qemuMonitorJSONCommandWithFd () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #6  0x00007fd721aae435 in qemuMonitorJSONSetCapabilities () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #7  0x00007fd721a7bf7c in qemuConnectMonitor () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #8  0x00007fd721a880c9 in qemuProcessReconnect () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #9  0x00007fd73ab4aa9a in virThreadHelper () from /lib64/libvirt.so.0
> #10 0x00007fd73a7da432 in start_thread (arg=<optimized out>) at
> pthread_create.c:477
> #11 0x00007fd73a6ed913 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> 

So this thread is trying to reconnect to the domain. It has sent something on the monitor and is now waiting for the reply (my guess is it's switching the monitor to QMP). And talking on the monitor requires that the thread acquired a job first (not seen in this stack trace though).

> 
> Thread 2 (Thread 0x7fd72bd6d700 (LWP 1290769)):
> #0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fd72bd6c6f0,
> clockid=735495808, expected=0, futex_word=0x7fd6d41bef30) at
> ../sysdeps/nptl/futex-internal.h:320
> #1  __pthread_cond_wait_common (abstime=0x7fd72bd6c6f0, clockid=735495808,
> mutex=0x7fd6d41bee40, cond=0x7fd6d41bef08) at pthread_cond_wait.c:520
> #2  __pthread_cond_timedwait (cond=0x7fd6d41bef08, mutex=0x7fd6d41bee40,
> abstime=0x7fd72bd6c6f0) at pthread_cond_wait.c:656
> #3  0x00007fd73ab4ad67 in virCondWaitUntil () from /lib64/libvirt.so.0
> #4  0x00007fd721a430dc in qemuDomainObjBeginJobInternal () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #5  0x00007fd721a47605 in qemuDomainObjBeginJob () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #6  0x00007fd721ada74d in qemuDomainShutdownFlags () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #7  0x00007fd73ad03df7 in virDomainShutdown () from /lib64/libvirt.so.0
> #8  0x000055c1ed6ad781 in remoteDispatchDomainShutdownHelper ()
> #9  0x00007fd73ac2f4d0 in virNetServerProgramDispatch () from
> /lib64/libvirt.so.0
> #10 0x00007fd73ac3458c in virNetServerHandleJob () from /lib64/libvirt.so.0
> #11 0x00007fd73ab4b74f in virThreadPoolWorker () from /lib64/libvirt.so.0
> #12 0x00007fd73ab4aa6c in virThreadHelper () from /lib64/libvirt.so.0
> #13 0x00007fd73a7da432 in start_thread (arg=<optimized out>) at
> pthread_create.c:477
> #14 0x00007fd73a6ed913 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> 

And this thread tries to do the shutdown. And before changing the state of the domain it is trying to acquire a job (to exclude with other threads that might want to change the state too; and by state I do not only mean running vs shutoff, just any modification on the domain - hotplug, arbitrary guest agent command, etc.). But it can't acquire it, because the other thread is still reconnecting to the monitor.

Therefore, this is completely different issue than this bug was resolving and hence I'm closing it again. But, you may want to attach gdb to the qemu process to see what state it's in - I'm guessing it's in D state since it's not even replying on the monitor, but that's my guess.
Comment 25 Germano Massullo (Thetra) 2020-10-02 17:05:35 UTC 
Today I experienced the problem again and I got the qemu-system-x86-4.2.1-1.fc32.x86_64 GDB output. Do you see anything interesting?


# gdb qemu-system-x86_64 89616
GNU gdb (GDB) Fedora 9.1-6.fc32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qemu-system-x86_64...
Missing separate debuginfo for /usr/bin/qemu-system-x86_64
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/67/ec19da8ebdf6ffcd2cb28de7542ad404ae5111.debug
Reading symbols from .gnu_debugdata for /usr/bin/qemu-system-x86_64...
(No debugging symbols found in .gnu_debugdata for /usr/bin/qemu-system-x86_64)
Attaching to program: /usr/bin/qemu-system-x86_64, process 89616
[New LWP 89633]
[New LWP 89637]
[New LWP 89638]
[New LWP 89639]
[New LWP 89640]
[New LWP 89641]
[New LWP 89644]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
__lll_lock_wait (futex=futex@entry=0x56542ba43c28, private=0) at lowlevellock.c:52
52            lll_futex_wait (futex, 2, private); /* Wait if *futex == 2.  */
(gdb) set height 0
(gdb) set print elements 0
(gdb) set print frame-arguments all
(gdb) thread apply all backtrace

Thread 8 (Thread 0x7f7ead9ff700 (LWP 89644)):
#0  0x00007f7fd3ee9aaf in __GI___poll (fds=0x7f7ea8007210, nfds=2, timeout=2147483647) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f7fd51c9aae in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7f7ea8007210, timeout=<optimized out>, context=0x56542d74a500) at ../glib/gmain.c:4346
#2  g_main_context_iterate (context=0x56542d74a500, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4042
#3  0x00007f7fd51c9e33 in g_main_loop_run (loop=loop@entry=0x7f7ea80024c0) at ../glib/gmain.c:4241
#4  0x00007f7fd4a822db in red_worker_main (arg=0x56542d74a450) at red-worker.c:1131
#5  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f7fc9e2e700 (LWP 89641)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bc3c718) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40, cond=0x56542bc3c6f0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bc3c6f0, mutex=0x56542a314f40) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl ()
#4  0x0000565429a2c2a8 in qemu_wait_io_event ()
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn ()
#6  0x0000565429e25683 in qemu_thread_start ()
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f7fca62f700 (LWP 89640)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bc14b5c) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40, cond=0x56542bc14b30) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bc14b30, mutex=0x56542a314f40) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl ()
#4  0x0000565429a2c2a8 in qemu_wait_io_event ()
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn ()
#6  0x0000565429e25683 in qemu_thread_start ()
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f7fcae30700 (LWP 89639)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bbed0dc) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40, cond=0x56542bbed0b0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bbed0b0, mutex=0x56542a314f40) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl ()
#4  0x0000565429a2c2a8 in qemu_wait_io_event ()
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn ()
#6  0x0000565429e25683 in qemu_thread_start ()
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f7fcb631700 (LWP 89638)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bb9a01c) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40, cond=0x56542bb99ff0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bb99ff0, mutex=0x56542a314f40) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl ()
#4  0x0000565429a2c2a8 in qemu_wait_io_event ()
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn ()
#6  0x0000565429e25683 in qemu_thread_start ()
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f7fcbe32700 (LWP 89637)):
#0  0x00007f7fd3ee9aaf in __GI___poll (fds=0x7f7fc0003180, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f7fd51c9aae in g_main_context_poll (priority=<optimized out>, n_fds=3, fds=0x7f7fc0003180, timeout=<optimized out>, context=0x56542bb5b420) at ../glib/gmain.c:4346
#2  g_main_context_iterate (context=0x56542bb5b420, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4042
#3  0x00007f7fd51c9e33 in g_main_loop_run (loop=0x56542bb5b510) at ../glib/gmain.c:4241
#4  0x0000565429b50fb1 in iothread_run ()
#5  0x0000565429e25683 in qemu_thread_start ()
#6  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f7fd2620700 (LWP 89633)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000565429e25fd2 in qemu_event_wait ()
#2  0x0000565429e388c2 in call_rcu_thread ()
#3  0x0000565429e25683 in qemu_thread_start ()
#4  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f7fd2623ec0 (LWP 89616)):
#0  __lll_lock_wait (futex=futex@entry=0x56542ba43c28, private=0) at lowlevellock.c:52
#1  0x00007f7fd3fc8e53 in __GI___pthread_mutex_lock (mutex=0x56542ba43c28) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000565429e2576d in qemu_mutex_lock_impl ()
#3  0x0000565429db64d2 in qemu_chr_write_buffer ()
#4  0x0000565429db6b24 in qemu_chr_write ()
#5  0x0000565429ca1a8b in usbredir_write ()
#6  0x00007f7fd516fe3b in usbredirparser_do_write (parser_pub=0x56542c105800) at usbredirparser.c:1067
#7  0x0000565429b6b1b1 in vmc_write ()
#8  0x00007f7fd4a4377b in red_char_device_write_to_device (dev=dev@entry=0x56542bb59990) at char-device.c:482
#9  0x00007f7fd4a441ea in red_char_device_write_to_device (dev=0x56542bb59990) at char-device.c:326
#10 red_char_device_wakeup (dev=0x56542bb59990) at char-device.c:843
#11 0x0000565429b6b4e4 in spice_chr_write ()
#12 0x0000565429db650e in qemu_chr_write_buffer ()
#13 0x0000565429db6b24 in qemu_chr_write ()
#14 0x0000565429ca1a8b in usbredir_write ()
#15 0x00007f7fd516fe3b in usbredirparser_do_write (parser_pub=0x56542c105800) at usbredirparser.c:1067
#16 0x0000565429b57042 in vm_state_notify ()
#17 0x0000565429a2eda3 in vm_prepare_start ()
#18 0x0000565429a2eded in vm_start ()
#19 0x0000565429cd7cf5 in qmp_cont ()
#20 0x0000565429cfe196 in qmp_marshal_cont ()
#21 0x0000565429dd62c0 in qmp_dispatch ()
#22 0x0000565429cd46ed in monitor_qmp_dispatch ()
#23 0x0000565429cd4dac in monitor_qmp_bh_dispatcher ()
#24 0x0000565429e1fdac in aio_bh_poll ()
#25 0x0000565429e23494 in aio_dispatch ()
#26 0x0000565429e1fa32 in aio_ctx_dispatch ()
#27 0x00007f7fd51c978f in g_main_dispatch (context=0x56542ba94830) at ../glib/gmain.c:3309
#28 g_main_context_dispatch (context=0x56542ba94830) at ../glib/gmain.c:3974
#29 0x0000565429e226b8 in main_loop_wait ()
#30 0x0000565429b572af in main_loop ()
#31 0x00005654299de79c in main ()
(gdb)
Comment 26 Germano Massullo (Thetra) 2020-10-02 17:10:09 UTC 
In comment https://bugzilla.redhat.com/show_bug.cgi?id=1530346#c25 there was a debuginfo missing, the following backtrace is more complete

# gdb qemu-system-x86_64 89616
GNU gdb (GDB) Fedora 9.1-6.fc32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from qemu-system-x86_64...
Reading symbols from /usr/lib/debug/usr/bin/qemu-system-x86_64-4.2.1-1.fc32.x86_64.debug...
Attaching to program: /usr/bin/qemu-system-x86_64, process 89616
[New LWP 89633]
[New LWP 89637]
[New LWP 89638]
[New LWP 89639]
[New LWP 89640]
[New LWP 89641]
[New LWP 89644]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
__lll_lock_wait (futex=futex@entry=0x56542ba43c28, private=0) at lowlevellock.c:52
52            lll_futex_wait (futex, 2, private); /* Wait if *futex == 2.  */
(gdb) set height 0
(gdb) set print elements 0
(gdb) set print frame-arguments all
(gdb) thread apply all backtrace

Thread 8 (Thread 0x7f7ead9ff700 (LWP 89644)):
#0  0x00007f7fd3ee9aaf in __GI___poll (fds=0x7f7ea8007210, nfds=2, timeout=2147483647) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f7fd51c9aae in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7f7ea8007210, timeout=<optimized out>, context=0x56542d74a500) at ../glib/gmain.c:4346
#2  g_main_context_iterate (context=0x56542d74a500, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4042
#3  0x00007f7fd51c9e33 in g_main_loop_run (loop=loop@entry=0x7f7ea80024c0) at ../glib/gmain.c:4241
#4  0x00007f7fd4a822db in red_worker_main (arg=0x56542d74a450) at red-worker.c:1131
#5  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f7fc9e2e700 (LWP 89641)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bc3c718) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40 <qemu_global_mutex>, cond=0x56542bc3c6f0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bc3c6f0, mutex=mutex@entry=0x56542a314f40 <qemu_global_mutex>) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl (cond=<optimized out>, mutex=0x56542a314f40 <qemu_global_mutex>, file=0x565429e4b2d0 "/builddir/build/BUILD/qemu-4.2.1/cpus.c", line=1275) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:173
#4  0x0000565429a2c2a8 in qemu_wait_io_event (cpu=cpu@entry=0x56542bc15360) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1275
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn (arg=0x56542bc15360) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1323
#6  0x0000565429e25683 in qemu_thread_start (args=0x56542bc3c730) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:519
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f7fca62f700 (LWP 89640)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bc14b5c) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40 <qemu_global_mutex>, cond=0x56542bc14b30) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bc14b30, mutex=mutex@entry=0x56542a314f40 <qemu_global_mutex>) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl (cond=<optimized out>, mutex=0x56542a314f40 <qemu_global_mutex>, file=0x565429e4b2d0 "/builddir/build/BUILD/qemu-4.2.1/cpus.c", line=1275) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:173
#4  0x0000565429a2c2a8 in qemu_wait_io_event (cpu=cpu@entry=0x56542bbed8e0) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1275
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn (arg=0x56542bbed8e0) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1323
#6  0x0000565429e25683 in qemu_thread_start (args=0x56542bc14b70) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:519
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f7fcae30700 (LWP 89639)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bbed0dc) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40 <qemu_global_mutex>, cond=0x56542bbed0b0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bbed0b0, mutex=mutex@entry=0x56542a314f40 <qemu_global_mutex>) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl (cond=<optimized out>, mutex=0x56542a314f40 <qemu_global_mutex>, file=0x565429e4b2d0 "/builddir/build/BUILD/qemu-4.2.1/cpus.c", line=1275) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:173
#4  0x0000565429a2c2a8 in qemu_wait_io_event (cpu=cpu@entry=0x56542bbc50c0) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1275
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn (arg=0x56542bbc50c0) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1323
#6  0x0000565429e25683 in qemu_thread_start (args=0x56542bbed0f0) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:519
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f7fcb631700 (LWP 89638)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x56542bb9a01c) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56542a314f40 <qemu_global_mutex>, cond=0x56542bb99ff0) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x56542bb99ff0, mutex=mutex@entry=0x56542a314f40 <qemu_global_mutex>) at pthread_cond_wait.c:638
#3  0x0000565429e25a3d in qemu_cond_wait_impl (cond=<optimized out>, mutex=0x56542a314f40 <qemu_global_mutex>, file=0x565429e4b2d0 "/builddir/build/BUILD/qemu-4.2.1/cpus.c", line=1275) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:173
#4  0x0000565429a2c2a8 in qemu_wait_io_event (cpu=cpu@entry=0x56542bb75b00) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1275
#5  0x0000565429a2da80 in qemu_kvm_cpu_thread_fn (arg=0x56542bb75b00) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:1323
#6  0x0000565429e25683 in qemu_thread_start (args=0x56542bb9a030) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:519
#7  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f7fcbe32700 (LWP 89637)):
#0  0x00007f7fd3ee9aaf in __GI___poll (fds=0x7f7fc0003180, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f7fd51c9aae in g_main_context_poll (priority=<optimized out>, n_fds=3, fds=0x7f7fc0003180, timeout=<optimized out>, context=0x56542bb5b420) at ../glib/gmain.c:4346
#2  g_main_context_iterate (context=0x56542bb5b420, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4042
#3  0x00007f7fd51c9e33 in g_main_loop_run (loop=0x56542bb5b510) at ../glib/gmain.c:4241
#4  0x0000565429b50fb1 in iothread_run (opaque=0x56542ba6c000) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/iothread.c:82
#5  0x0000565429e25683 in qemu_thread_start (args=0x56542bb5b550) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:519
#6  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f7fd2620700 (LWP 89633)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000565429e25fd2 in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x56542a34ec68 <rcu_call_ready_event>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:459
#3  0x0000565429e388c2 in call_rcu_thread (opaque=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/rcu.c:260
#4  0x0000565429e25683 in qemu_thread_start (args=0x56542b93ee10) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:519
#5  0x00007f7fd3fc6432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f7fd3ef4913 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f7fd2623ec0 (LWP 89616)):
#0  __lll_lock_wait (futex=futex@entry=0x56542ba43c28, private=0) at lowlevellock.c:52
#1  0x00007f7fd3fc8e53 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x56542ba43c28) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000565429e2576d in qemu_mutex_lock_impl (mutex=0x56542ba43c28, file=0x565429f66bb0 "/builddir/build/BUILD/qemu-4.2.1/chardev/char.c", line=111) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/qemu-thread-posix.c:78
#3  0x0000565429db64d2 in qemu_chr_write_buffer (s=s@entry=0x56542ba43c00, buf=buf@entry=0x56542c06c5e0 "", len=80, offset=offset@entry=0x7ffcda183a60, write_all=write_all@entry=false) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/char.c:111
#4  0x0000565429db6b24 in qemu_chr_write (s=0x56542ba43c00, buf=buf@entry=0x56542c06c5e0 "", len=len@entry=80, write_all=write_all@entry=false) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/char.c:149
#5  0x0000565429db8e83 in qemu_chr_fe_write (be=be@entry=0x56542d779880, buf=buf@entry=0x56542c06c5e0 "", len=len@entry=80) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/char-fe.c:42
#6  0x0000565429ca1a8b in usbredir_write (priv=0x56542d7781d0, data=0x56542c06c5e0 "", count=80) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/hw/usb/redirect.c:293
#7  0x00007f7fd516fe3b in usbredirparser_do_write (parser_pub=0x56542c105800) at usbredirparser.c:1067
#8  0x0000565429b6b1b1 in vmc_write (sin=<optimized out>, buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/spice.c:34
#9  0x00007f7fd4a4377b in red_char_device_write_to_device (dev=dev@entry=0x56542bb59990) at char-device.c:482
#10 0x00007f7fd4a441ea in red_char_device_write_to_device (dev=0x56542bb59990) at char-device.c:326
#11 red_char_device_wakeup (dev=0x56542bb59990) at char-device.c:843
#12 0x00007f7fd4a7c735 in spice_server_char_device_wakeup (sin=sin@entry=0x56542ba43c90) at reds.c:3200
#13 0x0000565429b6b4e4 in spice_chr_write (chr=0x56542ba43c00, buf=0x56542c06c5e0 "", len=80) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/spice.c:202
#14 0x0000565429db650e in qemu_chr_write_buffer (s=s@entry=0x56542ba43c00, buf=buf@entry=0x56542c06c5e0 "", len=80, offset=offset@entry=0x7ffcda183c10, write_all=write_all@entry=false) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/char.c:114
#15 0x0000565429db6b24 in qemu_chr_write (s=0x56542ba43c00, buf=buf@entry=0x56542c06c5e0 "", len=len@entry=80, write_all=write_all@entry=false) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/char.c:149
#16 0x0000565429db8e83 in qemu_chr_fe_write (be=be@entry=0x56542d779880, buf=buf@entry=0x56542c06c5e0 "", len=len@entry=80) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/chardev/char-fe.c:42
#17 0x0000565429ca1a8b in usbredir_write (priv=0x56542d7781d0, data=0x56542c06c5e0 "", count=80) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/hw/usb/redirect.c:293
#18 0x00007f7fd516fe3b in usbredirparser_do_write (parser_pub=0x56542c105800) at usbredirparser.c:1067
#19 0x0000565429b57042 in vm_state_notify (running=running@entry=1, state=state@entry=RUN_STATE_RUNNING) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/vl.c:1431
#20 0x0000565429a2eda3 in vm_prepare_start () at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:2207
#21 0x0000565429a2eded in vm_start () at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/cpus.c:2213
#22 0x0000565429cd7cf5 in qmp_cont (errp=errp@entry=0x7ffcda183d20) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/monitor/qmp-cmds.c:173
#23 0x0000565429cfe196 in qmp_marshal_cont (args=<optimized out>, ret=<optimized out>, errp=0x7ffcda183d78) at qapi/qapi-commands-misc.c:835
#24 0x0000565429dd62c0 in do_qmp_dispatch (errp=0x7ffcda183d70, allow_oob=<optimized out>, request=<optimized out>, cmds=0x56542a34c580 <qmp_commands>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/qapi/qmp-dispatch.c:132
#25 qmp_dispatch (cmds=0x56542a34c580 <qmp_commands>, request=<optimized out>, allow_oob=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/qapi/qmp-dispatch.c:175
#26 0x0000565429cd46ed in monitor_qmp_dispatch (mon=0x56542bb5ae90, req=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/monitor/qmp.c:145
#27 0x0000565429cd4dac in monitor_qmp_bh_dispatcher (data=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/monitor/qmp.c:234
#28 0x0000565429e1fdac in aio_bh_call (bh=0x56542b9b4030) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/async.c:117
#29 aio_bh_poll (ctx=ctx@entry=0x56542b9b2be0) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/async.c:117
#30 0x0000565429e23494 in aio_dispatch (ctx=ctx@entry=0x56542b9b2be0) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/aio-posix.c:459
#31 0x0000565429e1fa32 in aio_ctx_dispatch (source=source@entry=0x56542b9b2be0, callback=<optimized out>, user_data=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/async.c:260
#32 0x00007f7fd51c978f in g_main_dispatch (context=0x56542ba94830) at ../glib/gmain.c:3309
#33 g_main_context_dispatch (context=0x56542ba94830) at ../glib/gmain.c:3974
#34 0x0000565429e226b8 in glib_pollfds_poll () at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/main-loop.c:219
#35 os_host_main_loop_wait (timeout=520630118) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/main-loop.c:242
#36 main_loop_wait (nonblocking=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/util/main-loop.c:518
#37 0x0000565429b572af in main_loop () at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/vl.c:1812
#38 0x00005654299de79c in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /usr/src/debug/qemu-4.2.1-1.fc32.x86_64/vl.c:4473
Comment 27 Germano Massullo (Thetra) 2020-10-02 18:01:07 UTC 
in #qemu OFTC IRC channel I have been told that qemu is waiting for USB redirection, but I have not redirected any USB device on that VM, or other VM while the problem happened. No snapshots, it's just a very simply CentOS virtual machine with no default settings changed.

Everything started after getting error "Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainResume)" on virt-manager VM window. The VM logs are fine, even host dmesg.
I successfully managed to start and open a virt-manager window of another "VM-2", while still having the problem with "VM"
Comment 28 Michal Privoznik 2020-10-06 12:15:14 UTC 
(In reply to Germano Massullo from comment #27)
> in #qemu OFTC IRC channel I have been told that qemu is waiting for USB
> redirection, but I have not redirected any USB device on that VM, or other
> VM while the problem happened. No snapshots, it's just a very simply CentOS
> virtual machine with no default settings changed.

Maybe it's added automatically by virt-install. May be worth dumping the domain XML (virsh dumpxml).

But the stack trace looks dangerously close to the one mentioned in this commit message:

https://git.qemu.org/?p=qemu.git;a=commit;h=394642a8d3742c885e397d5bb5ee0ec40743cdc6

which is contained in qemu-5.0.0 release. Anyway, if qemu deadlocks there is not much libvirt can do and any state changing operation must fail (state in broader sense - not just running/paused/...). Can you please upgrade your QEMU and see if that fixes your problem?
Comment 29 Germano Massullo (Thetra) 2020-10-06 12:23:13 UTC 
(In reply to Michal Privoznik from comment #28)
> Can you please upgrade your QEMU and see if that fixes your problem?

I use the lastest in Fedora 32 release. When I will update to Fedora 33 I can test the 5.x version

> Maybe it's added automatically by virt-install. May be worth dumping the
> domain XML (virsh dumpxml).

<domain type='kvm'>
  <name>centos8</name>
  <uuid>**removed**</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://centos.org/centos/8"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4096000</memory>
  <currentMemory unit='KiB'>4096000</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/centos8.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/centos8-1.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/centos8-2.qcow2'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/centos8-3.qcow2'/>
      <target dev='vdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/centos8-4.qcow2'/>
      <target dev='vde' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:d4:02:c2'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
</domain>
Comment 30 Michal Privoznik 2020-10-06 20:07:40 UTC 
(In reply to Germano Massullo from comment #29)
> (In reply to Michal Privoznik from comment #28)
> > Can you please upgrade your QEMU and see if that fixes your problem?
> 
> I use the lastest in Fedora 32 release. When I will update to Fedora 33 I
> can test the 5.x version

Alright. I've backported the patch on the top of Fedora 32, if you want to give it a try:

https://koji.fedoraproject.org/koji/taskinfo?taskID=52863336

> 
> > Maybe it's added automatically by virt-install. May be worth dumping the
> > domain XML (virsh dumpxml).
> 
> <domain type='kvm'>

>     <redirdev bus='usb' type='spicevmc'>
>       <address type='usb' bus='0' port='2'/>
>     </redirdev>
>     <redirdev bus='usb' type='spicevmc'>
>       <address type='usb' bus='0' port='3'/>
>     </redirdev>
> </domain>

But the redirdevs are here.
Comment 31 Germano Massullo (Thetra) 2020-10-06 20:24:52 UTC 
(In reply to Michal Privoznik from comment #30)
> (In reply to Germano Massullo from comment #29)
> > (In reply to Michal Privoznik from comment #28)
> > > Can you please upgrade your QEMU and see if that fixes your problem?
> > 
> > I use the lastest in Fedora 32 release. When I will update to Fedora 33 I
> > can test the 5.x version
> 
> Alright. I've backported the patch on the top of Fedora 32, if you want to
> give it a try:
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=52863336

Should I restart other services besides libvirtd ?
Comment 32 Michal Privoznik 2020-10-07 03:30:22 UTC 
(In reply to Germano Massullo from comment #31)
> Should I restart other services besides libvirtd ?

No, it should be enough to install only this newer QEMU. Libvirt realizes that the next time you try to start a guest.
Comment 33 Germano Massullo (Thetra) 2020-12-02 08:42:05 UTC 
Cleaning needinfo request since at the moment I am no longer able to reproduce the problem