Bug 2213257

Summary:	virtqemud seems to die with I/O error
Product:	[Fedora] Fedora	Reporter:	Shane Hart <glanzick>
Component:	libvirt	Assignee:	Libvirt Maintainers <libvirt-maint>
Status:	CLOSED DUPLICATE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	38	CC:	berrange, clalancette, cristian.ciupitu, crobinso, dan, jforbes, laine, libvirt-maint, virt-maint, zbyszek
Target Milestone:	---	Keywords:	Regression, Upgrades
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-06-27 15:58:00 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Shane Hart 2023-06-07 16:18:21 UTC

Updated to systemd 253.5 today and libvirtd/virtqemud seem to be dying and causing virsh commands to hang.

I generally spin up new VMs with virt-install and use virsh domifaddr to get the IP after they boot.  They boot fine, run my given task to completion and then do cleanup where they query virsh again.  At this point they hang and virtqemud log says:

Jun 07 11:02:00 cloud-dev01 systemd[1]: Starting virtqemud.service - Virtualization qemu daemon...
Jun 07 11:02:00 cloud-dev01 systemd[1]: Started virtqemud.service - Virtualization qemu daemon.
Jun 07 11:58:49 cloud-dev01 virtqemud[3817]: libvirt version: 9.0.0, package: 3.fc38 (Fedora Project, 2023-05-07-17:31:32, )
Jun 07 11:58:49 cloud-dev01 virtqemud[3817]: hostname: cloud-dev01
Jun 07 11:58:49 cloud-dev01 virtqemud[3817]: End of file while reading data: Input/output error
Jun 07 11:59:52 cloud-dev01 virtqemud[3817]: End of file while reading data: Input/output error

Restarting the libvirtd/virtqemud goes back to the beginning of the process.  I have also tried removing the socket activation from libvirtd.

Reproducible: Always

Steps to Reproduce:
1. Install new VM using vert-install.
2. Boot VM and use for a while (20-30 minutes).
3. Use any virsh command and have it hang.
Actual Results:  
virsh command hangs and virtqemud gives I/O error.

Expected Results:  
virsh command should work.

Comment 1 Shane Hart 2023-06-07 17:56:05 UTC

I downgraded my systemd packages to 253.4 and we're back to not hanging, so that's good at least.

Comment 2 Shane Hart 2023-06-07 22:49:18 UTC

I've added links to systemd and libvirt's issue trackers where someone else has reported this problem.

Comment 3 Christian Labisch 2023-06-08 12:00:07 UTC

I can confirm what Shane describes ... exactly the same thing here : systemd 253.5 is the culprit - after downgrading the systemd packages everything works as expected again.
@Shane : Can you add the component "systemd" to your bug report ? I am not able to do this. Another related bug report -> https://bugzilla.redhat.com/show_bug.cgi?id=2213398

Comment 4 Shane Hart 2023-06-08 13:11:57 UTC

I cannot add a second component it seems; sorry, I'm not super familiar with BugZilla.   It's possible that this could just be a systemd bug instead, there seems to be some confusion on the matter.

Comment 5 Cristian Ciupitu 2023-06-08 22:21:30 UTC

I seem to have the same issue with:
systemd-253.5-1.fc38.x86_64
libvirt-daemon-driver-qemu-9.2.0-1.fc38.x86_64

Comment 6 Shane Hart 2023-06-22 15:22:23 UTC

It looks like a PR has been merged upstream.  I don't know the etiquette on this, whether I should leave this open until it gets to Fedora or just close now.

Comment 7 Dan Horák 2023-06-27 15:58:00 UTC


*** This bug has been marked as a duplicate of bug 2213660 ***