RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2078693 - libvirt can not identify shared memory after restarting virtqemud
Summary: libvirt can not identify shared memory after restarting virtqemud
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: 9.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 9.3
Assignee: Ján Tomko
QA Contact: liang cong
Jiri Herrmann
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-26 03:15 UTC by yafu
Modified: 2023-11-07 09:38 UTC (History)
12 users (show)

Fixed In Version: libvirt-9.2.0-1.el9
Doc Type: Bug Fix
Doc Text:
.`virtiofs` devices could not be attached after restarting `virtqemud` or `libvirtd` Previously, restarting the `virtqemud` or `libvirtd` services prevented `virtiofs` storage devices from being attached to virtual machines (VMs) on your host. This bug has been fixed, and you can now attach `virtiofs` devices in the described scenario as expected.
Clone Of:
Environment:
Last Closed: 2023-11-07 08:30:47 UTC
Type: Bug
Target Upstream Version: 9.2.0
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 2177701 0 medium CLOSED Can't hotplug virtiofs device after restarting libvirt daemon 2023-11-14 16:59:58 UTC
Red Hat Issue Tracker LIBVIRTAT-14495 0 None None None 2023-07-04 01:52:58 UTC
Red Hat Issue Tracker RHELPLAN-119912 0 None None None 2022-04-26 03:15:55 UTC
Red Hat Product Errata RHSA-2023:6409 0 None None None 2023-11-07 08:31:21 UTC

Internal Links: 2177701

Description yafu 2022-04-26 03:15:05 UTC
Description of problem:
libvirt can not identify shared memory after restarting virtqemud

Version-Release number of selected component (if applicable):
libvirt-8.2.0-1.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Start a guest with shared memory:
<domain>
...
<memoryBacking>
    <access mode='shared'/>
  </memoryBacking>
...

2.Prepare a virtiofs device xml:
#cat fs.xml
<filesystem accessmode="passthrough" type="mount">
	<driver queue="512" type="virtiofs"/>
	<source dir="/var/tmp/mount_tag0"/>
	<target dir="mount_tag0"/>
	<alias name="ua-05e63df0-c4ab-11ec-ba3d-fa163ed3db7d"/>
	<binary path="/usr/libexec/virtiofsd" xattr="on">
		<cache mode="none"/>
               <lock flock="off" posix="off"/>
</binary>
</filesystem>


3.Create the monut dir:
#mkdir /var/tmp/mount_tag0

4.Attach the device to the vm:
# virsh attach-device avocado-vt-vm1 fs.xml 
Device attached successfully

5.Detach virtiofs device from the vm:
# virsh detach-device avocado-vt-vm1 fs.xml 
Device detached successfully

6.Restart virtqemud:
#systemctl restart virtqemud

7.Attach virtiofs devcie to the vm again:
# virsh attach-device avocado-vt-vm1 fs.xml 
error: Failed to attach device from fs.xml
error: unsupported configuration: 'virtiofs' requires shared memory

8.Check the live vm xml and qemu cmd line, shared memory still exists:
#virsh dumpxml avocado-vt-vm1
<domain>
...
<memoryBacking>
    <access mode='shared'/>
  </memoryBacking>
...

#ps aux | grep shared
qemu      267628 73.4 18.9 3534168 761076 ?      Sl   03:00   1:36 /usr/bin/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on ... -m 2048 -object memory-backend-file,id=pc.ram,mem-path=/var/lib/libvirt/qemu/ram/6-avocado-vt-vm1/pc.ram,***share=on***


Actual results:
libvirt can not identify shared memory after restarting virtqemud, so can not attach the virtiofs device to vm after restarting virtqemud

Expected results:
libvirt should always identify shared memory.

Additional info:

Comment 1 Ján Tomko 2023-03-13 14:10:42 UTC
Upstream patch:
https://listman.redhat.com/archives/libvir-list/2023-March/238702.html

Comment 2 Ján Tomko 2023-03-14 16:15:02 UTC
Pushed upstream as:
commit d5c7b7870e45575f81fffcb611c2546d0e02e778
Author:     Ján Tomko <jtomko>
CommitDate: 2023-03-14 17:10:01 +0100

    qemu: relax shared memory check for vhostuser daemons
    
    For some vhostuser daemons, we validate that the guest memory is shared
    with the host.
    
    With earlier versions of QEMU, it was only possible to mark memory
    as shared by defining an explicit NUMA topology.  Later, QEMU exposed
    the name of the default memory backend (defaultRAMid) so we can mark
    that memory as shared.
    
    Since libvirt commit:
      commit bff2ad5d6b1f25da02802273934d2a519159fec7
        qemu: Relax validation for mem->access if guest has no NUMA
    we already check for the case when user requests shared memory,
    but QEMU did not expose defaultRAMid.
    
    Drop the duplicit check from vhostuser device validation, to make
    it pass on hotplug even after libvirtd restart.
    
    This avoids the need to store the defaultRAMid, since we don't really
    need it for anything after the VM has been already started.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2078693
    https://bugzilla.redhat.com/show_bug.cgi?id=2177701
    
    Signed-off-by: Ján Tomko <jtomko>
    Reviewed-by: Michal Privoznik <mprivozn>

git describe: v9.1.0-200-gd5c7b7870e

Comment 3 liang cong 2023-03-20 10:10:46 UTC
Preverified on upstream build: libvirt v9.1.0-248-g27d8bcc337

Test steps:
1.Start a guest with shared memory:
...
<memoryBacking>
    <access mode='shared'/>
  </memoryBacking>
...

2.Prepare a virtiofs device xml:
#cat fs.xml
<filesystem accessmode="passthrough" type="mount">
	<driver queue="512" type="virtiofs"/>
	<source dir="/var/tmp/mount_tag0"/>
	<target dir="mount_tag0"/>
	<alias name="ua-05e63df0-c4ab-11ec-ba3d-fa163ed3db7d"/>
	<binary path="/usr/libexec/virtiofsd" xattr="on">
		<cache mode="none"/>
               <lock flock="off" posix="off"/>
</binary>
</filesystem>


3.Create the monut dir:
#mkdir /var/tmp/mount_tag0

4.Attach the device to the vm:
# virsh attach-device vm1 fs.xml 
Device attached successfully

5.Detach virtiofs device from the vm:
# virsh detach-device vm1 fs.xml 
Device detached successfully

6.Restart virtqemud:
#systemctl restart virtqemud

7.Attach virtiofs devcie to the vm again:
# virsh attach-device vm1 fs.xml 
Device attached successfully

8. Detach virtiofs device from the vm again:
# virsh detach-device vm1 fs.xml 
Device detached successfully

Comment 4 Thomas Huth 2023-03-21 08:12:57 UTC
A similar ticket for RHEL8 can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=2177701

Comment 9 liang cong 2023-05-22 07:46:41 UTC
Verified on upstream build:
# rpm -q libvirt qemu-kvm
libvirt-9.3.0-2.el9.x86_64
qemu-kvm-8.0.0-3.el9.x86_64

Test steps:
1.Start a guest with shared memory:
...
<memoryBacking>
    <access mode='shared'/>
  </memoryBacking>
...

2.Prepare a virtiofs device xml:
#cat fs.xml
<filesystem accessmode="passthrough" type="mount">
	<driver queue="512" type="virtiofs"/>
	<source dir="/var/tmp/mount_tag0"/>
	<target dir="mount_tag0"/>
	<alias name="ua-05e63df0-c4ab-11ec-ba3d-fa163ed3db7d"/>
	<binary path="/usr/libexec/virtiofsd" xattr="on">
		<cache mode="none"/>
               <lock flock="off" posix="off"/>
</binary>
</filesystem>


3.Create the monut dir:
#mkdir /var/tmp/mount_tag0

4.Attach the device to the vm:
# virsh attach-device vm1 fs.xml 
Device attached successfully

5.Detach virtiofs device from the vm:
# virsh detach-device vm1 fs.xml 
Device detached successfully

6.Restart virtqemud:
#systemctl restart virtqemud

7.Attach virtiofs devcie to the vm again:
# virsh attach-device vm1 fs.xml 
Device attached successfully

8. Detach virtiofs device from the vm again:
# virsh detach-device vm1 fs.xml 
Device detached successfully

Comment 13 Ján Tomko 2023-05-23 12:55:27 UTC
It's a per-VM issue. When starting the VM, libvirt probes the QEMU capabilities, including whether it exposes the alias of the default memory device. Libvirt uses the alias to tell QEMU to mark the guest memory as shared. For older QEMU/libvirt, other methods of creating shared memory are used:
https://libvirt.org/kbase/virtiofs.html#other-options-for-vhost-user-memory-setup

Libvirt does not save the memory alias in the VM's status XML. After daemon restart, the QEMU capabilities are not probed again, but read from the status XML. The bug here is that the validation required either a memory alias, or one of the alternate ways of having shared memory. The fix removed the requirement for a memory alias, because this is already validated at VM startup.

The "workarounds" here are
a) do not restart the daemon
b) use a VM with one of the alternate setups of shared memory
c) restart the VM
to be able to hotplug a virtiofs <filesystem>

However none of them are applicable if you already have a running VM and restarted the daemon.

Comment 15 errata-xmlrpc 2023-11-07 08:30:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6409


Note You need to log in before you can comment on or make changes to this bug.