Bug 2177701

Summary: Can't hotplug virtiofs device after restarting libvirt daemon
Product: Red Hat Enterprise Linux 8 Reporter: smitterl
Component: libvirtAssignee: Ján Tomko <jtomko>
Status: CLOSED ERRATA QA Contact: Lili Zhu <lizhu>
Severity: high Docs Contact: Parth Shah <pashah>
Priority: medium    
Version: 8.8CC: bfiuczyn, clegoate, jdenemar, jherrman, jtomko, lcong, lmen, nanli, thuth, virt-maint
Target Milestone: rcKeywords: Triaged
Target Release: 8.9   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-8.0.0-20.module+el8.9.0+18894+a91bbb94 Doc Type: Known Issue
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-14 15:33:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 9.2.0
Embargoed:
Attachments:
Description Flags
domain definition none

Description smitterl 2023-03-13 12:34:12 UTC
Description of problem:
After restarting the libvirt daemon, can't hotplug virtiofs device


Version-Release number of selected component (if applicable):
libvirt-8.0.0-18.module+el8.8.0+18287+112bb4e6.s390x

How reproducible:
100%


Steps to Reproduce:
1. Start a VM with 
  <memoryBacking>
    <access mode='shared'/>
  </memoryBacking>

2. systemctl restart libvirtd
3. Hotplug virtiofs device
<filesystem accessmode="passthrough" type="mount">
 <driver queue="512" type="virtiofs" />
 <source dir="/var/tmp/mount_tag0" />
 <target dir="mount_tag0" />
 <alias name="ua-8d2edac2-c191-11ed-885e-6a94bb98894f" />
 <binary path="/usr/libexec/virtiofsd" xattr="on">
  <cache mode="none" />
  <lock flock="off" posix="off" />
 </binary>
</filesystem>

Actual results:
error: unsupported configuration: 'virtiofs' requires shared memory

Expected results:
The device is attached successfully

Additional info:
After destroy-start VM, the device can be attached successfully.

Comment 3 smitterl 2023-03-13 13:19:37 UTC
This does not reproduce with
libvirt-8.0.0-10.3.module+el8.7.0+18207+556790c8.s390x

Comment 4 smitterl 2023-03-13 13:24:56 UTC
It reproduces with libvirt-8.0.0-14.module+el8.8.0+17806+11a519fc.s390x

Comment 5 Ján Tomko 2023-03-13 13:52:17 UTC
Can you share the full domain XML that made it work for libvirt-8.0.0-10.3 on s390x?
Also, are you sure you restarted libvirtd before hotplug, when it worked, i.e. the older libvirt version?

The x86_64 test case uses <numa> to specify shared memory (at least the few cases I looked at).
Ever since this libvirt commit in libvirt 6.9.0 we also allow shared memory with implicit memory object ID:
https://gitlab.com/libvirt/libvirt/-/commit/e2425a17270d3c25730977d37b5679189714b913
but we did not save it on libvirt restart. And IIUC s390x has no NUMA. So I'm confused how this ever worked.

Comment 6 Ján Tomko 2023-03-13 14:10:28 UTC
Upstream patch:
https://listman.redhat.com/archives/libvir-list/2023-March/238702.html

Comment 7 smitterl 2023-03-13 14:36:10 UTC
(In reply to Ján Tomko from comment #5)
> Can you share the full domain XML that made it work for libvirt-8.0.0-10.3
> on s390x?
> Also, are you sure you restarted libvirtd before hotplug, when it worked,
> i.e. the older libvirt version?
> 
You're right, I can reproduce this with libvirt-8.0.0-10.3.module+el8.7.0+18207+556790c8.s390x after all.

> The x86_64 test case uses <numa> to specify shared memory (at least the few
> cases I looked at).
> Ever since this libvirt commit in libvirt 6.9.0 we also allow shared memory
> with implicit memory object ID:
> https://gitlab.com/libvirt/libvirt/-/commit/
> e2425a17270d3c25730977d37b5679189714b913
> but we did not save it on libvirt restart. And IIUC s390x has no NUMA. So
> I'm confused how this ever worked.

Comment 8 smitterl 2023-03-13 14:38:56 UTC
Created attachment 1950249 [details]
domain definition

Comment 9 Ján Tomko 2023-03-14 16:15:35 UTC
Pushed upstream as:
commit d5c7b7870e45575f81fffcb611c2546d0e02e778
Author:     Ján Tomko <jtomko>
CommitDate: 2023-03-14 17:10:01 +0100

    qemu: relax shared memory check for vhostuser daemons
    
    For some vhostuser daemons, we validate that the guest memory is shared
    with the host.
    
    With earlier versions of QEMU, it was only possible to mark memory
    as shared by defining an explicit NUMA topology.  Later, QEMU exposed
    the name of the default memory backend (defaultRAMid) so we can mark
    that memory as shared.
    
    Since libvirt commit:
      commit bff2ad5d6b1f25da02802273934d2a519159fec7
        qemu: Relax validation for mem->access if guest has no NUMA
    we already check for the case when user requests shared memory,
    but QEMU did not expose defaultRAMid.
    
    Drop the duplicit check from vhostuser device validation, to make
    it pass on hotplug even after libvirtd restart.
    
    This avoids the need to store the defaultRAMid, since we don't really
    need it for anything after the VM has been already started.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2078693
    https://bugzilla.redhat.com/show_bug.cgi?id=2177701
    
    Signed-off-by: Ján Tomko <jtomko>
    Reviewed-by: Michal Privoznik <mprivozn>

git describe: v9.1.0-200-gd5c7b7870e

Comment 10 Thomas Huth 2023-03-20 16:15:58 UTC
@smitterl : Do you think we still need to get this fixed in RHEL 8.8 ? ... since it does not seem to be a regression, I'd rather move it to 8.9 and lower the "severity" ?

Comment 11 smitterl 2023-03-21 08:01:49 UTC
(In reply to Thomas Huth from comment #10)
> @smitterl : Do you think we still need to get this fixed in RHEL
> 8.8 ? ... since it does not seem to be a regression, I'd rather move it to
> 8.9 and lower the "severity" ?

I agree. What's more, the corresponding RHEL 9 seems to be in the works for 9.3 https://bugzilla.redhat.com/show_bug.cgi?id=2078693

Comment 14 Lili Zhu 2023-05-18 13:49:33 UTC
Build the rpm with patch:
https://gitlab.com/redhat/rhel/src/libvirt/-/merge_requests/112

1. start the guest with
  <memoryBacking>
    <source type='file'/>
    <access mode='shared'/>
  </memoryBacking>

2. restart libvirtd

3. prepare the virtiofs device xml
# cat virtiofs.xml 
<filesystem accessmode="passthrough" type="mount">
 <driver queue="512" type="virtiofs" />
 <source dir="/var/tmp/mount_tag0" />
 <target dir="mount_tag0" />
 <alias name="ua-8d2edac2-c191-11ed-885e-6a94bb98894f" />
 <binary path="/usr/libexec/virtiofsd" xattr="on">
  <cache mode="none" />
  <lock flock="off" posix="off" />
 </binary>
</filesystem>

4. hotplug the virtiofs device
# virsh attach-device avocado-vt-vm1 virtiofs.xml 
Device attached successfully

Comment 18 Lili Zhu 2023-06-11 08:03:33 UTC
Verify this bug with:
libvirt-8.0.0-20.module+el8.9.0+18894+a91bbb94.x86_64

Verification steps are the same with those in Comment #14

Mark the bug as verified

Comment 19 liang cong 2023-06-12 06:58:22 UTC
Verified this bug on S390 build:
# rpm -q libvirt
libvirt-8.0.0-20.module+el8.9.0+18894+a91bbb94.s390x

Test steps:
1. Start a VM with 
  <memoryBacking>
    <access mode='shared'/>
  </memoryBacking>

2. systemctl restart libvirtd
3. Prepare a virtiofs config file virtiofs.xml
<filesystem accessmode="passthrough" type="mount">
 <driver queue="512" type="virtiofs" />
 <source dir="/var/tmp/mount_tag0" />
 <target dir="mount_tag0" />
 <alias name="ua-8d2edac2-c191-11ed-885e-6a94bb98894f" />
 <binary path="/usr/libexec/virtiofsd" xattr="on">
  <cache mode="none" />
  <lock flock="off" posix="off" />
 </binary>
</filesystem>

4. Hotplug virtiofs device
# virsh attach-device avocado-vt-vm1 virtiofs.xml 
Device attached successfully

5. Hotunplug virtiofs device
# virsh detach-device avocado-vt-vm1 virtiofs.xml 
Device detached successfully

Comment 21 errata-xmlrpc 2023-11-14 15:33:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6980