RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2177701 - Can't hotplug virtiofs device after restarting libvirt daemon
Summary: Can't hotplug virtiofs device after restarting libvirt daemon
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: libvirt
Version: 8.8
Hardware: All
OS: Unspecified
medium
high
Target Milestone: rc
: 8.9
Assignee: Ján Tomko
QA Contact: Lili Zhu
Parth Shah
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-13 12:34 UTC by smitterl
Modified: 2023-11-14 16:59 UTC (History)
10 users (show)

Fixed In Version: libvirt-8.0.0-20.module+el8.9.0+18894+a91bbb94
Doc Type: Known Issue
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-14 15:33:25 UTC
Type: Bug
Target Upstream Version: 9.2.0
Embargoed:

Attachments (Terms of Use)
domain definition (2.05 KB, text/plain)
2023-03-13 14:38 UTC, smitterl 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 2078693 0 high CLOSED libvirt can not identify shared memory after restarting virtqemud 2023-11-07 09:40:37 UTC
Red Hat Issue Tracker LIBVIRTAT-14495 0 None None None 2023-07-04 02:24:38 UTC
Red Hat Issue Tracker RHELPLAN-151567 0 None None None 2023-03-13 12:36:49 UTC
Red Hat Product Errata RHSA-2023:6980 0 None None None 2023-11-14 15:34:55 UTC

Internal Links: 2078693

Description smitterl 2023-03-13 12:34:12 UTC 
Description of problem:
After restarting the libvirt daemon, can't hotplug virtiofs device


Version-Release number of selected component (if applicable):
libvirt-8.0.0-18.module+el8.8.0+18287+112bb4e6.s390x

How reproducible:
100%


Steps to Reproduce:
1. Start a VM with 
  <memoryBacking>
    <access mode='shared'/>
  </memoryBacking>

2. systemctl restart libvirtd
3. Hotplug virtiofs device
<filesystem accessmode="passthrough" type="mount">
 <driver queue="512" type="virtiofs" />
 <source dir="/var/tmp/mount_tag0" />
 <target dir="mount_tag0" />
 <alias name="ua-8d2edac2-c191-11ed-885e-6a94bb98894f" />
 <binary path="/usr/libexec/virtiofsd" xattr="on">
  <cache mode="none" />
  <lock flock="off" posix="off" />
 </binary>
</filesystem>

Actual results:
error: unsupported configuration: 'virtiofs' requires shared memory

Expected results:
The device is attached successfully

Additional info:
After destroy-start VM, the device can be attached successfully.
Comment 3 smitterl 2023-03-13 13:19:37 UTC 
This does not reproduce with
libvirt-8.0.0-10.3.module+el8.7.0+18207+556790c8.s390x
Comment 4 smitterl 2023-03-13 13:24:56 UTC 
It reproduces with libvirt-8.0.0-14.module+el8.8.0+17806+11a519fc.s390x
Comment 5 Ján Tomko 2023-03-13 13:52:17 UTC 
Can you share the full domain XML that made it work for libvirt-8.0.0-10.3 on s390x?
Also, are you sure you restarted libvirtd before hotplug, when it worked, i.e. the older libvirt version?

The x86_64 test case uses <numa> to specify shared memory (at least the few cases I looked at).
Ever since this libvirt commit in libvirt 6.9.0 we also allow shared memory with implicit memory object ID:
https://gitlab.com/libvirt/libvirt/-/commit/e2425a17270d3c25730977d37b5679189714b913
but we did not save it on libvirt restart. And IIUC s390x has no NUMA. So I'm confused how this ever worked.
Comment 6 Ján Tomko 2023-03-13 14:10:28 UTC 
Upstream patch:
https://listman.redhat.com/archives/libvir-list/2023-March/238702.html
Comment 7 smitterl 2023-03-13 14:36:10 UTC 
(In reply to Ján Tomko from comment #5)
> Can you share the full domain XML that made it work for libvirt-8.0.0-10.3
> on s390x?
> Also, are you sure you restarted libvirtd before hotplug, when it worked,
> i.e. the older libvirt version?
> 
You're right, I can reproduce this with libvirt-8.0.0-10.3.module+el8.7.0+18207+556790c8.s390x after all.

> The x86_64 test case uses <numa> to specify shared memory (at least the few
> cases I looked at).
> Ever since this libvirt commit in libvirt 6.9.0 we also allow shared memory
> with implicit memory object ID:
> https://gitlab.com/libvirt/libvirt/-/commit/
> e2425a17270d3c25730977d37b5679189714b913
> but we did not save it on libvirt restart. And IIUC s390x has no NUMA. So
> I'm confused how this ever worked.
Comment 8 smitterl 2023-03-13 14:38:56 UTC 
Created attachment 1950249 [details]
domain definition
Comment 9 Ján Tomko 2023-03-14 16:15:35 UTC 
Pushed upstream as:
commit d5c7b7870e45575f81fffcb611c2546d0e02e778
Author:     Ján Tomko <jtomko>
CommitDate: 2023-03-14 17:10:01 +0100

    qemu: relax shared memory check for vhostuser daemons
    
    For some vhostuser daemons, we validate that the guest memory is shared
    with the host.
    
    With earlier versions of QEMU, it was only possible to mark memory
    as shared by defining an explicit NUMA topology.  Later, QEMU exposed
    the name of the default memory backend (defaultRAMid) so we can mark
    that memory as shared.
    
    Since libvirt commit:
      commit bff2ad5d6b1f25da02802273934d2a519159fec7
        qemu: Relax validation for mem->access if guest has no NUMA
    we already check for the case when user requests shared memory,
    but QEMU did not expose defaultRAMid.
    
    Drop the duplicit check from vhostuser device validation, to make
    it pass on hotplug even after libvirtd restart.
    
    This avoids the need to store the defaultRAMid, since we don't really
    need it for anything after the VM has been already started.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2078693
    https://bugzilla.redhat.com/show_bug.cgi?id=2177701
    
    Signed-off-by: Ján Tomko <jtomko>
    Reviewed-by: Michal Privoznik <mprivozn>

git describe: v9.1.0-200-gd5c7b7870e
Comment 10 Thomas Huth 2023-03-20 16:15:58 UTC 
@smitterl : Do you think we still need to get this fixed in RHEL 8.8 ? ... since it does not seem to be a regression, I'd rather move it to 8.9 and lower the "severity" ?
Comment 11 smitterl 2023-03-21 08:01:49 UTC 
(In reply to Thomas Huth from comment #10)
> @smitterl : Do you think we still need to get this fixed in RHEL
> 8.8 ? ... since it does not seem to be a regression, I'd rather move it to
> 8.9 and lower the "severity" ?

I agree. What's more, the corresponding RHEL 9 seems to be in the works for 9.3 https://bugzilla.redhat.com/show_bug.cgi?id=2078693
Comment 14 Lili Zhu 2023-05-18 13:49:33 UTC 
Build the rpm with patch:
https://gitlab.com/redhat/rhel/src/libvirt/-/merge_requests/112

1. start the guest with
  <memoryBacking>
    <source type='file'/>
    <access mode='shared'/>
  </memoryBacking>

2. restart libvirtd

3. prepare the virtiofs device xml
# cat virtiofs.xml 
<filesystem accessmode="passthrough" type="mount">
 <driver queue="512" type="virtiofs" />
 <source dir="/var/tmp/mount_tag0" />
 <target dir="mount_tag0" />
 <alias name="ua-8d2edac2-c191-11ed-885e-6a94bb98894f" />
 <binary path="/usr/libexec/virtiofsd" xattr="on">
  <cache mode="none" />
  <lock flock="off" posix="off" />
 </binary>
</filesystem>

4. hotplug the virtiofs device
# virsh attach-device avocado-vt-vm1 virtiofs.xml 
Device attached successfully
Comment 18 Lili Zhu 2023-06-11 08:03:33 UTC 
Verify this bug with:
libvirt-8.0.0-20.module+el8.9.0+18894+a91bbb94.x86_64

Verification steps are the same with those in Comment #14

Mark the bug as verified
Comment 19 liang cong 2023-06-12 06:58:22 UTC 
Verified this bug on S390 build:
# rpm -q libvirt
libvirt-8.0.0-20.module+el8.9.0+18894+a91bbb94.s390x

Test steps:
1. Start a VM with 
  <memoryBacking>
    <access mode='shared'/>
  </memoryBacking>

2. systemctl restart libvirtd
3. Prepare a virtiofs config file virtiofs.xml
<filesystem accessmode="passthrough" type="mount">
 <driver queue="512" type="virtiofs" />
 <source dir="/var/tmp/mount_tag0" />
 <target dir="mount_tag0" />
 <alias name="ua-8d2edac2-c191-11ed-885e-6a94bb98894f" />
 <binary path="/usr/libexec/virtiofsd" xattr="on">
  <cache mode="none" />
  <lock flock="off" posix="off" />
 </binary>
</filesystem>

4. Hotplug virtiofs device
# virsh attach-device avocado-vt-vm1 virtiofs.xml 
Device attached successfully

5. Hotunplug virtiofs device
# virsh detach-device avocado-vt-vm1 virtiofs.xml 
Device detached successfully
Comment 21 errata-xmlrpc 2023-11-14 15:33:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6980
Note You need to log in before you can comment on or make changes to this bug.