Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2045932

Summary: support for using virt-install to define a guest with externally launched virtiofsd
Product: Red Hat Enterprise Linux 9 Reporter: Lili Zhu <lizhu>
Component: virt-managerAssignee: Jonathon Jongsma <jjongsma>
virt-manager sub component: Common QA Contact: Hongzhou Liu <hongzliu>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: hongzliu, jjongsma, juzhou, smitterl, tyan, tzheng, virt-maint, xiagao, ymankad
Version: 9.0Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virt-manager-3.2.0-14.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 13:28:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lili Zhu 2022-01-26 01:42:42 UTC
Description of problem:
As it is shown in https://libvirt.org/kbase/virtiofs.html#externally-launched-virtiofsd, libvirt support externally launched virtiofsd. However, we can not define a guest with externally launched virtiofsd using virt-install currently.

Version-Release number of selected component (if applicable):
virt-install-3.2.0-12.el9.noarch

How reproducible:
100%

Steps to Reproduce:
1. try to define a guest with externally launched virtiofsd use virt-install
# virt-install --connect qemu:///system -n avocado-vt-vm1 --hvm --accelerate -r 1024 --vcpus=2 --disk path=/var/lib/libvirt/images/avocado-vt-vm1.qcow2,bus=virtio,format=qcow2 --network bridge=virbr0,model=virtio --import --noreboot --noautoconsole --serial pty --memballoon model=virtio --graphics vnc --video virtio --machine q35 --filesystem type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.queue=1024,target.dir=myfs --memorybacking access.mode=shared
ERROR    Unknown --filesystem options: ['source.socket']

Expected results:
Can use virt-install to define a guest with externally launched virtiofsd



Additional info:

Comment 1 Lili Zhu 2022-01-26 01:45:09 UTC
Upstream Patch:
commit 520b3a3b35359ddd07f3a818d6cc3e6740079eff
Author: Cole Robinson <crobinso>
Date:   Fri Jul 30 13:15:29 2021 -0400

    cli: --filesystem: add binary.sandbox.mode and source.socket
    
    Signed-off-by: Cole Robinson <crobinso>

Comment 3 Hongzhou Liu 2022-02-08 02:50:20 UTC
Pre-verify with virt-install-3.2.0-13.el9.noarch

step1: install a vm with virtiofsd
virt-install --name t1 --memory 4096 --disk path=/home/t2.img,size=10 --location http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.0/compose/BaseOS/x86_64/os/ --filesystem type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.queue=1024,target.dir=myfs --memorybacking access.mode=shared

Result: the vm can be installed correctly, change the status to tested, Thanks

Comment 6 Hongzhou Liu 2022-02-08 09:04:18 UTC
Reproduce this bug with  virt-install-3.2.0-12.el9.noarch

Error message is the same with Comment 0.  This bug can be reproduced 
	

Now verify this bug with

Packages:
virt-install-3.2.0-13.el9.noarch
virt-manager-3.2.0-13.el9.noarch

Step 1: Prepare a virtiofsd 
# /usr/libexec/virtiofsd --socket-path=/tmp/vm001-vhost-fs.sock -o source=/tmp/fs/vm001
[2022-02-08T07:57:42Z INFO  virtiofsd] Waiting for vhost-user socket connection...

Step 2: On another terminator, run the command below:
# virt-install --name t1 --memory 4096 \
--disk path=/home/t2.img,size=10 \
--location http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.0/compose/BaseOS/x86_64/os/ \
--filesystem type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.queue=1024,target=tag,source.dir=/tmp/fs/vm001/,accessmode=passthrough \
--memorybacking access.mode=shared

Result: the vm can be installed successfully

Step3: Check the xml configuration with filesystem after installation finishing.

…
 <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs' queue='1024'/>
      <binary path='/usr/libexec/virtiofsd'/>
      <source dir='/tmp/fs/vm001/'/>
      <target dir='tag'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </filesystem>

…

On man page of virt-install:
 accessmode or mode
              The  access  mode  for the source directory from the guest OS. Only used with QEMU and type=mount. Valid modes are 'passthrough'
              (the default), 'mapped', or 'squash'. See libvirt domain XML documentation for more info.

It says virt-install will use passthrough as default, but when I run the command blow without accessmode=passthrough it will failed, The error message as below:

# virt-install --name t1 --memory 4096 --disk path=/home/t2.img,size=10 --location http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.0/compose/BaseOS/x86_64/os/ --filesystem type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.queue=1024,target=tag,source.dir=/tmp/fs/vm001/

Starting install...
Retrieving file vmlinuz...                                                                                                 |  10 MB  00:00:01     
Retrieving file initrd.img...                                                                                              |  76 MB  00:00:01     
ERROR    unsupported configuration: virtiofs only supports passthrough accessmode
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start t1
otherwise, please restart your installation.

In my opinion, if the default value of access mode is passthrough, I should not need to define it on the command line.

Should we fix this problem in this BZ or open a new BZ for this?

Comment 7 Hongzhou Liu 2022-02-08 09:40:05 UTC
(In reply to Hongzhou Liu from comment #6)
> Reproduce this bug with  virt-install-3.2.0-12.el9.noarch
> 
> Error message is the same with Comment 0.  This bug can be reproduced 
> 	
> 
> Now verify this bug with
> 
> Packages:
> virt-install-3.2.0-13.el9.noarch
> virt-manager-3.2.0-13.el9.noarch
> 
> Step 1: Prepare a virtiofsd 
> # /usr/libexec/virtiofsd --socket-path=/tmp/vm001-vhost-fs.sock -o
> source=/tmp/fs/vm001
> [2022-02-08T07:57:42Z INFO  virtiofsd] Waiting for vhost-user socket
> connection...
> 
> Step 2: On another terminator, run the command below:
> # virt-install --name t1 --memory 4096 \
> --disk path=/home/t2.img,size=10 \
> --location
> http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.
> 0/compose/BaseOS/x86_64/os/ \
> --filesystem
> type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.
> queue=1024,target=tag,source.dir=/tmp/fs/vm001/,accessmode=passthrough \
> --memorybacking access.mode=shared
> 
> Result: the vm can be installed successfully
> 
> Step3: Check the xml configuration with filesystem after installation
> finishing.
> 
> …
>  <filesystem type='mount' accessmode='passthrough'>
>       <driver type='virtiofs' queue='1024'/>
>       <binary path='/usr/libexec/virtiofsd'/>
>       <source dir='/tmp/fs/vm001/'/>
>       <target dir='tag'/>
>       <alias name='fs0'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
>     </filesystem>
> 
> …
> 
> On man page of virt-install:
>  accessmode or mode
>               The  access  mode  for the source directory from the guest OS.
> Only used with QEMU and type=mount. Valid modes are 'passthrough'
>               (the default), 'mapped', or 'squash'. See libvirt domain XML
> documentation for more info.
> 
> It says virt-install will use passthrough as default, but when I run the
> command blow without accessmode=passthrough it will failed, The error
> message as below:
> 
> # virt-install --name t1 --memory 4096 --disk path=/home/t2.img,size=10
> --location
> http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.
> 0/compose/BaseOS/x86_64/os/ --filesystem
> type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.
> queue=1024,target=tag,source.dir=/tmp/fs/vm001/
> 
> Starting install...
> Retrieving file vmlinuz...                                                  
> |  10 MB  00:00:01     
> Retrieving file initrd.img...                                               
> |  76 MB  00:00:01     
> ERROR    unsupported configuration: virtiofs only supports passthrough
> accessmode
> Domain installation does not appear to have been successful.
> If it was, you can restart your domain by running:
>   virsh --connect qemu:///system start t1
> otherwise, please restart your installation.
> 
> In my opinion, if the default value of access mode is passthrough, I should
> not need to define it on the command line.
> 
> Should we fix this problem in this BZ or open a new BZ for this?


Add --debug to check the value for accseemode

...
<filesystem type="mount" accessmode="mapped">
...

Base on man page, it should be passthrough if I did not define access mode.

Comment 8 Jonathon Jongsma 2022-02-08 16:39:27 UTC
It looks like this is a documentation error. The default was changed to 'mapped' in release 1.3.0, but the manpage was not updated:

commit 85307b9bd2971cfc972c976c78e411f018647861
Author: Cole Robinson <crobinso>
Date:   Sun Sep 20 20:33:46 2015 -0400

    devicefilesystem: Default to accessmode=mapped for qemu
    
    libvirt qemu default's to accessmode=passthrough, which really only
    works correctly when qemu is run as root, which isn't common for libvirt
    nowadays. So use accessmode=mapped which has a better chance of working


I can submit a new patch to update the manpage.

Comment 9 Hongzhou Liu 2022-02-09 01:38:54 UTC
(In reply to Jonathon Jongsma from comment #8)
> It looks like this is a documentation error. The default was changed to
> 'mapped' in release 1.3.0, but the manpage was not updated:
> 
> commit 85307b9bd2971cfc972c976c78e411f018647861
> Author: Cole Robinson <crobinso>
> Date:   Sun Sep 20 20:33:46 2015 -0400
> 
>     devicefilesystem: Default to accessmode=mapped for qemu
>     
>     libvirt qemu default's to accessmode=passthrough, which really only
>     works correctly when qemu is run as root, which isn't common for libvirt
>     nowadays. So use accessmode=mapped which has a better chance of working
> 
> 
> I can submit a new patch to update the manpage.

Thanks for responding, change the status to assigned.

Comment 12 Hongzhou Liu 2022-02-10 02:09:05 UTC
Verify with virt-manager-3.2.0-14.el9.noarch

Step 1: check the man page for virt-install, introduction for accessmode 

 accessmode or mode
              The  access  mode  for the source directory from the guest OS. Only used with QEMU and type=mount. Valid modes are 'mapped' (the
              default), 'passthrough', or 'squash'. See libvirt domain XML documentation for more info.
"mapped" now is the default in this document which is meet the expect. Change the status to verified. Thanks

Comment 13 Hongzhou Liu 2022-02-10 02:49:56 UTC
Test the feature to make sure this bug has fixed.

Packages:
virt-install-3.2.0-13.el9.noarch
virt-manager-3.2.0-13.el9.noarch

Step 1: Prepare a virtiofsd 
# /usr/libexec/virtiofsd --socket-path=/tmp/vm001-vhost-fs.sock -o source=/tmp/fs/vm001
[2022-02-08T07:57:42Z INFO  virtiofsd] Waiting for vhost-user socket connection...

Step 2: On another terminator, run the command below:
# virt-install --name t1 --memory 4096 \
--disk path=/home/t2.img,size=10 \
--location http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.0/compose/BaseOS/x86_64/os/ \
--filesystem type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.queue=1024,target=tag,source.dir=/tmp/fs/vm001/,accessmode=passthrough \
--memorybacking access.mode=shared

Result: the vm can be installed successfully

Step3: Check the xml configuration with filesystem after installation finishing.

…
 <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs' queue='1024'/>
      <binary path='/usr/libexec/virtiofsd'/>
      <source dir='/tmp/fs/vm001/'/>
      <target dir='tag'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </filesystem>

…

This bug has been fixed

Comment 14 Hongzhou Liu 2022-02-10 06:30:28 UTC
(In reply to Hongzhou Liu from comment #13)
> Test the feature to make sure this bug has fixed.
> 
> Packages:
> virt-install-3.2.0-13.el9.noarch
> virt-manager-3.2.0-13.el9.noarch
> 
> Step 1: Prepare a virtiofsd 
> # /usr/libexec/virtiofsd --socket-path=/tmp/vm001-vhost-fs.sock -o
> source=/tmp/fs/vm001
> [2022-02-08T07:57:42Z INFO  virtiofsd] Waiting for vhost-user socket
> connection...
> 
> Step 2: On another terminator, run the command below:
> # virt-install --name t1 --memory 4096 \
> --disk path=/home/t2.img,size=10 \
> --location
> http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.
> 0/compose/BaseOS/x86_64/os/ \
> --filesystem
> type=mount,driver.type=virtiofs,source.socket=/var/virtiofsd.sock,driver.
> queue=1024,target=tag,source.dir=/tmp/fs/vm001/,accessmode=passthrough \
> --memorybacking access.mode=shared
> 
> Result: the vm can be installed successfully
> 
> Step3: Check the xml configuration with filesystem after installation
> finishing.
> 
> …
>  <filesystem type='mount' accessmode='passthrough'>
>       <driver type='virtiofs' queue='1024'/>
>       <binary path='/usr/libexec/virtiofsd'/>
>       <source dir='/tmp/fs/vm001/'/>
>       <target dir='tag'/>
>       <alias name='fs0'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
>     </filesystem>
> 
> …
> 
> This bug has been fixed




Sorry I post wrong steps with wrong package version.

Here I verify this again

Packages
virt-manager-3.2.0-14.el9.noarch
virt-install-3.2.0-14.el9.noarch

Step 1:
Run the command to start virtiofsd
# /usr/libexec/virtiofsd --socket-path=/var/virtiofsd.sock -o source=/tmp/fs/vm001/ -o cache=always
[2022-02-10T02:01:51Z INFO  virtiofsd] Waiting for vhost-user socket connection...

Step 2: 
Open another terminal and run the command below to create a vm
#virt-install --name t3 --memory 4096 \
--disk path=/home/t4.img,size=20 \
--location http://download.eng.pek2.redhat.com/rhel-9/nightly/RHEL-9-Beta/latest-RHEL-9.0/compose/BaseOS/x86_64/os/ \
--filesystem type=mount,driver.type=virtiofs,source.socket=/tmp/vm002-vhost-fs.sock,driver.queue=1024,target=tag,source.dir=/tmp/fs/vm002/,accessmode=passthrough\
--memorybacking access.mode=shared
Result: the vm can be installed successfully

Step 3:
Check the xml config for filesystem
...
 <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs' queue='1024'/>
      <binary path='/usr/libexec/virtiofsd'/>
      <source dir='/tmp/fs/vm002/'/>
      <target dir='tag'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </filesystem>
...

This bug has VERIFIED.

Comment 16 errata-xmlrpc 2022-05-17 13:28:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: virt-manager), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2515