Bug 996543 - can't use iso images on readonly nfs filesystem
can't use iso images on readonly nfs filesystem
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 7.0
Assigned To: Michal Privoznik
Virtualization Bugs
:
: 1023970 1044413 (view as bug list)
Depends On:
Blocks: 1095135
  Show dependency treegraph
 
Reported: 2013-08-13 08:04 EDT by Gerd Hoffmann
Modified: 2015-06-01 09:25 EDT (History)
12 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-19.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1095135 (view as bug list)
Environment:
Last Closed: 2014-06-13 09:20:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
pool config #1 (460 bytes, text/plain)
2013-08-13 08:07 EDT, Gerd Hoffmann
no flags Details
pool config #2 (591 bytes, text/plain)
2013-08-13 08:09 EDT, Gerd Hoffmann
no flags Details
domain xml (2.94 KB, text/plain)
2014-01-16 09:35 EST, Gerd Hoffmann
no flags Details

  None (edit)
Description Gerd Hoffmann 2013-08-13 08:04:47 EDT
Description of problem:
$subject

Version-Release number of selected component (if applicable):
libvirt-1.1.1-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. create a guest
2. attach a iso image as virtual cdrom,
   where the iso image is located on a read-only nfs export
3. try to boot the guest

Actual results:
qemu doesn't start

Expected results:
guest boots fine

Additional info:
Comment 1 Gerd Hoffmann 2013-08-13 08:07:43 EDT
Created attachment 786128 [details]
pool config #1

First attempt: using directory storage pool, which is backed by a autofs-mounted (readonly) nfs filesystem

Result:

error: Failed to start domain rhel6-el7-base
error: unable to set security context 'system_u:object_r:svirt_image_t:s0' on '/mort/distiso/rhel6/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso': Read-only file system
Comment 3 Gerd Hoffmann 2013-08-13 08:09:13 EDT
Created attachment 786129 [details]
pool config #2

Second attempt: configure the very same nfs export as netfs storage pool.

Result:

error: Failed to start domain rhel6-el7-base
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=/var/lib/libvirt/images/distiso-rhel6-netfs/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso,if=none,id=drive-scsi0-0-4-0,readonly=on,format=raw,cache=none: could not open disk image /var/lib/libvirt/images/distiso-rhel6-netfs/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso: Permission denied
Comment 4 Gerd Hoffmann 2013-08-13 10:09:55 EDT
(In reply to Gerd Hoffmann from comment #3)
> Created attachment 786129 [details]
> pool config #2
> 
> Second attempt: configure the very same nfs export as netfs storage pool.

That one works after setting "setsebool -P virt_use_nfs 1"
Comment 5 Gerd Hoffmann 2013-08-16 02:57:45 EDT
Installing a virtual machine from a nfs-iso using Boxes failed.  Boxes doesn't offer to browse libvirt storage so I specified /mort/distiso/rhel6/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso as iso image, so it's most likely the same failure as in comment 1.  Boxes just said "failed" without a detailed error message though.
Comment 6 Eric Blake 2013-09-09 19:48:48 EDT
(In reply to Gerd Hoffmann from comment #4)
> (In reply to Gerd Hoffmann from comment #3)
> > Created attachment 786129 [details]
> > pool config #2
> > 
> > Second attempt: configure the very same nfs export as netfs storage pool.
> 
> That one works after setting "setsebool -P virt_use_nfs 1"

That sounds like a configuration error if setsebool is able to make it start working again, and about the only thing libvirt could do is try to issue a nicer error message if something won't work because of a configuration error.

I'm still trying to reproduce this setup, and so far, have failed to do so.  You didn't go into very many details at how you set up a read-only NFS mount, so I did (as root):

# mkdir /mnt/sysroot /nfs
# cp /path/to/live.iso /nfs
# echo '/nfs localhost' >> /etc/exports
# systemctl start nfs
# exportfs
/nfs          	localhost
# getsebool virt_use_nfs
virt_use_nfs --> on
# mount -t nfs localhost:/nfs /mnt/sysroot
# : > /mnt/sysroot/foo
bash: /mnt/sysroot/foo: Read-only file system
# ls /mnt/sysroot
live.iso
# virsh dumpxml guest
...
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/mnt/sysroot/live.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
...
# virsh start guest
Domain guest started

and the guest can read the iso contents just fine (note that I skipped the use of a storage pool and went straight for an absolute path in the guest XML).  Expecting it to work with virt_use_nfs off is nonsense.  I'm inclined to close this as not a bug if I can't get more details on what I need to do to reproduce it, or to retitle it if the real bug is just a poor error message when virt_use_nfs is disabled.
Comment 7 Eric Blake 2013-09-09 19:59:01 EDT
I did my testing on Fedora 19.  When I temporarily did 'setsebool virt_use_nfs off', the next attempt failed with a reasonable message:

# virsh start f18-live
error: Failed to start domain f18-live
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/4 (label charserial0)
qemu-system-x86_64: -drive file=/mnt/sysroot/Fedora-18-x86_64-netinst.iso,if=none,id=drive-virtio-disk0,readonly=on,format=raw: could not open disk image /mnt/sysroot/Fedora-18-x86_64-netinst.iso: Permission denied

The permission denied may not be the most insight in the world into the SELinux setting, but it is at least accurate.
Comment 8 Gerd Hoffmann 2013-09-10 02:49:37 EDT
nilsson kraxel ~# virsh start fedora-el7-nfs-iso
error: Failed to start domain fedora-el7-nfs-iso
error: unable to set security context 'system_u:object_r:svirt_image_t:s0' on '/mort/distiso/fedora/Fedora-19-x86_64-DVD.iso': Read-only file system

nilsson kraxel ~# mount | grep distiso
spunk.home.kraxel.org:/storage/distiso on /mort/distiso type nfs4 (ro,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.108,local_lock=none,addr=192.168.2.14)

/mort is managed by autofs, so /mort/distiso is mounted on demand.
Tried to start the guest twice, so for the second attempt /mort/distiso is already mounted, doesn't change behavior.

nilsson kraxel ~# grep mort /etc/auto.master
/mort           /etc/auto.map.kraxel.org
nilsson kraxel ~# grep distiso /etc/auto.map.kraxel.org
distiso         -ro,intr        spunk.home.kraxel.org:/storage/distiso
Comment 9 Eric Blake 2013-09-10 08:05:26 EDT
Hmm, this could be a case where NFS4 on RHEL 7 includes SELinux labeling support, whereas NFS on F19 still lacks it.  I'll try and get a RHEL 7 box up and running today, and repeat my experiments there.  I also wonder if this is the failure that would happen for _any_ read only file system, not just capable NFSv4.
Comment 10 chhu 2013-09-18 02:45:04 EDT
Reproduced this issue with packages:
libvirt-1.1.1-2.el7.x86_64
qemu-kvm-1.5.2-2.el7.x86_64

Steps:
On machine A: configure the nfs server with file below:
1) # more /etc/exports 
/mnt/nfs *(ro,sync,no_root_squash)

# service nfs restart

2)put a ISO file in directory: /mnt/nfs
# ls /mnt/nfs/
RHEL6.4-20130130.0-Server-x86_64-DVD1.iso

On machine B:
1) # mount -t nfs machineA-ip:/mnt/nfs /mnt
2) # mount |grep mnt
10.66.7.108:/mnt/nfs on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.66.5.93,local_lock=none,addr=10.66.7.108)
3) # touch /mnt/a
touch: cannot touch ‘/mnt/a’: Read-only file system

4) define and start a domain
# virsh define rh7-qcow2.xml 
Domain rh7-qcow2 defined from rh7-qcow2.xml

# virsh start rh7-qcow2
Domain rh7-qcow2 started

5) add the iso image as virtual cdrom, where the iso image is located on a read-only nfs export
# virsh edit rh7-qcow2
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso'/>
<target dev='vda' bus='virtio'/>
</disk>

6)detroy/start the domain
# virsh destroy rh7-qcow2
Domain rh7-qcow2 destroyed
# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=/mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso: Permission denied

6) set virt_use_nfs then start the domain
# getsebool virt_use_nfs
virt_use_nfs --> on

# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=/mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso: Permission denied
Comment 11 Giuseppe Scrivano 2013-12-18 07:06:55 EST
*** Bug 1044413 has been marked as a duplicate of this bug. ***
Comment 12 Michal Privoznik 2014-01-13 12:17:33 EST
I believe there's a warning thrown into the logs:

2014-01-13 17:01:31.999+0000: 2537: info : virSecuritySELinuxSetFileconHelper:923 : Setting security context 'system_u:object_r:svirt_image_t:s0:c163,c590' on '/mnt/masina_nfs/f17.iso' not supported. Consider setting virt_use_nfs

It's the result of my patch:

commit 1888363d8bfd2ac165ddfd495624a449b0df9d58
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Sep 22 10:57:24 2011 +0200
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Fri Sep 23 12:15:55 2011 +0200

    selinux: Correctly report warning if virt_use_nfs not set
    
    Previous patch c9b37fee tried to deal with virt_use_nfs. But
    setfilecon() returns EOPNOTSUPP on NFS so we need to move the
    warning to else branch.

contained in v0.9.7-rc1~257

(can't recall nor quick find which bug was I fixing)

Having said that, I'm not sure what else can we do. Failing to set the selinux context doesn't mean the file will not be accessible - it still can be, as enabling virt_use_nfs case shows. Hence, libvirt does the best-effort approach and let qemu find out. What we can do (and I'm not convinced we should do that) is: once we fail to set selinux label on a NFS storage nad virt_use_nfs == off error out instead of our best-effort approach. Gerd (or anybody else), do you have any other opinion?
Comment 13 Gerd Hoffmann 2014-01-15 07:26:11 EST
  Hi,

> Having said that, I'm not sure what else can we do. Failing to set the
> selinux context doesn't mean the file will not be accessible - it still can
> be, as enabling virt_use_nfs case shows.

I think any read-only mounted filesystem can hit this.

> Hence, libvirt does the best-effort
> approach and let qemu find out. What we can do (and I'm not convinced we
> should do that) is: once we fail to set selinux label on a NFS storage nad
> virt_use_nfs == off error out instead of our best-effort approach. Gerd (or
> anybody else), do you have any other opinion?

Maybe treat read/write and read/only images differently?  read/write is unlikely to work without proper labeling.  A label failure for read/only images has a much higher chance to succeed nevertheless.

The testing (comment #10) is wrong btw.  It attaches a iso image, but as disk, therefore qemu tries to open the file in read/write mode which isn't going to succeed on a read/only filesystem no matter how it is labeled.
Comment 14 Michal Privoznik 2014-01-15 09:17:33 EST
(In reply to Gerd Hoffmann from comment #13)
>   Hi,
> 
> > Having said that, I'm not sure what else can we do. Failing to set the
> > selinux context doesn't mean the file will not be accessible - it still can
> > be, as enabling virt_use_nfs case shows.
> 
> I think any read-only mounted filesystem can hit this.

The problem is, the FS is mounted RW and it's the nfsd which actually denies write() with EROFS. From the client POV the FS is still RW. I don't think that client is able to find out RO NFS which is mounted as RW (beside actually creating some dummy file - certainly something we do not want from libvirt to do).

> 
> > Hence, libvirt does the best-effort
> > approach and let qemu find out. What we can do (and I'm not convinced we
> > should do that) is: once we fail to set selinux label on a NFS storage nad
> > virt_use_nfs == off error out instead of our best-effort approach. Gerd (or
> > anybody else), do you have any other opinion?
> 
> Maybe treat read/write and read/only images differently?  read/write is
> unlikely to work without proper labeling.  A label failure for read/only
> images has a much higher chance to succeed nevertheless.

Agreed. That's why libvirt doesn't fail at this point but rather throws a warning in the logs and then let qemu find out.

> 
> The testing (comment #10) is wrong btw.  It attaches a iso image, but as
> disk, therefore qemu tries to open the file in read/write mode which isn't
> going to succeed on a read/only filesystem no matter how it is labeled.

So my question is: is this really a bug afterall?
Comment 15 Gerd Hoffmann 2014-01-16 06:44:27 EST
> > Maybe treat read/write and read/only images differently?  read/write is
> > unlikely to work without proper labeling.  A label failure for read/only
> > images has a much higher chance to succeed nevertheless.
> 
> Agreed. That's why libvirt doesn't fail at this point but rather throws a
> warning in the logs and then let qemu find out.

It seems to only do that for nfs pools.

With a directory pool pointing to autofs-mounted nfs storage still fails (see comment 8), even with virt_use_nfs=on
Comment 16 Michal Privoznik 2014-01-16 09:19:11 EST
Gerd, can you please then attach the domain XML? I wonder if boxes are using static selinux label or something.
Comment 17 Gerd Hoffmann 2014-01-16 09:35:42 EST
Created attachment 851121 [details]
domain xml

Isn't created by boxes, no special selinux stuff in there, attaching nevertheless.
Comment 18 Michal Privoznik 2014-01-17 04:29:44 EST
If that's the case, I guess you've got the /etc/libvirt/qemu.conf set up to use selinux. Is that right? Can you paste all the uncomented variables here then please?
Comment 19 Gerd Hoffmann 2014-01-17 04:48:42 EST
Nothing changed in /etc/libvirt/qemu.conf
Comment 20 Michal Privoznik 2014-01-17 07:44:31 EST
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2014-January/msg00795.html
Comment 24 Yedidyah Bar David 2014-01-30 03:04:03 EST
*** Bug 1023970 has been marked as a duplicate of this bug. ***
Comment 25 Ludek Smid 2014-06-13 09:20:59 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.