Created attachment 786128 [details]
pool config #1

First attempt: using directory storage pool, which is backed by a autofs-mounted (readonly) nfs filesystem

Result:

error: Failed to start domain rhel6-el7-base
error: unable to set security context 'system_u:object_r:svirt_image_t:s0' on '/mort/distiso/rhel6/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso': Read-only file system

Created attachment 786129 [details]
pool config #2

Second attempt: configure the very same nfs export as netfs storage pool.

Result:

error: Failed to start domain rhel6-el7-base
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=/var/lib/libvirt/images/distiso-rhel6-netfs/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso,if=none,id=drive-scsi0-0-4-0,readonly=on,format=raw,cache=none: could not open disk image /var/lib/libvirt/images/distiso-rhel6-netfs/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso: Permission denied

(In reply to Gerd Hoffmann from comment #3)
> Created attachment 786129 [details]
> pool config #2
> 
> Second attempt: configure the very same nfs export as netfs storage pool.

That one works after setting "setsebool -P virt_use_nfs 1"

Installing a virtual machine from a nfs-iso using Boxes failed.  Boxes doesn't offer to browse libvirt storage so I specified /mort/distiso/rhel6/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso as iso image, so it's most likely the same failure as in comment 1.  Boxes just said "failed" without a detailed error message though.

(In reply to Gerd Hoffmann from comment #4)
> (In reply to Gerd Hoffmann from comment #3)
> > Created attachment 786129 [details]
> > pool config #2
> > 
> > Second attempt: configure the very same nfs export as netfs storage pool.
> 
> That one works after setting "setsebool -P virt_use_nfs 1"

That sounds like a configuration error if setsebool is able to make it start working again, and about the only thing libvirt could do is try to issue a nicer error message if something won't work because of a configuration error.

I'm still trying to reproduce this setup, and so far, have failed to do so.  You didn't go into very many details at how you set up a read-only NFS mount, so I did (as root):

# mkdir /mnt/sysroot /nfs
# cp /path/to/live.iso /nfs
# echo '/nfs localhost' >> /etc/exports
# systemctl start nfs
# exportfs
/nfs          	localhost
# getsebool virt_use_nfs
virt_use_nfs --> on
# mount -t nfs localhost:/nfs /mnt/sysroot
# : > /mnt/sysroot/foo
bash: /mnt/sysroot/foo: Read-only file system
# ls /mnt/sysroot
live.iso
# virsh dumpxml guest
...
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/mnt/sysroot/live.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
...
# virsh start guest
Domain guest started

and the guest can read the iso contents just fine (note that I skipped the use of a storage pool and went straight for an absolute path in the guest XML).  Expecting it to work with virt_use_nfs off is nonsense.  I'm inclined to close this as not a bug if I can't get more details on what I need to do to reproduce it, or to retitle it if the real bug is just a poor error message when virt_use_nfs is disabled.

I did my testing on Fedora 19.  When I temporarily did 'setsebool virt_use_nfs off', the next attempt failed with a reasonable message:

# virsh start f18-live
error: Failed to start domain f18-live
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/4 (label charserial0)
qemu-system-x86_64: -drive file=/mnt/sysroot/Fedora-18-x86_64-netinst.iso,if=none,id=drive-virtio-disk0,readonly=on,format=raw: could not open disk image /mnt/sysroot/Fedora-18-x86_64-netinst.iso: Permission denied

The permission denied may not be the most insight in the world into the SELinux setting, but it is at least accurate.

nilsson kraxel ~# virsh start fedora-el7-nfs-iso
error: Failed to start domain fedora-el7-nfs-iso
error: unable to set security context 'system_u:object_r:svirt_image_t:s0' on '/mort/distiso/fedora/Fedora-19-x86_64-DVD.iso': Read-only file system

nilsson kraxel ~# mount | grep distiso
spunk.home.kraxel.org:/storage/distiso on /mort/distiso type nfs4 (ro,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.108,local_lock=none,addr=192.168.2.14)

/mort is managed by autofs, so /mort/distiso is mounted on demand.
Tried to start the guest twice, so for the second attempt /mort/distiso is already mounted, doesn't change behavior.

nilsson kraxel ~# grep mort /etc/auto.master
/mort           /etc/auto.map.kraxel.org
nilsson kraxel ~# grep distiso /etc/auto.map.kraxel.org
distiso         -ro,intr        spunk.home.kraxel.org:/storage/distiso

Hmm, this could be a case where NFS4 on RHEL 7 includes SELinux labeling support, whereas NFS on F19 still lacks it.  I'll try and get a RHEL 7 box up and running today, and repeat my experiments there.  I also wonder if this is the failure that would happen for _any_ read only file system, not just capable NFSv4.

Reproduced this issue with packages:
libvirt-1.1.1-2.el7.x86_64
qemu-kvm-1.5.2-2.el7.x86_64

Steps:
On machine A: configure the nfs server with file below:
1) # more /etc/exports 
/mnt/nfs *(ro,sync,no_root_squash)

# service nfs restart

2)put a ISO file in directory: /mnt/nfs
# ls /mnt/nfs/
RHEL6.4-20130130.0-Server-x86_64-DVD1.iso

On machine B:
1) # mount -t nfs machineA-ip:/mnt/nfs /mnt
2) # mount |grep mnt
10.66.7.108:/mnt/nfs on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.66.5.93,local_lock=none,addr=10.66.7.108)
3) # touch /mnt/a
touch: cannot touch ‘/mnt/a’: Read-only file system

4) define and start a domain
# virsh define rh7-qcow2.xml 
Domain rh7-qcow2 defined from rh7-qcow2.xml

# virsh start rh7-qcow2
Domain rh7-qcow2 started

5) add the iso image as virtual cdrom, where the iso image is located on a read-only nfs export
# virsh edit rh7-qcow2
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso'/>
<target dev='vda' bus='virtio'/>
</disk>

6)detroy/start the domain
# virsh destroy rh7-qcow2
Domain rh7-qcow2 destroyed
# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=/mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso: Permission denied

6) set virt_use_nfs then start the domain
# getsebool virt_use_nfs
virt_use_nfs --> on

# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=/mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /mnt/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso: Permission denied

*** Bug 1044413 has been marked as a duplicate of this bug. ***

I believe there's a warning thrown into the logs:

2014-01-13 17:01:31.999+0000: 2537: info : virSecuritySELinuxSetFileconHelper:923 : Setting security context 'system_u:object_r:svirt_image_t:s0:c163,c590' on '/mnt/masina_nfs/f17.iso' not supported. Consider setting virt_use_nfs

It's the result of my patch:

commit 1888363d8bfd2ac165ddfd495624a449b0df9d58
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Sep 22 10:57:24 2011 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Sep 23 12:15:55 2011 +0200

    selinux: Correctly report warning if virt_use_nfs not set
    
    Previous patch c9b37fee tried to deal with virt_use_nfs. But
    setfilecon() returns EOPNOTSUPP on NFS so we need to move the
    warning to else branch.

contained in v0.9.7-rc1~257

(can't recall nor quick find which bug was I fixing)

Having said that, I'm not sure what else can we do. Failing to set the selinux context doesn't mean the file will not be accessible - it still can be, as enabling virt_use_nfs case shows. Hence, libvirt does the best-effort approach and let qemu find out. What we can do (and I'm not convinced we should do that) is: once we fail to set selinux label on a NFS storage nad virt_use_nfs == off error out instead of our best-effort approach. Gerd (or anybody else), do you have any other opinion?

  Hi,

> Having said that, I'm not sure what else can we do. Failing to set the
> selinux context doesn't mean the file will not be accessible - it still can
> be, as enabling virt_use_nfs case shows.

I think any read-only mounted filesystem can hit this.

> Hence, libvirt does the best-effort
> approach and let qemu find out. What we can do (and I'm not convinced we
> should do that) is: once we fail to set selinux label on a NFS storage nad
> virt_use_nfs == off error out instead of our best-effort approach. Gerd (or
> anybody else), do you have any other opinion?

Maybe treat read/write and read/only images differently?  read/write is unlikely to work without proper labeling.  A label failure for read/only images has a much higher chance to succeed nevertheless.

The testing (comment #10) is wrong btw.  It attaches a iso image, but as disk, therefore qemu tries to open the file in read/write mode which isn't going to succeed on a read/only filesystem no matter how it is labeled.

(In reply to Gerd Hoffmann from comment #13)
>   Hi,
> 
> > Having said that, I'm not sure what else can we do. Failing to set the
> > selinux context doesn't mean the file will not be accessible - it still can
> > be, as enabling virt_use_nfs case shows.
> 
> I think any read-only mounted filesystem can hit this.

The problem is, the FS is mounted RW and it's the nfsd which actually denies write() with EROFS. From the client POV the FS is still RW. I don't think that client is able to find out RO NFS which is mounted as RW (beside actually creating some dummy file - certainly something we do not want from libvirt to do).

> 
> > Hence, libvirt does the best-effort
> > approach and let qemu find out. What we can do (and I'm not convinced we
> > should do that) is: once we fail to set selinux label on a NFS storage nad
> > virt_use_nfs == off error out instead of our best-effort approach. Gerd (or
> > anybody else), do you have any other opinion?
> 
> Maybe treat read/write and read/only images differently?  read/write is
> unlikely to work without proper labeling.  A label failure for read/only
> images has a much higher chance to succeed nevertheless.

Agreed. That's why libvirt doesn't fail at this point but rather throws a warning in the logs and then let qemu find out.

> 
> The testing (comment #10) is wrong btw.  It attaches a iso image, but as
> disk, therefore qemu tries to open the file in read/write mode which isn't
> going to succeed on a read/only filesystem no matter how it is labeled.

So my question is: is this really a bug afterall?

> > Maybe treat read/write and read/only images differently?  read/write is
> > unlikely to work without proper labeling.  A label failure for read/only
> > images has a much higher chance to succeed nevertheless.
> 
> Agreed. That's why libvirt doesn't fail at this point but rather throws a
> warning in the logs and then let qemu find out.

It seems to only do that for nfs pools.

With a directory pool pointing to autofs-mounted nfs storage still fails (see comment 8), even with virt_use_nfs=on

Gerd, can you please then attach the domain XML? I wonder if boxes are using static selinux label or something.

Created attachment 851121 [details]
domain xml

Isn't created by boxes, no special selinux stuff in there, attaching nevertheless.

If that's the case, I guess you've got the /etc/libvirt/qemu.conf set up to use selinux. Is that right? Can you paste all the uncomented variables here then please?

Nothing changed in /etc/libvirt/qemu.conf

Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2014-January/msg00795.html

Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-January/msg00487.html

*** Bug 1023970 has been marked as a duplicate of this bug. ***

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.