Bug 862756

Summary: Can't use kernel on r/o file system for VM direct kernel boot because it can't be chowned
Product: Red Hat Enterprise Linux 7 Reporter: David Jaša <djasa>
Component: libvirtAssignee: Erik Skultety <eskultet>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: cwei, dyuan, gsun, laine, mzhan, rbalakri, riek, sreichar, weizhan, yury, zpeng
Target Milestone: beta   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 702044 Environment:
Last Closed: 2015-06-01 13:19:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 702044    
Bug Blocks:    
Attachments:
Description Flags
libvirtd.log none

Description David Jaša 2012-10-03 13:21:39 UTC
Fixes for the cloned bug should be copied to direct kernel boot code path, too. You can not currently use kernel/initrd from read only nfs to boot the VM.


+++ This bug was initially created as a clone of Bug #702044 +++

Description of problem:
When trying to attach iso from read-only filesystem (like NFS server administered by somebody else), it fails because iso's ownership cannot be changed to qemu:qemu

Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1. run VM with empty cdrom device 
2. run 'virsh attach-disk win7x64 /path/on/readonly_fs/to/image.iso hdc --type cdrom;
3.
  
Actual results:
this error is throwed:
error: Failed to attach disk
error: unable to set user and group to '107:107' on '/mnt/trash/ic114/RHEV-toolsSetup_2.3_395.iso': Nepřípustný argument

Expected results:
disk is attached as read-only

Additional info:
* virt-manager fails with the same error
* 107:107 are UID:GID in local system
* unix permissions don't block qemu or libvirt from reading the isos, but selinux may block them (I don't know how to confirm this now)

--- Additional comment from laine on 2011-06-02 22:28:56 CEST ---

Nepřípustný argument == EINVAL

The error comes (as you'd guess) from chown(2). But the manpage for chown doesn't mention the possibility of returning EINVAL. virSecurityDACSetOwnership already special cases EOPNOTSUPP, EPERM, and EROFS to ignore them. Would it be acceptable in all cases to just ignore EINVAL as well?

--- Additional comment from laine on 2011-06-02 22:34:25 CEST ---

*** Bug 709454 has been marked as a duplicate of this bug. ***

--- Additional comment from laine on 2011-06-03 18:28:17 CEST ---

The following fix has been pushed to upstream libvirt, and will be in the next rebase for RHEL6:

commit 62ed801c13787cc844e75db7cd2d4c8a42454fcc
Author: Laine Stump <laine>
Date:   Fri Jun 3 11:59:09 2011 -0400

    security driver: ignore EINVAL when chowning an image file
    
    This fixes:
    
      https://bugzilla.redhat.com/show_bug.cgi?id=702044
      https://bugzilla.redhat.com/show_bug.cgi?id=709454
    
    Both of these complain of a failure to use an image file that resides
    on a read-only NFS volume. The function in the DAC security driver
    that chowns image files to the qemu user:group before using them
    already has special cases to ignore failure of chown on read-only file
    systems, and in a few other cases, but it hadn't been checking for
    EINVAL, which is what is returned if the qemu user doesn't even exist
    on the NFS server.
    
    Since the explanation of EINVAL in the chown man page almost exactly
    matches the log message already present for the case of EOPNOTSUPP,
    I've just added EINVAL to that same conditional.

--- Additional comment from laine on 2011-06-03 21:54:01 CEST ---

The question in Comment 1 was already answered in IRC (the answer was "yes").

--- Additional comment from veillard on 2011-06-23 05:14:19 CEST ---

This should be fixed by the libvirt-0.9.2-1.el6 rebase

--- Additional comment from zpeng on 2011-06-28 08:37:18 CEST ---

I can reproduce this in libvirt-0.8.7-18.el6
#mount -t nfs4 refarch.cloud.lab.eng.bos.redhat.com:/pub /testnfs

# virsh attach-disk rhel6 /testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso hdc --type cdrom
error: Failed to attach disk
error: unable to set user and group to '107:107' on '/testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso': Invalid argument

then update to libvirt-0.9.2-1.el6
# virsh attach-disk rhel6 /testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso hdc --type cdrom
error: Failed to attach disk
error: internal error unable to execute QEMU command 'change': Could not open '/testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso'
is the verification enough?  or need other settings and verify points?

--- Additional comment from djasa on 2011-06-28 09:35:59 CEST ---

(In reply to comment #7)
> I can reproduce this in libvirt-0.8.7-18.el6
> #mount -t nfs4 refarch.cloud.lab.eng.bos.redhat.com:/pub /testnfs
> 
> # virsh attach-disk rhel6
> /testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso hdc --type cdrom
> error: Failed to attach disk
> error: unable to set user and group to '107:107' on
> '/testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso': Invalid argument
> 
> then update to libvirt-0.9.2-1.el6
> # virsh attach-disk rhel6
> /testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso hdc --type cdrom
> error: Failed to attach disk
> error: internal error unable to execute QEMU command 'change': Could not open
> '/testnfs/kits/os/Fedora/14/x86_64/Fedora-14-x86_64-DVD.iso'
> is the verification enough?  or need other settings and verify points?

It seems to me that libvirt can't read the file. Are the read permissions ok? Maybe you could try on read-only local filesystem first.

--- Additional comment from zpeng on 2011-06-28 13:08:55 CEST ---

Hi David:
i re-test this , seems the bug fixed
 setup read-only nfs on 10.66.4.239
#cat /etc/exports
/tmp/nfs *(ro)
on test machine :
#mount -t nfs4 10.66.4.239:/tmp/nfs /mnt
using libvirt-0.8.7-18.el6
# virsh attach-disk demo /mnt/boot.iso hdc --type cdrom
error: Failed to attach disk
error: unable to set user and group to '107:107' on '/mnt/boot.iso': Invalid argument

clear env.

update to libvirt-0.9.2-1.el6
# virsh attach-disk demo /mnt/boot.iso hdc --type cdrom
Disk attached successfully

--- Additional comment from rwu on 2011-07-06 12:31:56 CEST ---

Set it as VERIFIED per comment9

--- Additional comment from laine on 2011-09-28 08:26:23 CEST ---

*** Bug 712170 has been marked as a duplicate of this bug. ***

--- Additional comment from yury on 2011-09-28 09:39:26 CEST ---

Perfect, thanks!

--- Additional comment from errata-xmlrpc on 2011-12-06 12:06:34 CET ---

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html

Comment 1 Laine Stump 2012-10-04 17:35:22 UTC
initrd and kernel files are both already chowned using the same function as disk images (virSecurityDACSetOwnership()), so if there is a problem, that isn't it.

Please include the exact error message you are seeing (either from libvirtd.log, or from the guest's qemu logfile if that's where the problem is occurring) as well as the build of libvirt and qemu-kvm on your system.

Comment 2 David Jaša 2012-10-05 13:18:09 UTC
Created attachment 622165 [details]
libvirtd.log

qemu log says that permission to the kernel file is denied but there is no apparent problem that should cause it:


qemu.log:
/usr/libexec/qemu-kvm <snip>  -kernel /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux/vmlinuz -initrd /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux/initrd.img -append repo=http://download.englab.brq.redhat.com//mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os <snip>
char device redirected to /dev/pts/13
qemu: could not load kernel '/mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux/vmlinuz': Permission denied
2012-10-05 12:29:20.507+0000: shutting down


permissions seem to be ok for the file:
$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux/vmlinuz
-rwxr-xr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux/vmlinuz

and all the directories all the way to the / do have r+x permissions for all:
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux
drwxr-sr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os
drwxr-sr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64
drwxr-sr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora
drwxr-sr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1
drwxr-sr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt/stage
drwxrwsr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt/stage
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora/fedora-alt
drwxr-xr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora/fedora-alt
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub/fedora
drwxr-xr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub/fedora
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync/pub
drwxr-xr-x. nobody nobody system_u:object_r:nfs_t:s0       /mnt/globalsync/pub
[djasa@dhcp-29-7 ~]$ ls -ldZ /mnt/globalsync
lrwxrwxrwx. root root system_u:object_r:mnt_t:s0       /mnt/globalsync -> /net/nfs.englab.brq.redhat.com
[djasa@dhcp-29-7 ~]$ ls -ldZ /net/nfs.englab.brq.redhat.com
dr-xr-xr-x. root root system_u:object_r:autofs_t:s0    /net/nfs.englab.brq.redhat.com
[djasa@dhcp-29-7 ~]$ ls -ldZ /net
drwxr-xr-x. root root system_u:object_r:autofs_t:s0    /net
[djasa@dhcp-29-7 ~]$ ls -ldZ /
dr-xr-xr-x. root root system_u:object_r:root_t:s0      /


audit log doesn't receive any event during VM start attempt.

qemu-kvm-0.12.1.2-2.319.el6.x86_64
libvirt-0.10.2-1.el6.x86_64

Comment 5 Laine Stump 2013-07-02 06:39:21 UTC
Moving to 6.6 for capacity.

Comment 8 Jiri Denemark 2014-04-04 21:36:45 UTC
This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.

Comment 11 Erik Skultety 2015-05-21 12:35:51 UTC
(In reply to David Jaša from comment #2)
> Created attachment 622165 [details]
> libvirtd.log
> 
> qemu log says that permission to the kernel file is denied but there is no
> apparent problem that should cause it:
> 
> qemu.log:
> /usr/libexec/qemu-kvm <snip>  -kernel
> /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/
> isolinux/vmlinuz -initrd
> /mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/
> isolinux/initrd.img -append
> repo=http://download.englab.brq.redhat.com//mnt/globalsync/pub/fedora/fedora-
> alt/stage/18-Beta-TC1/Fedora/x86_64/os <snip>
> char device redirected to /dev/pts/13
> qemu: could not load kernel
> '/mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/
> isolinux/vmlinuz': Permission denied
> 2012-10-05 12:29:20.507+0000: shutting down

looking at the log file, see this line: "2012-10-05 13:09:58.499+0000: 20208: warning : virSecuritySELinuxSetFileconHelper:788 : Setting security context 'system_u:object_r:virt_content_t:s0' on '/mnt/globalsync/pub/fedora/fedora-alt/stage/18-Beta-TC1/Fedora/x86_64/os/isolinux/vmlinuz' not supported. Consider setting virt_use_nfs", with virt_use_nfs enabled in selinux I was able to start the VM successfully