Bug 225166 - Fully virtualized virt-install fails from a read-write NFS share
Fully virtualized virt-install fails from a read-write NFS share
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
5.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Daniel Berrange
: Reopened
Depends On:
Blocks: 234654 246139 240441 296411
  Show dependency treegraph
 
Reported: 2007-01-29 11:26 EST by Chris Lalancette
Modified: 2010-10-22 08:43 EDT (History)
5 users (show)

See Also:
Fixed In Version: RHEA-2007-0635
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-11-07 12:09:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Do a readonly loop device setup for readonly disks (662 bytes, patch)
2007-02-01 12:00 EST, Daniel Berrange
no flags Details | Diff

  None (edit)
Description Chris Lalancette 2007-01-29 11:26:18 EST
Description of problem:
I've finally managed to reproduce a problem using virt-install to install
fully-virtualized guests.  You must have SELinux enabled to reproduce this. 
Here's the steps.

1)  Mount remote server to serve out the boot.iso (e.g. mount
server:/path/to/RHEL/ /mnt/tmp
2)  virt-install -n rhel4fvtest -r 1024 -f /var/lib/xen/images/rhel4fvtest.dsk
-s 6 --nonsparse -c /mnt/tmp/images/boot.iso -v --vnc
3)  It will say "Starting install...", and after some time (because of the
--nonsparse), it will spew a traceback like the following:

libvir: Xen Daemon error : POST operation failed: (xend.err 'destroyDevice()
takes exactly 3 arguments (2 given)')
Failed to get devices for domain rhel4fvtest
Traceback (most recent call last):
  File "/usr/sbin/virt-install", line 447, in ?
    main()
  File "/usr/sbin/virt-install", line 411, in main
    dom = guest.start_install(conscb)
  File "/usr/lib/python2.4/site-packages/virtinst/XenGuest.py", line 367, in
start_install
    self.domain = self.conn.createLinux(cxml, 0)
  File "/usr/lib64/python2.4/site-packages/libvirt.py", line 249, in createLinux
   if ret is None:raise libvirtError('virDomainCreateLinux() failed')
libvirt.libvirtError: virDomainCreateLinux() failed

The interesting thing here is that although virt-install failed, it still
*created* the fully virt domain.  However, it never unpaused the domain. 
Interestingly, you can actually go ahead and unpause the domain (xm unpause
<dom>), but because virt-install failed it will never write out a valid
configuration.  I'm not sure what the solution is (since I understand that we
don't necessarily want to run around destroying domains), but this behavior is
confusing to end users.
Comment 1 Stephen Tweedie 2007-01-29 12:40:38 EST
I can't reproduce this: running enforcing, with an iso on NFS via autofs, it
runs fine.

What selinux avc denials are you getting?  Thanks.
Comment 2 Chris Lalancette 2007-01-29 13:23:01 EST
Here are the denials I am getting:

type=ANOM_PROMISCUOUS msg=audit(1170094440.915:25): dev=vif1.0 prom=256
old_prom=0 auid=4294967295
type=SYSCALL msg=audit(1170094440.915:25): arch=c000003e syscall=16 success=yes
exit=0 a0=3 a1=89a2 a2=7fffa5ef2a30 a3=21000 items=0 ppid=3597 pid=3690
auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
tty=(none) comm="brctl" exe="/usr/sbin/brctl"
subj=system_u:system_r:udev_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1170094441.095:26): avc:  denied  { read write } for 
pid=3742 comm="ifconfig" name="rhel4fvtest2.dsk" dev=sda3 ino=9493705
scontext=system_u:system_r:ifconfig_t:s0 tcontext=root:object_r:xen_image_t:s0
tclass=file
type=AVC msg=audit(1170094441.095:26): avc:  denied  { read } for  pid=3742
comm="ifconfig" name="boot.iso" dev=0:14 ino=28361949
scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:nfs_t:s0
tclass=file
type=SYSCALL msg=audit(1170094441.095:26): arch=c000003e syscall=59 success=yes
exit=0 a0=1a68d070 a1=1a68bb80 a2=1a68d440 a3=0 items=0 ppid=3696 pid=3742
auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
tty=(none) comm="ifconfig" exe="/sbin/ifconfig"
subj=system_u:system_r:ifconfig_t:s0 key=(null)
type=AVC_PATH msg=audit(1170094441.095:26): 
path="/mnt/bigpapi/released/RHEL-4/U4/AS/x86_64/tree/images/boot.iso"
type=AVC_PATH msg=audit(1170094441.095:26): 
path="/var/lib/xen/images/rhel4fvtest2.dsk"
type=ANOM_PROMISCUOUS msg=audit(1170094441.103:27): dev=tap0 prom=256 old_prom=0
auid=4294967295
type=SYSCALL msg=audit(1170094441.103:27): arch=c000003e syscall=16 success=yes
exit=0 a0=5 a1=89a2 a2=7fff9ac4b850 a3=21000 items=0 ppid=3696 pid=3747
auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
tty=(none) comm="brctl" exe="/usr/sbin/brctl" subj=system_u:system_r:xend_t:s0
key=(null)

Some of these seem very odd to me, so to verify, I tried the exact same
virt-install command as posted above, but I copied the boot.iso to
/var/lib/xen/images, and pointed the -c there.  The install started successfully.

Chris Lalancette
Comment 3 Chris Lalancette 2007-01-29 18:17:16 EST
As another data point:

I found that mounting an NFS share read-only allows the install to start
successfully.  It's only when mounting the very same NFS share read-write that I
run into the failure.

Chris Lalancette
Comment 4 Karl MacMillan 2007-01-31 11:23:01 EST
Can you try mounting the nfs share with the
context=system_u:object_r:xen_image_t option.
Comment 5 Chris Lalancette 2007-01-31 11:40:13 EST
Karl,
     I got the exact same failure when mounting the share with the above context
line.  I used "mount -o context=system_u:object_r:xen_image_t,bg,intr,hard
server:/vol/redhat /mnt/point"; that is, I got the same error on the console
(about destroyDevice), and I seemed to have the same errors in
/var/log/audit/audit.log

Chris Lalancette
Comment 7 Stephen Tweedie 2007-02-01 05:19:17 EST
Can you try a relabel, please?  I cannot reproduce this at all.  Also, is the
nfs iso dir exported with no_root_squash or any other non-standard options?

Comment 8 graeme 2007-02-01 05:32:59 EST
Hi

This should help reproduce, it is the original from a thread that I logged on
the xen-user list yesterday.
http://lists.xensource.com/archives/html/xen-users/2007-01/msg01305.html

So you know I am NOT running selinux and still rec. the same destroy device
error.  From my investigation it would appear that a 3rd input parameter named
"force" was added to the destroyDevice func. in DevController.py recently.

Hope this helps
G

---x---

Hello all

The chaps at XenServer directed me to this mailing list and to my surprise found
2 very recent hits on the error I am receiving.  Is it a bug perhaps?

Jordi logged this:
http://lists.xensource.com/archives/html/xen-users/2007-01/msg01270.html

and Shawn logged this one:
http://lists.xensource.com/archives/html/xen-users/2007-01/msg00892.html

Here is more detail on my problem, please let me know if you have found anything
to date.

On following the "Concise instructions for installing a CentOS VM on Xen"
http://mark.foster.cc/wiki/index.php/Centos-4_on_Xen
using a FC6 host and Oracle EL4 guest I am able to create the LVM partition,
bootstrap etc. without any errors however when I attempt to create the VM a
destroyDevice error is given.

So you know I have attempted to boot with the FC6 xen kernel first
(2.6.19-1.2895.fc6xen) and then installed the Red Hat xen kernel
vmlinuz-2.6.16.33-xen_3.0.4.1 by chroot and then rpm -ivh kernel-xen...
I wonder if the kernel is correct!?!

# xm create -c el4-1

gives...

"Using config file "/etc/xen/el4-1".
Error: destroyDevice() takes exactly 3 arguments (2 given)"

"xm list" while "xm create" is running
--------------------------------------
# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1892 2 r----- 2100.2
el4-1 24 96 1 --p--- 0.0


/etc/xen/el4-1 config file:
------------------
[root@localhost mnt]# cat /etc/xen/el4-1
kernel = "/root/boot/vmlinuz-2.6.16.33-xen_3.0.4.1"
# also used FC6: kernel = "/boot/vmlinuz-2.6.19-1.2895.fc6xen"
memory = 96
name = "el4-1"
disk = [
'phy:/dev/VolGroup01/LogVol01,sdb1,w','phy:/dev/VolGroup01/LogVol02,sdb2,w' ]
dhcp="dhcp"
root = "/dev/sdb1 ro"

guest fstab:
------------
[root@localhost mnt]# cat fstab
/dev/sdb1 / ext3 defaults 1 1
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sdb2 swap swap defaults 0 0


Comment 9 Daniel Berrange 2007-02-01 07:22:49 EST
Let me clarify *yet again* WRT to comment #8, because people seem to have a hard
time understanding this

'Error: destroyDevice() takes exactly 3 arguments (2 given)"  

Has absolutely *nothing* todo with the causes of device failure. Yes, there is a
bug in the usage of destroyDevice - this bug, however, is merely in the cleanup
path *after* hotplug has already failed. 

ie, the guest was booted. hotplug of the network/disk device failed, and thus
XenD went to tear down the domain - only at this last stage (after the real
failure) does it hit the 'destroyDevice' bug.

So, please ignore comment #8 in this ticket
Comment 10 Chris Lalancette 2007-02-01 10:17:45 EST
OK, so it seems like the SELinux denials are just a red herring.  I tried again
with a read-write NFS share with SELinux in permissive mode, and it still
failed.  Then I went back and remounted the NFS share in read-only mode, and I
was able to successfully start the install.  Finally, I turned SELinux back to
Enforcing, and tried again with the read-only share, and I was able to
successfully start the install again.  So I'm changing the summary here; while
the denials are odd (and may represent weird things that QEMU is doing), they
aren't a problem with SELinux or the policy (I don't think).

Chris Lalancette
Comment 12 Chris Lalancette 2007-02-01 10:31:54 EST
OK, more information from Dan Berrange:

<danpb> clalance: aahhhhh, easy test case
<danpb>  losetup /dev/loop0  /mnt/tmp/images/boot.iso
<danpb> that will work for a readonly NFS mount,  fail for a read-write one
<clalance> danpb: Oh, I see.  That is a good test case.
<clalance> I wonder if that is expected behavior.
<danpb> clalance: losetup is trying to open the file RW always
<danpb> now the actual ISO on NFS is read only
<danpb> so when NFS is mounted  RW, the open call  gets  -1  EACCESS
<danpb> but when NFS is mounted  RO,  the open call gets  -1   EROFS
<danpb> and in the EROFS  case, losetup will automatically fallback to trying to
create a read only looop device
<danpb> but it doesn't do this fallback for the EACCESS case
<danpb> clalance: ultimately this is a bug in Xen hotplug scripts
<danpb> because  we told Xen that we wanted a   readonly   CDROM device
<danpb> so losetup should be  given the  -r  flag to force it to do a read only
loopdevice regardless

Chris Lalancette
Comment 13 Daniel Berrange 2007-02-01 12:00:52 EST
Created attachment 147118 [details]
Do a readonly loop device setup for readonly disks

The attached patch changes the block hotplug script so that if the virtual disk
is exported to the guest in read only mode, we explicitly ask for a readonly
loop device. eg, using losetup -r. This should solve problems in exporting a
readonly file, from a read-write NFS mount
Comment 14 graeme 2007-02-01 12:25:33 EST
Hi Daniel
Sorry about comment 8 and thanks for the explanation.
Take care
G
Comment 15 Daniel Berrange 2007-02-11 12:26:45 EST
Merged upstream at:

http://xenbits2.xensource.com/xen-unstable.hg?rev/0c17f16f9f03

But the patch was later reverted in

http://xenbits2.xensource.com/xen-unstable.hg?rev/67faf6c59bfa

It was reverted because older versions of losetup do not support -r. Since we
don't care about old version of losetup, so recommend we keep this patch in our
tree. There isn't any other easy way to solve this problem without the -r option
that I can think of.
Comment 17 Daniel Walsh 2007-02-13 16:00:09 EST
If you look at the file context of the destination file you will see that it is
user_home_t,  You probably created the image file in your homedir and then used
the mv command to place it in /var/llib/xen/images.  The mv command maintains
the MAC label so you end up with a file labeled user_home_t instead of xend_lib_t.  
restorecon /var/lib/xen/images/boot.iso
will fix the labeleing.

If you installed setroubleshoot, it would have translated this error message for
you.  If you use install instead of mv it will do the right thing, or
use mv followed by restorecon.  Think of the MAC privs in this case the same way
you would of the ownership.  If you moved a file from your home dir to this
directory it would still be owned by you, and you would have to do a chown.  
Comment 18 Archana K. Raghavan 2007-02-13 16:12:47 EST
Reopening this since this is not an selinux problem .... that was just one
update from IBM.
Comment 19 Daniel Berrange 2007-02-13 16:26:43 EST
The patch I attached in comment #13  and also clarified in comment #15 is the
solution we need to track for this ticket. SELinux is just a diversion - its
nothing todo with the core problem here - the core problem is that loop devices
must be setup readonly if the image ISO itself is readonly. Hence my patch to
call losetup with the -r flag.
Comment 20 Konrad Rzeszutek 2007-04-23 15:58:25 EDT
 (In reply to comment #19)   
> The patch I attached in comment #13  and also clarified in comment #15 is the   
> solution we need to track for this ticket. SELinux is just a diversion - its   
> nothing todo with the core problem here - the core problem is that loop   
devices   
> must be setup readonly if the image ISO itself is readonly. Hence my patch to   
> call losetup with the -r flag.   
>    
   
Daniel,   
   
Will this  patch (comment #13) make it in RHE5 U1?  
Comment 21 RHEL Product and Program Management 2007-04-24 11:43:42 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 24 Daniel Berrange 2007-06-28 17:48:18 EDT
Built in xen-3.0.3-31.el5 into dist-5E-qu-candidate
Comment 26 Chris Lalancette 2007-07-02 15:39:18 EDT
I just tested this out with the latest 5.1 Beta packages, and all seems well;
with the same NFS mount that was causing problems before, I am now able to start
installs properly.

Chris Lalancette
Comment 29 errata-xmlrpc 2007-11-07 12:09:30 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2007-0635.html

Note You need to log in before you can comment on or make changes to this bug.