Bug 539746 - launch fails when run inside a Xen guest, when no non-PV kernels are installed
launch fails when run inside a Xen guest, when no non-PV kernels are installed
Status: ASSIGNED
Product: Virtualization Tools
Classification: Community
Component: libguestfs (Show other bugs)
unspecified
All Linux
medium Severity medium
: ---
: ---
Assigned To: Richard W.M. Jones
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-11-20 18:13 EST by Orion Poplawski
Modified: 2011-04-29 03:31 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Orion Poplawski 2009-11-20 18:13:28 EST
Description of problem:

trying out libguestfs for the first time.  Trying to debug a virtual machine that won't boot.

# virt-ls /dev/rootvg/xenfdev32 /boot
new guestfs handle 0x14b5e760
looking for supermin appliance in /usr/lib64/guestfs
launch: external command failed: PATH='/usr/lib64/guestfs':$PATH libguestfs-supermin-helper '/usr/lib64/guestfs' /tmp/libguestfsGF6gd6/kernel /tmp/libguestfsGF6gd6/initrd at /usr/bin/virt-ls line 173.
closing guestfs handle 0x14b5e760 (state 0)

# guestfish -i /dev/rootvg/xenfdev32
new guestfs handle 0x210a550
new guestfs handle 0x1ac6db40
looking for supermin appliance in /usr/lib64/guestfs
launch: external command failed: PATH='/usr/lib64/guestfs':$PATH libguestfs-supermin-helper '/usr/lib64/guestfs' /tmp/libguestfsOtQT18/kernel /tmp/libguestfsOtQT18/initrd at /usr/bin/virt-inspector line 217.
closing guestfs handle 0x1ac6db40 (state 0)
closing guestfs handle 0x210a550 (state 0)

Version-Release number of selected component (if applicable):
libguestfs-1.0.76-1.el5.4
Comment 1 Richard W.M. Jones 2009-11-24 04:40:18 EST
Please can you run:

libguestfs-test-tool

and attach the complete output here.
Comment 2 Richard W.M. Jones 2009-11-24 04:41:23 EST
And also check that multilib hasn't messed up
your installation:

rpm -V libguestfs
Comment 3 Orion Poplawski 2009-11-24 11:36:17 EST
# libguestfs-test-tool
===== Test starts here =====
LIBGUESTFS_DEBUG=1
new guestfs handle 0x10803150
library version: 1.0.76
guestfs_get_append: (null)
guestfs_get_autosync: 0
guestfs_get_memsize: 500
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_qemu: /usr/bin/qemu-system-x86_64
guestfs_get_verbose: 1
Launching appliance, timeout set to 120 seconds.
looking for supermin appliance in /usr/lib64/guestfs
libguestfs: error: external command failed: PATH='/usr/lib64/guestfs':$PATH libguestfs-supermin-helper '/usr/lib64/guestfs' /tmp/libguestfsNxWETA/kernel /tmp/libguestfsNxWETA/initrd
libguestfs-test-tool: failed to launch appliance
closing guestfs handle 0x10803150 (state 0)

# ls /usr/lib64/guestfs/
initramfs.epel-5.x86_64.supermin.hostfiles  kmod.whitelist
initramfs.epel-5.x86_64.supermin.img

No multi-lib, rpm -V checks out fine.
Comment 4 Richard W.M. Jones 2009-11-24 11:54:10 EST
Orion, I wonder if you have limited disk space on /tmp,
although it _should_ print an error message.

In any case is it possible you could do this (you don't
need to be root):

  libguestfs-supermin-helper /usr/lib64/guestfs /tmp/kernel /tmp/initrd
  echo $?

That should work without any error, and the status code should
be 0.

After running that you should get two files which will be
similar (but not exactly the same) to this:

  ls -lh /tmp/kernel /tmp/initrd 
  -rw-rw-r-- 1 rjones rjones 69M 2009-11-24 16:51 /tmp/initrd
  lrwxrwxrwx 1 rjones rjones  46 2009-11-24 16:51 /tmp/kernel -> /boot/vmlinuz-2.6.32-0.33.rc5.git1.fc13.x86_64
Comment 5 Richard W.M. Jones 2009-11-24 12:12:27 EST
FYI, this is from a RHEL 5.4 64 bit machine with
the same version of libguestfs as you reported:

 $ libguestfs-supermin-helper /usr/lib64/guestfs /tmp/kernel /tmp/initrd
 $ echo $?
 0
 $ ls -lh /tmp/kernel /tmp/initrd 
 -rw-r--r-- 1 rjones rjones 96M Nov 24 17:12 /tmp/initrd
 lrwxrwxrwx 1 rjones rjones  28 Nov 24 17:11 /tmp/kernel -> /boot/vmlinuz-2.6.18-164.el5
 $ rpm -q libguestfs
 libguestfs-1.0.76-1.el5.4

You wouldn't expect the /tmp/kernel and /tmp/initrd files to
be exactly the same, but they should be broadly similar.
Comment 6 Orion Poplawski 2009-11-24 12:24:09 EST
[root@hammer ~]# libguestfs-supermin-helper /usr/lib64/guestfs /tmp/kernel /tmp/initrd
[root@hammer ~]# ls -l /tmp
total 0
drwx------ 2 root root 60 Nov 24 10:22 ssh-wyWkq20901
drwx------ 2 root root 60 Nov 24 09:25 ssh-ynlri20346
Comment 7 Orion Poplawski 2009-11-24 12:26:21 EST
[root@hammer ~]# libguestfs-supermin-helper /usr/lib64/guestfs /tmp/kernel /tmp/initrd
[root@hammer ~]# echo $?
1
[root@hammer ~]# df /tmp
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 1.8G     0  1.8G   0% /tmp
Comment 8 Orion Poplawski 2009-11-24 12:29:34 EST
I think I might have it:

21088 stat("/boot/vmlinuz-*.x86_64*", 0xbf0e578) = -1 ENOENT (No such file or directory)       
21088 lstat("/boot/vmlinuz-*.x86_64*", 0xbf0e578) = -1 ENOENT (No such file or directory)      

ls /boot/vmlinuz*
/boot/vmlinuz-2.6.18-164.2.1.el5xen  /boot/vmlinuz-2.6.18-164.el5virttest17xen
/boot/vmlinuz-2.6.18-164.6.1.el5xen


I need some guest kernels installed?
Comment 9 Richard W.M. Jones 2009-11-24 12:38:06 EST
Ah ha ... you're running this inside Xen, right?

This is actually a proper bug in the script in that case,
or perhaps more accurately it's a missing dependency.

It's looking for a bootable kernel (one that can be booted
in qemu, which Xen kernels can't be).  And so that script
expects to find a non-Xen kernel in /boot which it can use.

So it does:

kernels=$(ls -1vr /boot/vmlinuz-*.$arch* 2>/dev/null | grep -v xen; ls -1vr /boot/vmlinuz-* 2>/dev/null | grep -v xen)

If you install a non-Xen kernel in your Xen guest, it should
work, although libguestfs will likely be quite slow because
it won't be able to use hardware virtualization[1].

(BTW, /usr/bin/libguestfs-supermin-helper is just a shell
script)

[1] http://libguestfs.org/FAQ.html#slow
Comment 10 Richard W.M. Jones 2009-11-24 12:39:06 EST
This code is supposed to print an error to stderr.  I
wonder why it didn't do that?

if [ -z "$modpath" ]; then
    echo "$0: failed to find a suitable kernel" >&2
    exit 1
fi
Comment 11 Orion Poplawski 2009-11-24 12:59:23 EST
Never gets there because kernels is empty:

# sh -x libguestfs-supermin-helper /usr/lib64/guestfs /tmp/kernel /tmp/initrd
+ unset CDPATH
+ set -e
++ cd /usr/lib64/guestfs
++ pwd
+ sourcedir=/usr/lib64/guestfs
+ kernel=/tmp/kernel
+ initrd=/tmp/initrd
++ echo x86_64
++ sed 's/^i.86$/i?86/'
+ arch=x86_64
++ ls -1vr '/boot/vmlinuz-*.x86_64*'
++ grep -v xen
+ kernels=
Comment 12 Richard W.M. Jones 2009-11-24 13:20:04 EST
(In reply to comment #11)
> Never gets there because kernels is empty:

Yup, that'll be a problem :-)

I pushed this patch upstream:
https://www.redhat.com/archives/libguestfs/2009-November/msg00225.html

and the same to febootstrap:
http://git.annexia.org/?p=febootstrap.git;a=commitdiff;h=a1c6fbdf00f249ddaadc733afaa5ac6e8fbcc30e

I think the best thing here is if you could install the kernel
package and try again.  *Note* that I don't mean you have to
boot your Xen guest with a fullvirt kernel (which probably
wouldn't work).  It just needs to use that kernel in order to
boot the libguestfs appliance using qemu.
Comment 13 Orion Poplawski 2009-11-24 13:26:07 EST
Installed "kernel" and getting much farther:

# guestfish -i /dev/rootvg/xenfdev32
Could not open '/dev/kqemu' - QEMU acceleration layer not activated: No such file or directory
get_e2uuid: get_e2uuid: tune2fs -l: /sbin/tune2fs: Filesystem has unsupported feature(s) while trying to open /dev/hda1
Couldn't find valid filesystem superblock. at /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Sys/Guestfs/Lib.pm line 693

I'm trying to probe the filesystem types for the filesystems in /dev/rootvg/xenfdev32 because I suspect that anaconda in F-13 is forcing /boot to be ext4 even though I specify ext3.  I was hoping that libguestfs could help determine that, but I really have no idea how to use it.

Thanks for all the help.
Comment 14 Richard W.M. Jones 2009-11-24 13:40:09 EST
(In reply to comment #13)
> Installed "kernel" and getting much farther:
> 
> # guestfish -i /dev/rootvg/xenfdev32
> Could not open '/dev/kqemu' - QEMU acceleration layer not activated: No such
> file or directory
> get_e2uuid: get_e2uuid: tune2fs -l: /sbin/tune2fs: Filesystem has unsupported
> feature(s) while trying to open /dev/hda1
> Couldn't find valid filesystem superblock. at
> /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Sys/Guestfs/Lib.pm
> line 693
> 
> I'm trying to probe the filesystem types for the filesystems in
> /dev/rootvg/xenfdev32 because I suspect that anaconda in F-13 is forcing /boot
> to be ext4 even though I specify ext3.  I was hoping that libguestfs could help
> determine that, but I really have no idea how to use it.

This is a separate bug in virt-inspector (which is a sub-
package of libguestfs).  Can you open another bug about it?

For testing this, I'd start simple:

 guestfish -a /dev/rootvg/xenfdev32

Type "run" at the prompt.

Then try out some guestfish commands, eg:

 list-devices
 list-partitions
 lvs
 file /dev/hda1
 vfs_type /dev/hda1
 mount /dev/hda1 /

There's much more about guestfish in the man page:
http://libguestfs.org/guestfish.1.html
and in the recipes page:
http://libguestfs.org/recipes.html

> Thanks for all the help.  

And thanks for your persistence!
Comment 15 Richard W.M. Jones 2009-11-24 13:41:44 EST
(In reply to comment #14)
>  guestfish -a /dev/rootvg/xenfdev32

If you don't want to modify the image, do:

 guestfish --ro -a /dev/rootvg/xenfdev32
Comment 16 Orion Poplawski 2009-11-24 13:45:22 EST
I think now I need access to the "e4" versions of the "e2" utilities in guestfish.  I think I can hack up a version for that.
Comment 17 Richard W.M. Jones 2009-11-24 13:49:26 EST
(In reply to comment #16)
> I think now I need access to the "e4" versions of the "e2" utilities in
> guestfish.  I think I can hack up a version for that.  

Right ... RHEL 5.4 has the non-standard ext4dev type and
e4fsprogs.  So that'll be the second area where you are
exploring new territory :-)
Comment 18 Richard W.M. Jones 2009-11-24 13:50:26 EST
Updating summary to more accurately reflect the bug.
Comment 19 Richard W.M. Jones 2009-11-24 13:52:15 EST
This is a Fedora EPEL packaging bug.  Changing Product
field to reflect that.
Comment 20 Richard W.M. Jones 2009-11-24 13:53:38 EST
I talked to Dan Berrange and he things the best way to
solve this is to add:

 Requires: kernel

into the spec file.  However this will have the unfortunate
side-effect that users will have a non-Xen kernel added to
their grub.conf (if they are using this in a Xen guest --
it won't affect anyone else).
Comment 21 Richard W.M. Jones 2009-11-24 13:59:08 EST
On second thoughts, kernel is a virtual provides of
kernel-xen, so we can't just do that.
Comment 22 Orion Poplawski 2009-11-24 14:00:13 EST
It at least isn't installed as the default:

default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
# hiddenmenu
title CentOS (2.6.18-164.6.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.6.1.el5 ro root=/dev/rootvg/root crashkernel=128M@16M
        initrd /initrd-2.6.18-164.6.1.el5.img
title CentOS (2.6.18-164.6.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-164.6.1.el5
        module /vmlinuz-2.6.18-164.6.1.el5xen ro root=/dev/rootvg/root crashkernel=128M@16M
        module /initrd-2.6.18-164.6.1.el5xen.img
Comment 23 Richard W.M. Jones 2009-11-24 16:55:14 EST
Moving Product back to Virtualization.  This isn't just a simple
packaging bug ...
Comment 24 Richard W.M. Jones 2009-11-25 05:04:00 EST
BTW, for recovering broken VMs you might want to
look at virt-rescue which gives you a shell:

http://libguestfs.org/virt-rescue.1.html
Comment 25 Marek Goldmann 2011-03-09 02:21:37 EST
I can confirm that for fully updated CentOS 5 host installing kernel package on this host is sufficient to make libguestfs work.

Note You need to log in before you can comment on or make changes to this bug.