Bug 221407

Summary: xen vm built with virt-install on file-backed disk won't boot
Product: [Fedora] Fedora Reporter: J Robinson <jrobinson852>
Component: xenAssignee: Xen Maintainance List <xen-maint>
Status: CLOSED WONTFIX QA Contact: Martin Jenner <mjenner>
Severity: medium Docs Contact:
Priority: medium    
Version: 6CC: bstein, katzj, triage
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard: bzcl34nup
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-05-06 13:17:50 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On: 336201    
Bug Blocks:    
Attachments:
Description Flags
xend-debug.log as requested
none
xend.log as requested none

Description J Robinson 2007-01-04 10:18:12 EST
Description of problem:

virtual machines created via 'virt-install' (on a disk-based partition) will not
boot.

Version-Release number of selected component (if applicable):


# rpm -qa | grep xen
xen-libs-3.0.3-1.fc6
kernel-xen-2.6.18-1.2868.fc6
kernel-xen-2.6.18-1.2869.fc6 
xen-3.0.3-1.fc6

kernel-xen-2.6.18-1.2869.fc6  is the active kernel.

These tests are on a 3ghz p4 with HT & 2GB ram.


How reproducible:

Always

Steps to Reproduce:
1) Start with an updated FC6 machine on a private network (mine was 192.168.0.144). 

2) run

/usr/sbin/virt-install        -n test20070104 -f
/home/admin/xen/test20070104.disk -s 5 -r 256 -l
http://mirrors.kernel.org/fedora/core/6/i386/os/ --nographics

When the FC6 installer begins,
 
3) On "language" screen, choose 
English

4) on "Configure TCP/IP" screen, choose
don't use DHCP,
enable IPv4
don'e enable IPv6

5) on "Manual TCP/IP Configuration screen", choose:
IPv4 address:  192.168.0.145 / 255.255.255.0 (YOUR PRIVATE IP HERE)
Gateway:          192.168.0.14  (YOUR GATEWAY HERE)
Name Server: 192.168.0.144 (YOUR NAMESERVER HERE)

6) installer then shows "Retrieving images/minstg2.img... ", then runs the
anaconda installer and says "Welcome to Fedora Core!".  Choose "OK"

7) installer shows "searching for redhat installations", then asks: 

The partition table on device xvda was  unreadable. ... Would you like to
initialize this drive, erasing ALL DATA? Choose "Yes"

8) installer shows "Partitioning type" screen. Leave the default selected (xvda)
and choose "OK" and confirm on the next warning screen with "Yes"

9) Installer shows "Review Partition Layout" screen. Choose "No" to accept
default partitioning layout without review or modification.

10) Configure guest's eth0 network configuration. Should be set with data from
step 5. Make sure DHCP and IPv6 are disabled, and that "activate on boot" is
enabled, and choose "OK"

11) installer shows "Miscellaneous Network Settings" screen. Choose OK to accept
settings you set up in step 5.

12) installer shows "Hostname Configuration" screen. Set the name "manually"
to "test20070104"	

13) installer shows "Time Zone Selection" Screen. Choose "System clock uses UTC"
and your time zone (I'm in EST)

14) installer shows "Root Password" screen. Input the root password for your new
VM twice and choose OK.

15) Installer shows "Retrieving installation information..."

16) installer presents "Package selections screen". Unselect "Office and
Productivity", and select "Customize software selection", and choose OK.

17) enable only "Administration Tools", "Editors" and  "Virtualization". Disable
"GNOME Desktop Environment", "Graphical Internet", "Printing Support",
"Text-based Internet" and "X Window System". 

18) Installer runs dependency check

19) Installers says "Installation to begin".  Choose "OK"

9:42AM

20) installer formats / and /boot filesystems, begins install, which takes about
30 minutes on my machine and ends with "Congratulations, your Fedora Core
installation is complete. Remove any media used during the installation process
and press <Enter> to reboot your system. "  Choose "Reboot" to reboot the VM.

21) Vm shows "rebooting system. // Restarting system. // Domain installation
still in progress.  You can reconnect to the console to complete the
installation process."

22) at the command line, type 
xm create -c test20070104

  
Actual results:

Result: 
# xm create -c test20070104
Using config file "/etc/xen/test20070104".
Traceback (most recent call last):
  File "/usr/bin/pygrub", line 494, in ?
    g = Grub(file, isconfig)
  File "/usr/bin/pygrub", line 149, in __init__
    self.read_config(file, isconfig)
  File "/usr/bin/pygrub", line 328, in read_config
    fs = fstype.open_fs(fn, offset)
  File "/usr/lib/python/grub/fsys/ext2/__init__.py", line 35, in open_fs
    return Ext2Fs(fn, offset = offset)
ValueError: unable to open file
No handlers could be found for logger "xend"
Error: Boot loader didn't return any data!
Usage: xm create <ConfigFile> [options] [vars]


Expected results:
Boot new VM and finish install or present user with a Login: prompt.

Additional info:
Comment 1 J Robinson 2007-01-04 10:22:09 EST
This process had worked at one point in FC5 (with /usr/sbin/xenguest-install.py
which was FC5's version of /usr/sbin/virt-install )
Comment 2 Daniel Berrange 2007-01-04 10:26:40 EST
I'm a little confused about what sort of disk you were trying to create. You say

"virtual machines created via 'virt-install' (on a disk-based partition) will
not boot."

which suggests you're trying a partition backed virtual disk, but the argument
you listed as giving to virt-install:

 "-f /home/admin/xen/test20070104.disk"

looks very much like a file-backed disk, rather than a physical disk.

Can you upload the '/etc/xen/test20070104' config file, and the xend.log &
xend-debug.log from /var/log/xen to this ticket, so I can see the exact config
created, and how xend interpreted it.
Comment 3 J Robinson 2007-01-04 10:38:25 EST
Hello, Daniel:

Apologies for the confusion. 

Yes, I meant "file-backed disk" (like used in the command from step 2). I've
changed the summary to reflect this.

Uploading logs and config file shortly.
Comment 4 J Robinson 2007-01-04 10:40:30 EST
xen guest Config file, as requested by Daniel:

# cat /etc/xen/test20070104
# Automatically generated xen config file
name = "test20070104"
memory = "256"
disk = [ 'tap:aio:/home/admin/xen/test20070104.disk,xvda,w', ]
vif = [ 'mac=00:16:3e:74:5d:c2, bridge=xenbr0', ]
nographic=1
uuid = "7004d98c-9f3d-7cc0-ad2a-a29739aa500e"
bootloader="/usr/bin/pygrub"
vcpus=1
on_reboot   = 'restart'
on_crash    = 'restart'
Comment 5 J Robinson 2007-01-04 10:45:02 EST
Created attachment 144806 [details]
xend-debug.log as requested

requested by Daniel.
Comment 6 J Robinson 2007-01-04 10:52:28 EST
Created attachment 144811 [details]
xend.log as requested

Let me know if you need anything else, or want me to rerun the steps with
changes
Comment 7 Red Hat Bugzilla 2007-07-24 20:05:11 EDT
change QA contact
Comment 8 J Robinson 2007-10-12 12:24:38 EDT
Any movement on this? Let me know if people would like me to retest with
currecnt fc6
Comment 9 Daniel Berrange 2007-10-12 12:42:01 EDT
FYI, using this path /home/admin/xen/test20070104.disk  will not work if SELinux
is enabld - /var/lib/xen/images is the recommended location which has neccessary
security context to run under SELinux. That said, since you did get through the
install I'm puzzelled why it would fail to later restart it. What kind of
filesystem do you have on /home ?  ext3 or something wierd like xfs ?

Also, I'd recommend updating to the latest FC6 packages for xen, libvirt,
python-virtinst and virt-manager since there's been many bugfixes since the
versions reported in the initial description of this bug.
Comment 10 J Robinson 2007-10-12 13:11:23 EDT
Hello, Daniel:

I'm not using selinux; and /home is an ext3 partition on /dev/md2, a RAID1
partition.

I have updated all the packages on the system and am running the kernel
2.6.20-1.3002.fc6xen

When I just tried, here's what I got. Let me know if you need more data.

# /usr/sbin/virt-install -b private -n test20071012d -f
/home/admin/xen/test20071012d.disk -s 5 -r 256 -l
http://mirrors.kernel.org/fedora/core/6/i386/os/ --nographics --debug

resulted in:

Fri, 12 Oct 2007 13:08:01 DEBUG    Disk path not found: Assuming file disk type.
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768
libvir: Xen Daemon error : failed Xen syscall xenDaemonDomainDumpXML domain ID <
0 and xendConfigVersion < 3 -1562082768


Starting install...
libvir: Xen error : Domain not found: xenUnifiedDomainLookupByUUID
Fri, 12 Oct 2007 13:08:01 DEBUG    Fetching URI
http://mirrors.kernel.org/fedora/core/6/i386/os//Fedora
Fri, 12 Oct 2007 13:08:02 DEBUG    Saved file to /var/lib/xen/virtinst-Fedora.AfhFBq
Retrieving file Fedora...                                        775 B 00:00 
Fri, 12 Oct 2007 13:08:02 DEBUG    Detected a Fedora distro
Fri, 12 Oct 2007 13:08:02 DEBUG    Fetching URI
http://mirrors.kernel.org/fedora/core/6/i386/os//images/xen/vmlinuz
Retrieving file vmlinuz..  90% |======================   | 1.5 MB    00:00 ETA
Fri, 12 Oct 2007 13:08:04 DEBUG    Saved file to
/var/lib/xen/virtinst-vmlinuz.8j0Te6
Retrieving file vmlinuz.. 100% |=========================| 1.6 MB    00:02     
Fri, 12 Oct 2007 13:08:04 DEBUG    Fetching URI
http://mirrors.kernel.org/fedora/core/6/i386/os//images/xen/initrd.img
Retrieving file initrd.im  98% |======================== | 5.6 MB    00:00 ETA
Fri, 12 Oct 2007 13:08:10 DEBUG    Saved file to
/var/lib/xen/virtinst-initrd.img.T5BaHy
Retrieving file initrd.im 100% |=========================| 5.7 MB    00:05     
libvir: Xen error : Domain not found: xenUnifiedDomainLookupByName
Creating storage file...  100% |=========================| 5.0 GB    00:00     A 
Fri, 12 Oct 2007 13:08:10 DEBUG    Creating guest from '<domain type='xen'>
  <name>test20071012d</name>
  <currentMemory>262144</currentMemory>
  <memory>262144</memory>
  <uuid>698cde55-b2d9-d1be-14fd-8805334effb1</uuid>
  <os>
    <type>linux</type>
    <kernel>/var/lib/xen/virtinst-vmlinuz.8j0Te6</kernel>
    <initrd>/var/lib/xen/virtinst-initrd.img.T5BaHy</initrd>
    <cmdline> method=http://mirrors.kernel.org/fedora/core/6/i386/os/</cmdline>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <vcpu>1</vcpu>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap'/>
      <source file='/home/admin/xen/test20071012d.disk'/>
      <target dev='xvda'/>
    </disk>

    <interface type='bridge'>
      <source bridge='private'/>
      <mac address='00:16:3e:03:a8:d8'/>
    </interface>


  </devices>
</domain>
'
libvir: Xen Daemon error : POST operation failed: (xend.err 'Device 0 (vif)
could not be connected. Could not find bridge device private')
Fri, 12 Oct 2007 13:08:12 DEBUG    Removing /var/lib/xen/virtinst-vmlinuz.8j0Te6
Fri, 12 Oct 2007 13:08:12 DEBUG    Removing /var/lib/xen/virtinst-initrd.img.T5BaHy
virDomainCreateLinux() failed POST operation failed: (xend.err 'Device 0 (vif)
could not be connected. Could not find bridge device private')
Domain installation may not have been
 successful.  If it was, you can restart your domain
 by running 'virsh start test20071012d'; otherwise, please
 restart your installation.
Fri, 12 Oct 2007 13:08:12 ERROR    virDomainCreateLinux() failed POST operation
failed: (xend.err 'Device 0 (vif) could not be connected. Could not find bridge
device private')
Traceback (most recent call last):
  File "/usr/sbin/virt-install", line 474, in ?
    main()
  File "/usr/sbin/virt-install", line 438, in main
    dom = guest.start_install(conscb,progresscb)
  File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 710, in
start_install
    return self._do_install(consolecb, meter)
  File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 727, in
_do_install
    self.domain = self.conn.createLinux(install_xml, 0)
  File "/usr/lib/python2.4/site-packages/libvirt.py", line 525, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: virDomainCreateLinux() failed POST operation failed: (xend.err
'Device 0 (vif) could not be connected. Could not find bridge device private')
                         
Comment 11 J Robinson 2007-10-12 13:17:19 EDT
Hello again, Daniel:

I realize this looks like a wholly different bug than I originally filed: it's
now "can't install xen instance from virt-install".

Please let me know if/how you'd like me to proceed!

Thanks
Comment 12 Daniel Berrange 2007-10-12 13:23:56 EDT
> Device 0 (vif) could not be connected. Could not find bridge device private

Do you actually have a bridge device called 'private' ?  The usual Xen bridge is
called xenbr0  or xenbr1 depending on whether you use  eth0 or eth1 as your
primary physical NIC ...
Comment 13 J Robinson 2007-10-12 13:27:43 EDT
OOPS! Earlier comments today were due to " -b private" being added to
virt-install command line. Retesting...
Comment 14 J Robinson 2007-10-12 13:49:58 EDT
OK. Sorry for the misfire earlier today.

Using: 

/usr/sbin/virt-install -n test20071012e -f /home/admin/xen/test20071012e.disk -s
5 -r 256 --os-variant=fedora6 --os-type=linux -l
http://mirrors.kernel.org/fedora/core/6/i386/os/ --nographics --debug

it stalls at the "Welcome to Fedora Core" screen. I waited several minutes.

On the bottom it says:
  <Tab>/<Alt-Tab> between elements  | <Space> selects | <F12> next screen      

and in the middle there's a series of lines similar to "├───────────── ",
(indicating some sort of installer output, which includes no text) but the
installer won't respond to any presses of Tab, Space, Enter, cntrl-C, or other
chars.

"xm list" shows:
test20071012e                              1      255     1 -b----      2.0

"xm destroy test20071012e" does kill the xen instance.

Let me know how you'd like me to proceed...
Comment 15 J Robinson 2007-10-12 14:13:55 EDT
BAH! I didn't wait long enough. the installer errors with:

Unable to retrieve  
http://mirrors.kernel.org//fedora/core/6/i386/os//images/minstg2.img.

Looking at ifconfig, it seems that the bridge maybe using a wrong ip address?

virbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:209 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:13280 (12.9 KiB)  TX bytes:8222 (8.0 KiB)

Not clear how to proceed; I'm using the same process as I posted 2007-01-04
(with --os-variant=fedora6 --os-type=linux added)
Comment 16 J Robinson 2007-10-12 14:33:06 EDT
confirmed same results with firewalls disabled
Comment 17 Daniel Berrange 2007-10-12 14:46:35 EDT
Make sure that the sysctl  setting  net.ipv4.ip_forward = 1

The virbr0  interface is connected to outside world using NAT, so requires IP
forwarding to be enabled. You shouldn't disable firewall because that breaks the
NAT rules that are added to give the VM connectivity.
Comment 18 J Robinson 2007-10-12 16:27:54 EDT
Hello, Daniel;

Thanks for your response. I believe I had that sysctl setting correct, as this
output shows:

# sysctl -A | grep ip_forward
net.ipv4.ip_forward = 1

Let me know if more info is needed.
Comment 19 Anchor Systems Managed Hosting 2007-12-12 21:45:35 EST
I had this same problem on RHEL 5.1. Upgrading to the latest kernel fixed
the problem.
Comment 20 Bug Zapper 2008-04-04 01:29:01 EDT
Fedora apologizes that these issues have not been resolved yet. We're
sorry it's taken so long for your bug to be properly triaged and acted
on. We appreciate the time you took to report this issue and want to
make sure no important bugs slip through the cracks.

If you're currently running a version of Fedora Core between 1 and 6,
please note that Fedora no longer maintains these releases. We strongly
encourage you to upgrade to a current Fedora release. In order to
refocus our efforts as a project we are flagging all of the open bugs
for releases which are no longer maintained and closing them.
http://fedoraproject.org/wiki/LifeCycle/EOL

If this bug is still open against Fedora Core 1 through 6, thirty days
from now, it will be closed 'WONTFIX'. If you can reporduce this bug in
the latest Fedora version, please change to the respective version. If
you are unable to do this, please add a comment to this bug requesting
the change.

Thanks for your help, and we apologize again that we haven't handled
these issues to this point.

The process we are following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp

We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.

And if you'd like to join the bug triage team to help make things
better, check out http://fedoraproject.org/wiki/BugZappers
Comment 21 Bug Zapper 2008-05-06 13:17:48 EDT
This bug is open for a Fedora version that is no longer maintained and
will not be fixed by Fedora. Therefore we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen thus bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.