Bug 472236

Summary: KVM guests cannot boot PXE "local"
Product: [Fedora] Fedora Reporter: James Laska <jlaska>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: high    
Version: rawhideCC: anton, berrange, bkearney, bpeck, clalance, crobinso, dwmw2, ewan, gfa, itamar, jaswinder, jforbes, jturner, k.georgiou, lars, lutter, mgregg, mhlavink, ondrejj, quintela, renich, shawn.starr, tburke, virt-maint
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=728010
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 728010 (view as bug list) Environment:
Last Closed: 2012-07-24 15:15:23 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
Screenshot
none
Guest XML configuration
none
/var/log/libvirt/qemu/vguest2.log
none
vguest1.xml (w/ multiple <boot> and dev="virtio")
none
vguest1.xml (w/ multiple <boot> and dev="e1000")
none
no prompt none

Description James Laska 2008-11-19 10:07:54 EST
Description of problem:

All QA automated systems rely on PXE local booting for proper provisioning and testing.  All systems are configured in the BIOS to boot PXE first.

When we want to provision the systems, we modify the PXE target (using RHTS or now cobbler).

When we want to boot locally to run tests, we set the default PXE target to "local".

KVM guests do no honor the PXE "local" target.  It seems that once you boot PXE, KVM doesn't attach the already installed disks.

Version-Release number of selected component (if applicable):

kernel-2.6.27.5-113.fc10.x86_64
libvirt-0.4.6-3.fc10.x86_64
kvm-74-5.fc10.x86_64

How reproducible:

Every time.

Steps to Reproduce:
1. Set KVM guest PXE target to "Network Boot" using virt-manager
2. Boot the KVM guest.
3. In the PXE menu, type "local"
  
Actual results:

 * See attached screenshot, xml, and libvirt logfile.

Expected results:

The system should behave as a "real" system behaves and boot the local disk.

Additional info:

 * This makes adding KVM guests into test automation a bit funky since we'll need to do a workaround which involves:

When you want to reprovision a guest:
 1) virsh destroy $GUEST
 2) virsh undefine $GUEST
 3) Edit xml to boot off network
 4) virsh define $XMLFILE
 5) virsh start $GUEST

We'd then need to repeat to have it boot to local disk.
Comment 1 James Laska 2008-11-19 10:08:23 EST
Created attachment 324048 [details]
Screenshot
Comment 2 James Laska 2008-11-19 10:08:44 EST
Created attachment 324049 [details]
Guest XML configuration
Comment 3 James Laska 2008-11-19 10:09:06 EST
Created attachment 324050 [details]
/var/log/libvirt/qemu/vguest2.log
Comment 4 Michael DeHaan 2008-11-19 10:25:27 EST
Being able to boot KVM-via-PXE statefully would be highly useful for my testing in Cobbler land as well, and would help with virtual deployment (and re-deployment) of non-Linux guests.
Comment 5 Daniel Berrange 2008-11-19 10:33:38 EST
The XML only specifies a single device for booting. Can you try setting multiple devices

    <boot dev='network'/>
    <boot dev='cdrom'/>
    <boot dev='hd'/>

Which should tell the BIOS to try to boot network, then cdrom, then harddisk in that order.
Comment 6 James Laska 2008-11-19 10:47:41 EST
Using ...

  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='network'/>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>

Results in ...

# cat /var/log/libvirt/qemu/vguest2.log 
/usr/bin/qemu-kvm -S -M pc -m 1024 -smp 2 -name vguest2 -monitor pty -boot ndc -drive file=/dev/VolGroup00/vguest2,if=virtio,index=0,boot=on -net nic,macaddr=54:52:00:29:89:e5,vlan=0,model=virtio -net tap,fd=16,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -vnc 127.0.0.1:1 -k en-us 
char device redirected to /dev/pts/3
char device redirected to /dev/pts/4
Too many option ROMS

Amy I doing that right?
Comment 7 Cole Robinson 2008-11-19 10:50:57 EST
Wow! I didn't know you could specify multiple boot devs. Using

    <boot dev='network'/>
    <boot dev='hd'/>

And then pressing 'q' to not boot from networking successfully boots from disk. James, try just the above and see if it does the job for you.
Comment 8 Michael DeHaan 2008-11-19 11:13:53 EST
Cole, what we are looking for is when the bootloader is fed the following PXE configuration it should boot from the local disk:

DEFAULT local
PROMPT 0
TIMEOUT 0
TOTALTIMEOUT 0
ONTIMEOUT local

LABEL local
        LOCALBOOT 0


This will enable us to create a KVM "empty shell" that we can assign what OS it is running just based on changing the PXE configuration.

Pressing "q" would be interactive and less useful -- you'd have to catch it really really quickly or you'd be reinstalling.
Comment 9 James Laska 2008-11-19 11:18:52 EST
(In reply to comment #7)
> Wow! I didn't know you could specify multiple boot devs. Using
> 
>     <boot dev='network'/>
>     <boot dev='hd'/>
> 
> James, try just the above and see if it does the job for you.

With those options in my XML ... my guest fails to start.

# virsh dumpxml vguest2 | grep -C2 "<boot"
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='network'/>
    <boot dev='hd'/>
  </os>
  <features>

# virsh start vguest2
libvir: QEMU error : internal error QEMU quit during monitor startup
error: Failed to start domain vguest2

# tail /var/log/libvirt/qemu/vguest2.log 
/usr/bin/qemu-kvm -S -M pc -m 1024 -smp 2 -name vguest2 -monitor pty -boot nc -drive file=/dev/VolGroup00/vguest2,if=virtio,index=0,boot=on -net nic,macaddr=54:52:00:29:89:e5,vlan=0,model=virtio -net tap,fd=12,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb -vnc 127.0.0.1:1 -k en-us 
char device redirected to /dev/pts/3
char device redirected to /dev/pts/4
Too many option ROMS

What am I missing?
Comment 10 Cole Robinson 2008-11-19 11:26:01 EST
jlaska: hmm, works on F9. sounds like a bug.

mdehaan: you may just have to test it and see what happens. I let the guest boot to our pxe server which doesn't seem to have an explicit 'local' option. Hitting enter without a selection seems to imply local, but qemu then prompts for the boot from (n)etwork or (q)uit. 

Maybe qemu is smart enough to notice a 'boot from local' directive from the PXE server, and won't prompt. You'll just have to test it since I'm not sure how to go about it.
Comment 11 Michael DeHaan 2008-11-19 11:29:08 EST
Cole, that's what james was trying to do above when he filed the bug, and I watched it happen.

"""
KVM guests do no honor the PXE "local" target.  It seems that once you boot
PXE, KVM doesn't attach the already installed disks.
"""

What specifically should I test?
Comment 12 Cole Robinson 2008-11-19 11:40:55 EST
I just wasn't sure if:

not entering a selection on my pxe server & pressing enter == deliberately selecting 'boot from local' on another pxe server == having the pxe server tell the machine/VM 'hey, boot from local' (which is what I understand RHTS does).

If those are all equivalent, then it sounds like qemu needs fixing to not prompt based on the pxe request.
Comment 13 James Laska 2008-11-19 11:47:41 EST
My take on this bug is that the F10 kvm/libvirt doesn't let me specify multiple <boot> options.  If that were fixed, I suspect it would open the door for PXE "local" booting.
Comment 14 Daniel Berrange 2008-11-19 12:05:05 EST
Yes, this is a bug in KVM. The trouble is the new -drive flag and its boot=on syntax is broken wrt to normal -boot arg. We need to use boot=on for VirtIO based disks, but when we do that, then this conflicts with the option ROM for PXE boot. This is a big mess and I'm not sure how to fix it, but it certainly needs addressing somehow, because this is a valid use case
Comment 15 Bug Zapper 2008-11-26 00:36:24 EST
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 16 Glauber Costa 2008-11-26 08:24:49 EST
James,

Do you still have this problem if you switch from virtio to e1000?

You should use this XML excerpt:
    <boot dev='network'/>
    <boot dev='hd'/>
Comment 17 James Laska 2008-11-26 09:29:52 EST
Created attachment 324720 [details]
vguest1.xml (w/ multiple <boot> and dev="virtio")

Glauber, 

Yeah, I still seem to have this problem using virtio.

# virsh start vguest1
libvir: QEMU error : internal error QEMU quit during monitor startup
error: Failed to start domain vguest1

# cat /var/log/libvirt/qemu/vguest1.log 
/usr/bin/qemu-kvm -S -M pc -m 1024 -smp 2 -name vguest1 -monitor pty -boot nc -drive file=/dev/VolGroup00/vguest1,if=ide,index=0,boot=on -drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=54:52:00:55:c8:17,vlan=0,model=virtio -net tap,fd=14,script=,vlan=0,ifname=vnet2 -serial pty -parallel none -usb -vnc 127.0.0.1:3 -k en-us 
char device redirected to /dev/pts/8
char device redirected to /dev/pts/9
Too many option ROMS

# virsh dumpxml vguest1
 <!-- see attachment -->
Comment 18 James Laska 2008-11-26 09:32:04 EST
Created attachment 324721 [details]
vguest1.xml (w/ multiple <boot> and dev="e1000")

Now with dev="e1000"

# virsh start vguest1
libvir: QEMU error : internal error QEMU quit during monitor startup
error: Failed to start domain vguest1

# cat /var/log/libvirt/qemu/vguest1.log 
/usr/bin/qemu-kvm -S -M pc -m 1024 -smp 2 -name vguest1 -monitor pty -boot nc -drive file=/dev/VolGroup00/vguest1,if=ide,index=0,boot=on -drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=54:52:00:55:c8:17,vlan=0,model=e1000 -net tap,fd=19,script=,vlan=0,ifname=vnet2 -serial pty -parallel none -usb -vnc 127.0.0.1:3 -k en-us 
char device redirected to /dev/pts/8
char device redirected to /dev/pts/9
Too many option ROMS
Comment 19 Glauber Costa 2008-11-26 12:57:58 EST
I believe the problem itself is very simple (although I don't really know a good solution without thinking a little bit...)

there's only 64k of memory available for option roms, and the virtio rom that ships with our packages is... 64k in size!. So after loading the virtio PXE option rom, we're unable to keep loading option roms, in particular, the extboot option rom we need to kick out virtio boots. ;-(

James said he could boot with an older rom I handled to him, which is 32k in size,
and the problem os "Too many option ROMS" went away.

However, he was still unable to boot from the local target, despite of the fact that he could do a local boot by pressing "q" 

So we really have two problems in here:

The first one is that we cannot boot from our current virtio ROM, because it is too large. We can try to quick fix it by building smaller images. This should be a new BZ agains the etherboot package.

And the other, the fact that roms do not honor the local target. For that, I believe we can keep using this BZ.
Comment 20 James Laska 2008-11-26 13:35:06 EST
(In reply to comment #19)
> So we really have two problems in here:
> 
> The first one is that we cannot boot from our current virtio ROM, because it is
> too large. We can try to quick fix it by building smaller images. This should
> be a new BZ agains the etherboot package.

Filed this as bug#473137
Comment 21 Mark McLoughlin 2009-10-13 02:40:37 EDT
Apparently this is still a problem with gPXE:

http://www.redhat.com/archives/fedora-virt/2009-October/msg00052.html

Glauber - please take a look
Comment 22 Fedora Admin XMLRPC Client 2010-03-09 11:54:05 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 23 Fedora Admin XMLRPC Client 2010-03-09 12:19:54 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 24 Bug Zapper 2010-03-15 08:09:58 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 25 Jan ONDREJ 2010-05-21 06:43:31 EDT
Still problem on Fedora 13 final + updates testing. Any change to fix this?
Comment 26 Jan ONDREJ 2010-05-21 06:56:35 EDT
I have some success to boot using PXE by booting manually. May be there is too short default timeout for dhcp request. Try this:

1. start virtual machine
2. when you are prompted to press CTRL-B do it
3. try to get dhcp address running this command: dhcp net0
4. repeat step 3 until you do not get address (reply "ok")
5. boot using command: autoboot

If you run "dhcp net0" command immediatelly, it will fail fist time, but second run gets IP address. Then I am able to boot from PXE.
Comment 27 Jan ONDREJ 2010-08-18 14:53:20 EDT
I think local boot works well on current fedora 13 stable. Do you still have this problem?

But another problem described here (timeout to boot from PXE) is still present. Should I open a new bug for this? Looks like it's enough to increase PXE network timeout by aprox. 3 seconds. Most simpler workaround is to select "Send Key -> Ctrl-Alt-Del" from menu immediatelly (or after 1-3 seconds) after guest start.
Comment 28 Michael Gregg 2010-09-29 15:18:05 EDT
I'm still having this dhcp timeout issue on f13. 

Opened https://bugzilla.redhat.com/show_bug.cgi?id=638735 to track it.
Comment 29 Bug Zapper 2011-06-02 14:23:57 EDT
This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 30 Bug Zapper 2011-06-27 10:02:15 EDT
Fedora 13 changed to end-of-life (EOL) status on 2011-06-25. Fedora 13 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 31 Shawn Starr 2011-06-27 19:30:42 EDT
Reopen, bump to rawhide, I haven't been able to test this recently.
Comment 32 Fedora Admin XMLRPC Client 2012-03-15 13:58:18 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 33 Cole Robinson 2012-07-10 08:12:06 EDT
With virt-manager on F17 this works for me, you just need to make sure that both network and harddrive boot options are selected, otherwise the disks aren't marked as bootable and things probably won't work.

Closing as WORKSFORME, please reopen if anyone still has issues on F17+
Comment 34 Renich Bon Ciric 2012-07-24 14:20:01 EDT
Created attachment 600144 [details]
no prompt
Comment 35 Renich Bon Ciric 2012-07-24 14:20:48 EDT
it seems it's not even prompting for ipxe now. I think something got hardcoded into the rom by accident.

Can somebody verify?
Comment 36 Cole Robinson 2012-07-24 15:15:23 EDT
Renich, given how old and long this bug report is, let's keep it closed. If you are still experiencing a similar issue, please open a new bug report with the following info:

Fedora version
qemu version
qemu command line (if using libvirt, /var/log/libvirt/qemu/$vmname.log)


At least on F17, PXE and boot from local is working fine for me.