Bug 457979

Summary: Update QEMU to use gPXE roms for iSCSI boot support
Product: [Fedora] Fedora Reporter: Matt Domsch <matt_domsch>
Component: qemuAssignee: Glauber Costa <gcosta>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: berrange, clalance, dwmw2, gcosta, katzj, k.georgiou, markmc, poelstra, shyam_iyer, virt-maint, wwlinuxengineering
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 488879 (view as bug list) Environment:
Last Closed: 2009-04-03 08:37:43 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On: 492181    
Bug Blocks: 480594, 488879    

Description Matt Domsch 2008-08-05 17:37:30 EDT
Description of problem:
To enable guests to directly boot from an iSCSI target, without libvirt one can manually start qemu-kvm  with the added command line parameter:

–option-rom pci_10ec_8139.rom 

where the .rom file came from the gPXE (etherboot) project, and provides an iSCSI initiator as an option ROM.  qemu then uses this option rom to retrieve the grub boot sectors, and it's off to the races.

libvirt needs the capability of passing such extra command line parameters to qemu.

Version-Release number of selected component (if applicable):
libvirt-0.4.4-2.fc10.x86_64
Comment 1 Daniel Berrange 2008-08-06 05:55:40 EDT
QEMU already automatically loads option ROMs for PXE boot - any reason why the updated ROM with iSCSI support can't just be integrated into QEMU so we don't have to specify a -option-rom explicitly.

Having users of libvirt need to know about loading option ROMs for booting isn't really going to lead to a good experiance. Most applications using libvirt will never even think to enable the option ROM support, even if we have it. Making it 'just work' in upstream QEMU will ensure all applications using libvirt can use the capability without needing special args.
Comment 2 Matt Domsch 2008-08-06 08:20:22 EDT
I'd be happy for qemu to start using the gPXE option ROMs directly for this capability.  But I also suspect there may be good reasons to want to pass other command line options to qemu from libvirt in the future, so having a general method to pass such options would be beneficial, even if the specific need for -option-rom isn't needed for long.
Comment 3 Daniel Berrange 2008-08-06 10:09:59 EDT
I've spoken with upstream KVM/QEMU maintainers and they would like to switch QEMU to using gPXE option ROMS by default, so I don't want to add any option ROM support into libvirt at this time. Re-assigning this bug component to track gPXE in KVM/QEMU


NB, we also explicitly do NOT enable passing of arbitrary QEMU command line args in libvirt. The libvirt XML format needs to be hypervisor agnostic and thus requires that we come up with a standardized representation for any piece of functionality. Allowing QEMU command line args would violate this requirement.
Comment 4 Matt Domsch 2008-08-06 11:14:25 EDT
that works for me.
Comment 5 Daniel Berrange 2009-03-11 08:41:45 EDT
Reassigning: The kvm package no longer exists in rawhide/F11, since it is now
part of 'qemu'.
Comment 6 Daniel Berrange 2009-03-11 09:26:08 EDT
The blocking issue for resolving this is finding a way to fit both extboot and gPXE option ROMs into QEMU's ROM space. Currently there just isn't enough room for both. 

Xen however managed to extend ROM space for their HVM guests, so its possible the same approach they used might work with QEMU/KVM too

changeset:   18977:883d01b2fd72
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Tue Jan 06 16:02:10 2009 +0000
files:       tools/firmware/hvmloader/config.h tools/firmware/hvmloader/e820.h tools/firmware/hvmloader/hvmloader.c tools/firmware/hvmloader/smbios.c tools/firmware/hvmloader/util.c tools/firmware/rombios/rombios.c
description:
rombios: Allow option ROMs to extend up to 0xEA000.
Comment 7 Glauber Costa 2009-03-12 14:21:44 EDT
I still don't see how xen does it.
I looked at the patch, and it seems crazy to me. Reason is bios starts at 0xe0000 and goes all the way up to 0x100000. So this is probably overriding bios data, unless there's a clever trick that is not in this patch itself to allow for it.

(btw, I checked xen's firmware, not bochs-bios directly, and they do have a call 0xe0000 in the middle of it...)

That said, I've been spending some time with gpxe today, and got the following:

* virtio net rom is 54k
* rtl is about the same size.

they both go well with extboot.

The only bigger rom is e1000, and it is itself bigger than the space allowed for option roms. I boot tested the ones that fit, and they seem to work on qemu, and fail in today's kvm-userspace.git (not sure why yet). The good news is that e1000 is not _that_ bigger, and it would fit if we decreased the space reserved to vgabios (it does not need it all anyway).

However, as I digged into it, I now fail to see how this would solve matt's problem. Even if we can shape up gpxe/qemu combination to a good state, qemu will only automatically load pxe roms for the network cards it recognizes. It does not seem to be the case for the rom he is trying to load. (Matt, correct me if I'm wrong). I tried to use that rom in place of the plain rtl8139, and it does not seem to work properly.

Matt, can you give us more info on what are you trying to do, and which hardware are you trying to emulate?
Comment 8 Matt Domsch 2009-03-13 11:17:43 EDT
I'll settle for whatever hardware I can get, as a starting point. :-)  rtl8139 and virtio for starters, we can worry about other NICs later.

The use case is relatively simple - "Boot using iSCSI in Guest".
I want virtual machines to be able to boot directly off an iSCSI LUN, using an iSCSI initiator present in the Virtual Machine BIOS somehow.  This exactly mimics booting a physical system, using an iSCSI boot-capable NIC.

Last summer I worked with a Dell intern to document how to boot such a system, using gPXE option ROMs loaded into qemu, as follows, using rtl8139:

qemu-kvm  -m 512 –boot n –net nic,macaddr=00:16:3e:17:82:ae,vlan=0 –net tap,script=no,vlan=0,ifname=vnet0 –option-rom pci_10ec_8139.rom

In this particular setup, the gPXE option ROM iSCSI initiator gets its target information from DHCP.  At some point it may be beneficial to pass this information via another method (qemu options -> BIOS iBFT -> initiator).

Once the boot loader and initrd, which are read via iSCSI, finish, the initrd carries an iSCSI initator too, reconnects, mounts the root file system, and it's off to the races.


The alternative architecture being driven pretty hard through the industry right now have the iSCSI connections terminate in the hypervisor, where the hypervisor can provide either a software or hardware-accelerated iSCSI initator.  In this world, guests won't see iSCSI at all.  But it breaks the "end-to-end" principle, and adds a whole lot more intelligence needed in the hypervisor, instead of focusing on making the hypervisor really really fast for network I/O.
Comment 9 Shyam kumar Iyer 2009-03-13 12:58:39 EDT
My $0.02 -

Unless we are dealing with VT-d aren't we concerned with just a small set of emulated drivers? rtl9139 being the prominent.

And I guess in the case of VT-d we won't use gpxe option roms.
Comment 10 Mark McLoughlin 2009-03-25 08:09:52 EDT
Adding to F11VirtTarget - probably a stretch, though
Comment 11 Glauber Costa 2009-03-25 10:09:28 EDT
The hard part of it from F11 POV is to get a gpxe package.

However, if the users are willing to provide their own roms, I believe it can be reasonably targeted for F11. The qemu part is not that difficult, I have already sent part of the patches upstream, discussed the other part with anthony, and although not yet accepted, didn't seem to find many resistance.
Comment 12 Matt Domsch 2009-03-25 15:00:05 EDT
Per #11, a gpxe package is up for review.
https://bugzilla.redhat.com/show_bug.cgi?id=492181
Comment 13 Mark McLoughlin 2009-03-27 14:16:29 EDT
glommer sent patch upstream:

  http://lists.gnu.org/archive/html/qemu-devel/2009-03/msg01295.html
Comment 14 Mark McLoughlin 2009-04-03 08:37:43 EDT
Okay, applied in rawhide:

* Thu Apr 02 2009 Glauber Costa <glommer@redhat.com> - 2:0.10-4
- Support botting gpxe roms.

http://cvs.fedoraproject.org/viewvc/rpms/qemu/devel/qemu-roms-more-room.patch?view=markup