RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 729694 - bootindex added after install completes. causes boot failure in KVM with mixed virtio/ide disks
Summary: bootindex added after install completes. causes boot failure in KVM with mixe...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.2
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 691780
TreeView+ depends on / blocked
 
Reported: 2011-08-10 14:55 UTC by Alexander Todorov
Modified: 2012-06-20 06:29 UTC (History)
19 users (show)

Fixed In Version: libvirt-0.9.10-1.el6
Doc Type: Bug Fix
Doc Text:
With certain combination of IDE and VirtIO disks a guest OS would not boot after installation. This happened because the order of disks in which they were presented to the guest during installation was different from the order used after installation. As a result of this, the system could have been installed on a disk which is not used as the primary bootable disk. Libvirt was fixed to always make sure the order of disks presented to a guest OS during installation is the correct order which will be used later once the guest OS is installed.
Clone Of: 621175
Environment:
Last Closed: 2012-06-20 06:29:47 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:0748 0 normal SHIPPED_LIVE Low: libvirt security, bug fix, and enhancement update 2012-06-19 19:31:38 UTC

Description Alexander Todorov 2011-08-10 14:55:56 UTC 
+++ This bug was initially created as a clone of Bug #621175 +++

--- Additional comment from gleb on 2011-08-10 17:47:35 EEST ---

> -virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
> +virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1

This is where the bug is! During installation virtio disk has no bootindex parameter so BIOS choose first ide as bootable. After install virt-manager tells bios to boot from virtio disk and this obviously fails. Please open the bug against virt-manager (or may be libvirt if virt-manager uses it to start guests).
Comment 1 Alexander Todorov 2011-08-10 15:01:02 UTC 
The diff above is from command line used to start the guest install and after install is complete and guest is rebooted. 

To reproduce: 

1) Use latest RHEL 6.2 host and guest system. 
2) Start a KVM guest with one virtio disk and 2 IDE disks. 
3) Perform GUI install selecting all default options in anaconda
4) Reboot. 

According to Ales in:
https://bugzilla.redhat.com/show_bug.cgi?id=621175#c70

Anaconda is doing its job properly and correctly finds the boot device. After install completes the guest is rebooted and is told to boot off /dev/vda instead of /dev/sda which causes the failure.

Versions:
virt-manager-0.9.0-5.el6.x86_64
libvirt-0.9.4-1.el6.x86_64
Comment 2 Cole Robinson 2011-08-29 17:55:43 UTC 
Okay, to summarize my understanding of the problem:

Installing a guest via URL in virt-manager, libvirt doesn't manually set a disk boot order on the command line. anaconda uses info from qemu to determine that the first IDE device should be the install dev (no idea what factors into this decision, but seems to be what bug 621175 is all about).

After install, virt-manager changes the guest boot configuration to use <boot dev='hd'/>. Libvirt manually marks the first disk in the XML as the boot drive. Since this _wasn't_ the same disk anaconda chose, guest doesn't boot.

Maybe libvirt should manually mark the first disk as the boot device even if booting from kernel/initrd? Reassigning to libvirt a CCing relevant people
Comment 3 Gleb Natapov 2011-08-31 11:02:16 UTC 
(In reply to comment #2)
> Okay, to summarize my understanding of the problem:
The summary is correct.

> 
> Installing a guest via URL in virt-manager, libvirt doesn't manually set a disk
> boot order on the command line. anaconda uses info from qemu to determine that
> the first IDE device should be the install dev (no idea what factors into this
> decision, but seems to be what bug 621175 is all about).
> 
> After install, virt-manager changes the guest boot configuration to use <boot
> dev='hd'/>. Libvirt manually marks the first disk in the XML as the boot drive.
> Since this _wasn't_ the same disk anaconda chose, guest doesn't boot.
> 
> Maybe libvirt should manually mark the first disk as the boot device even if
> booting from kernel/initrd? Reassigning to libvirt a CCing relevant people
Comment 4 Laine Stump 2011-10-05 04:03:18 UTC 
In order for the OS anaconda installed to boot, libvirt needs to use the same algorithm anaconda uses to determine the first boot device.

What is that algorithm? Is it really as simple as "the first IDE drive"? Is that really the proper thing to do in all occasions?
Comment 5 Gleb Natapov 2011-10-05 08:58:36 UTC 
(In reply to comment #4)
> In order for the OS anaconda installed to boot, libvirt needs to use the same
> algorithm anaconda uses to determine the first boot device.
> 
> What is that algorithm? Is it really as simple as "the first IDE drive"? Is
> that really the proper thing to do in all occasions?

Bios tells anaconda what is the HD it will boot from. It will be HD with the lowest bootindex among all HD on qemu command line.
Comment 6 Laine Stump 2011-10-05 14:02:55 UTC 
Ah, so the problem is that when virt-manager starts the guest the first time, it has the XML setup to boot from CD, so libvirt doesn't bother putting in bootindex anywhere. But then virt-manager changes to <boot dev='hd'/> for the second boot, causing libvirt to insert bootindex=1 somewhere, and it just gets it wrong. So is it a better solution to put bootindex=1 in the right place, or not put it anywhere and rely on default behavior of qemu?
Comment 7 Gleb Natapov 2011-10-05 14:10:29 UTC 
(In reply to comment #6)
> Ah, so the problem is that when virt-manager starts the guest the first time,
> it has the XML setup to boot from CD, so libvirt doesn't bother putting in
> bootindex anywhere. But then virt-manager changes to <boot dev='hd'/> for the
> second boot, causing libvirt to insert bootindex=1 somewhere, and it just gets
> it wrong. So is it a better solution to put bootindex=1 in the right place, or
> not put it anywhere and rely on default behavior of qemu?

bootindex should always be specified. The HD that libvirt consider to be bootable should have lowest bootindex between all HDs. I would advice to always append bootindex=10 to bootable disk. If VM wants to boot from a CD once add bootindex=1 to teh CD just for that one boot.
Comment 8 Jiri Denemark 2011-10-06 08:39:34 UTC 
Libvirt already includes bootindex for disk device if the xml mentions <boot dev='hd'/>. So we need to make sure that both <boot dev='cdrom'> and <boot dev='hd'/> are included during the installation phase. Another issue is that according to https://bugzilla.redhat.com/show_bug.cgi?id=621175#c69, the installation wasn't done by booting from a cd but rather by directly booting specified kernel with initrd in which case we don't emit any bootindex. I guess we should fix this and provide bootindex even if direct kernel boot is used.
Comment 11 yuping zhang 2011-11-29 09:16:13 UTC 
Reproduce this issue with:
libvirt-client-0.9.4-23.el6.x86_64
libvirt-0.9.4-23.el6.x86_64
libvirt-python-0.9.4-23.el6.x86_64
qemu-kvm-0.12.1.2-2.209.el6.x86_64

Steps:
1.Create 3 disks:
#qemu-img create /var/lib/libvirt/images/test-729694 4G
#qemu-img create /var/lib/libvirt/images/foo.img 4G
#qemu-img create /var/lib/libvirt/images/foo1.img 4G

2.Use virt-manager to create a new guest with HTTP.

3.Select the disk test-729694 during New VM Guilde.

4.Select "Customize configuration before install" at the last step.Then click
"Finish".

5.Change the disk bus of "test-729694" to Virtio.

6.Add the two disks foo.img and foo1.img as ide disk.Apply all change.Then
click "Begin Installation".

7.During installation:
 Click through to disk selection dialog. Select all disks to "Install Target
Device",and in "Install Target Tevices" window. Click "ATA QEMU HARDDISK" as
Boot Loader.

8.Install the guest with default option.

9.After reboot guest.
The system is not able to boot. The console was showing
the message:

Booting from Hard Disk...


During installation:
#ps -ef | grep kvm
...
-drive
file=/var/lib/libvirt/images/test-729694,if=none,id=drive-virtio-disk0,format=raw,cache=none
-device
virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
...
After reboot:
...
-drive
file=/var/lib/libvirt/images/test-729694,if=none,id=drive-virtio-disk0,format=raw,cache=none
-device
virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
...
Comment 12 Jiri Denemark 2012-01-25 09:41:39 UTC 
This issue should now be fixed upstream by v0.9.9-102-g24a0014:

commit 24a001493adefe27f223987ad380425a2338b0b9
Author: Jiri Denemark <jdenemar>
Date:   Mon Jan 23 16:35:25 2012 +0100

    qemu: Emit bootindex even for direct boot
    
    Direct boot (using kernel, initrd, and command line) is used by
    virt-install/virt-manager for network install. While any bootindex has
    no direct effect since -kernel is always first, we need it as a hint for
    SeaBIOS to present disks in the same order as they will be presented
    during normal boot.
Comment 15 yanbing du 2012-02-16 08:01:01 UTC 
Test this bug with libvirt-0.9.10-1.el6.x86_64, and it still not fix.
If fact, use the newest libvirt package, during installation, there's a bootindex=1 parameter appeared in the qemu command line, but it always point to the first disk(ide or virtio), no matter the boot loader installed on which disk.
So whether the guest can boot successfully just depends on if the boot loader installed on the first disk.
Test scenarios 1:
1. Choose http install method to install a guest via virt-manager, and the guest has 1 virtio disk and 2 IDE disks. The disk add order is virtio - IDE - IDE, after Begin Installation, the order in guest XML file do not change.
2. During installation, choose the second disk(IDE disk 1) to install Boot Loader.
3. Finished installation and reboot the guest. The system is not able to boot. The console was showing the message: "Booting from Hard Disk..."
4 During installation: 
# ps -ef |grep qemu-kvm
...
-drive file=/var/lib/libvirt/images/test-729694.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-drive file=/var/lib/libvirt/images/test-1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 
-drive file=/var/lib/libvirt/images/test-2.img,if=none,id=drive-ide0-0-1,format=raw,cache=none -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 
...
After reboot:
...
-drive file=/var/lib/libvirt/images/test-729694.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-drive file=/var/lib/libvirt/images/test-1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 
-drive file=/var/lib/libvirt/images/test-2.img,if=none,id=drive-ide0-0-1,format=raw,cache=none -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
...
There's no difference.

Test scenarios 2:
1. Choose http install method to install a guest via virt-manager, and the guest has 1 virtio disk and 2 IDE disks. The disk add order is IDE - virtio - IDE, after Begin Installation, the order in guest XML file change to IDE - IDE - virtio automatically.
2. During installation, choose the third disk(virtio disk) to install Boot Loader.
3. Finished installation and reboot the guest. The system is not able to boot. The console was showing the message: "Booting from Hard Disk..."
4 During installation:
# ps -ef |grep qemu-kvm
...

-drive file=/var/lib/libvirt/images/sss.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
-drive file=/var/lib/libvirt/images/sss-2.img,if=none,id=drive-ide0-0-1,format=raw,cache=none -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 
-drive file=/var/lib/libvirt/images/sss-1.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
...
After reboot:
...
-drive file=/var/lib/libvirt/images/sss.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
-drive file=/var/lib/libvirt/images/sss-2.img,if=none,id=drive-ide0-0-1,format=raw,cache=none -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 
-drive file=/var/lib/libvirt/images/sss-1.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
...
There's no difference.

BTW, there's another question, it seems that libvirt will reorder the disk order  after begin installation:
If i add disk following the order: IDE - virtio - IDE, it will reorder to: IDE - IDE - virtio.
If i add disk following the order: virtio - IDE - virtio, it will reorder to: virtio - virtio - IDE.
Is this a deliberately design?
Comment 16 Gleb Natapov 2012-02-16 08:14:22 UTC 
(In reply to comment #15)
> Test this bug with libvirt-0.9.10-1.el6.x86_64, and it still not fix.
> If fact, use the newest libvirt package, during installation, there's a
> bootindex=1 parameter appeared in the qemu command line, but it always point to
> the first disk(ide or virtio), no matter the boot loader installed on which
> disk.
> So whether the guest can boot successfully just depends on if the boot loader
> installed on the first disk.
> Test scenarios 1:
> 1. Choose http install method to install a guest via virt-manager, and the
> guest has 1 virtio disk and 2 IDE disks. The disk add order is virtio - IDE -
> IDE, after Begin Installation, the order in guest XML file do not change.
> 2. During installation, choose the second disk(IDE disk 1) to install Boot
> Loader.
Don't do that!

Anaconda should chose where to install boot loader correctly for you. If you override anaconda's choice you can't expect it to work. So to verify the bug you need to click through the anaconda boot loader install menu.
Comment 17 yanbing du 2012-02-16 09:18:27 UTC 
(In reply to comment #16)
> (In reply to comment #15)
> > Test this bug with libvirt-0.9.10-1.el6.x86_64, and it still not fix.
> > If fact, use the newest libvirt package, during installation, there's a
> > bootindex=1 parameter appeared in the qemu command line, but it always point to
> > the first disk(ide or virtio), no matter the boot loader installed on which
> > disk.
> > So whether the guest can boot successfully just depends on if the boot loader
> > installed on the first disk.
> > Test scenarios 1:
> > 1. Choose http install method to install a guest via virt-manager, and the
> > guest has 1 virtio disk and 2 IDE disks. The disk add order is virtio - IDE -
> > IDE, after Begin Installation, the order in guest XML file do not change.
> > 2. During installation, choose the second disk(IDE disk 1) to install Boot
> > Loader.
> Don't do that!
> 
> Anaconda should chose where to install boot loader correctly for you. If you
> override anaconda's choice you can't expect it to work. So to verify the bug
> you need to click through the anaconda boot loader install menu.

Ah, as you said, if use anaconda do it automatically, the guest can reboot successfully, so this can be verified.
Comment 18 Jiri Denemark 2012-05-10 13:51:16 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
With certain combination of IDE and VirtIO disks a guest OS would not boot
after installation. This happened because the order of disks in which they
were presented to the guest during installation was different from the order
used after installation. As a result of this, the system could have been
installed on a disk which is not used as the primary bootable disk. Libvirt
was fixed to always make sure the order of disks presented to a guest OS
during installation is the correct order which will be used later once the
guest OS is installed.
Comment 20 errata-xmlrpc 2012-06-20 06:29:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0748.html
Note You need to log in before you can comment on or make changes to this bug.