Description of problem:
A guest has 2 virtio disks: vda and vdb. /boot resides on /dev/vdb1. Grub is installed on /dev/vda, which is also where the guest is configured to boot from.
Firstly, a fresh installation in the above configuration will cause the boot to fail with 'Hard Disk Error' (screenshot rhel6-grub-hd_error).
If you attempt to fix the problem by manually running 'grub-install /dev/vda' from the rescue environment you get Error 21 trying to load stage 1.5 (screenshot rhel6-error_21).
The above configuration works fine if the disks are IDE. It also works fine if the drives are swapped so that /dev/vdb1 becomes /dev/vda1 and grub is manually installed on the new /dev/vda. The problem seems to be specific to attempting to read /boot from a drive other than the first one.
I have replicated this problem on RHEL 4.8, RHEL 5.4 and RHEL 6.0, all of which support VirtIO. I have also replicated the problem on a RHEL 5 KVM host, which uses Bochs BIOS, and an F14 host, which uses SeaBIOS. The behaviour is the same in all cases, although the error messages from RHEL 4 and RHEL 5 grub are slightly more verbose.
 I'd expect the behaviour to be identical immediately after installation to behaviour after running 'grub-install /dev/vda', so could this indicate a secondary problem in anaconda?
For some context, I originally hit this testing V2V conversion of a multiboot guest OS, where the first disk contained Windows XP and the second disk contained RHEL. Grub is configured to boot either OS. The conversion process enables virtio in both Windows and RHEL. After conversion, grub fails with error 21 and neither guest can be booted.
However, if you swap the 2 disks around so that RHEL, and therefore /boot, is on vda, and manually install grub to the new vda, grub can boot RHEL successfully. However, it fails trying to chainload XP, which is now on the second disk. grub.conf is obviously edited appropriate during the switchover too, so this isn't caused by it attempting to chainload the wrong partition.
Created attachment 517231 [details]
RHEL 6 guest failing to boot with 'Error 21'
Created attachment 517232 [details]
RHEL 6 failing to boot with 'Hard Disk Error'
Please attach anaconda logs from installation and instructions on how to reproduce the problem.