Description of problem: When installing ia64 rhel5.5 hvm guests, the guest can't boot from the boot.iso. In EFI shell screenshot is attached. Trying out the same for 5.4 guest works just fine. Version-Release number of selected component (if applicable): RHEL5.5-Server-20091227.0 tree. How reproducible: Very. Steps to Reproduce: 1. Copy RHEL5.5-Server-20091227.0 tree's boot.iso to local machine and try to install the guest with --cdrom argument using it. 2. 3. Actual results: Expected results: Additional info:
My guess is this isn't specific to virt-install. To be sure, can you post ~/.virtinst/virt-install.log from a 5.4 successful install, and the 5.5 error? Reassigning to xen in the interim.
Created attachment 386686 [details] EFI shell in vnc screenshot.
Problem is not in host xen kernel/userpsace, but in boot.iso file itself. I was able to boot RHEL5.4 from boot.iso without problems. It looks like bootloader is not able to read from boot.iso. I tried latest nightly build (100126) and there is elilo not working at all (elilo = unknown command). Testing initrd.img/vmlinuz from boot.iso on bare-metal shows, these are correct and working ones. Someone responsible for creating boot.iso should check this BZ (unfortunately I do not know who is this)
(In reply to comment #8) > Problem is not in host xen kernel/userpsace, but in boot.iso file itself. I was > able to boot RHEL5.4 from boot.iso without problems. It looks like bootloader > is not able to read from boot.iso. > > I tried latest nightly build (100126) and there is elilo not working at all > (elilo = unknown command). > > Testing initrd.img/vmlinuz from boot.iso on bare-metal shows, these are correct > and working ones. > > Someone responsible for creating boot.iso should check this BZ (unfortunately I > do not know who is this) Well, I don't know whether it's relevant to error I've run into when doing some testing on ia64 but my solution could work as well. The problem may be in boot order. Try exiting EFI shell using `exit` command. Then try to check boot options in EFI Boot Manager and reset the system. I don't know whether it will be working with that since I've been able to install some guest but I was unable to boot it... It took me pretty long time to get to know what the problem is but finally solution described above worked fine. Michal
@Michal: No this doesn't help in this case @Gurham: I try this with 5.4 kernel-xen and xen packages. It has the same result: - boot.iso from 5.4 worked correctly, boot.iso from 5.5 do not worked. This means that's no regression in both kernel-xen and xen packages and problem is related to boot.iso file.
Alex, is this issue still reproducable in latest composes?
Daniel, it will take me a while to figure it out. There aren't many HVM enabled ia64's in RHTS to use for manual testing. I've reserved a system and will test whenever it's available.
We have not changed the way ia64 boot.iso images are created. The method used in 5.4 is the same as the one used in 5.5, so unless one of the dependent components changed, I don't know what's happening. mkisofs looks unchanged (it was last changed in October 2008) elilo had one patch applied in December 2009, but it seems unrelated This is the elilo.conf file from boot.img, which is on boot.iso: prompt timeout=50 relocatable image=vmlinuz label=linux read-only initrd=initrd.img The first line ("prompt") is followed by a tab character before the new line. The last line is indented with a single tab character rather than 8 spaces like the label and read-only lines use. Still, this is exactly how it appears in RHEL 5.4. This is either an elilo bug or a bug in the firmware. I would think that we need to change elilo.conf, but since RHEL 5.4 works on the same setup, I don't think this has to do with elilo.conf.
(In reply to comment #14) Kind of shooting in the dark here... could it be because of the way boot.iso was composed by rel-eng?
(In reply to comment #17) > (In reply to comment #14) > > Kind of shooting in the dark here... could it be because of the way boot.iso > was composed by rel-eng? I'm sure I did not change anything in boot.iso creation process. Let's wait if Alex is able to reproduce this bug on latest nightlies.
Testing Nightly builds and Rel-eng builds: Nightly: 20091022 - OK 20091029 - OK 20091122 - OK 20091129 - OK 20091221 - OK 20091222 - OK 20091223 - kernel file not found 20091224 - OK 20091229 - OK 20100104 - OK 20100105 - OK 20100106 - OK 20100107 - OK 20100108 - elilo not found 20100110 - elilo not found 20100112 - kernel file not found 20100113 - kernel file not found 20100114 - kernel file not found 20100115 - elilo not found 20100117 - kernel file not found 20100118 - kernel file not found 20100119 - kernel file not found 20100121 - kernel file not found 20100122 - elilo not found 20100124 - kernel file not found 20100126 - elilo not found 20100127 - kernel file not found 20100128 - kernel file not found Rel-eng: 20091227 - kernel file not found 20100117 - kernel file not found elilo not found error output looks: ----------------------------------- 'elilo' not found Exit status code: Invalid Parameter ----------------------------------- kernel file not found looks: ---------------------------- near line 1: No image defined! near line 1: No image defines! forcing interactive mode due to config file error(s) ELILO elilo.c(line 71):Kernel file not found linux Exit status code: Load Error ---------------------------- "OK" and "kernel file not found" print outs at the start of elilo (this is printed when 5.4 boot.iso is used too): ------------------------------------------------------------------ fileops.c(line 520):No devname schemes worked, using builtin -----------------------------------------------------------------
Testing Daniels images: In both images is too much devices (fs0, fs1, blk0-4). fs0 contains EFI dir then redhat and EFI Firmware dir. fs1 looks like standart fs0 /mnt/redhat/devel/dmach/trees/RHEL5.5-Server-20100128.1/: - fs1: started successfully - fs0:/EFI/redhat: started successfully /mnt/redhat/devel/dmach/trees/RHEL5.5-Server-20100128.2/: - fs1: kernel not found - fs0:/EFI/redhat: started successfully
This also seems to point at anaconda (.2 fails, so downgrading elilo did not help; .1 passes, so downgrading anaconda fixed it).
QE Tested the composes with same results.
I can't access the hp-bl860c-03.rhts.bos.redhat.com machine, what's 'standard password'?
I processed the old version of anaconda (.1 compose) throught the whole installation process and it passed.
Hi Ales, unfortunately machine reservation was not extended enough to survive till Monday. Machine was reserved by Gurhan. Get ia64 with HVM support is hard as there is not much machine in rhts. Gurhan should help you with this.
Thanks Miroslav and Gurhan. Here's what I've found: the boot.iso images (containing boot.img images) that are failing (according to Comment 19) are those with size exceeding 33 MBytes. If we assume that the problem is in loading images bigger than certain size, everything else adds up: * why it came and went away on 20091224 * why we have not been able to find any anaconda change that could be even remotely suspicious * why we see the problem so early during boot, before kernel or anything can take over * why the images look allright otherwise * why reverting anaconda helped but reverting elilo not --- because anaconda is much bigger / the old version generated smaller image. * and especially, why we are seeing two types of errors (kernel vs. elilo): this could be because random parts of the image are missing each time It's easy to verify this: take any failing boot.iso, extract the initrd out of it, remove sbin/loader (that comes into play much later than the point we care for) build a new iso with the new imagerd and try to boot it. If kernel is found and starts booting we know we're home. Release Test Team, can I ask you to test this? Ask akozumpl over irc for details anytime. Thanks. Ales
Thanks for looking at it Ales. So, this points at either elilo or the guest firmware, right?
Right, if the theory is right.
(In reply to comment #38) > Release Test Team, can I ask you to test this? Ask akozumpl over irc for > details anytime. > Ales, yes RTT will test. I'm the ia64 arch owner from Release Test Team. Testing boot.iso on Xen is part of our tier #2 test cases and during beta and snapshots phase I'll execute this case at least once. The only thing is that I've been having big troubles getting access to HVM enabled ia64 hardware. No luck up to now.
SGI has reported seeing this same problematic behavior on bare metal installations on altix6 and other Altixes, as well as an hp rx2600.
This is present in the 0201.0 build. To reproduce simply start a FV domU from the boot.iso file.
I've tested booting from disc1 ISO images for RHEL5.5 and RHEL5.4. 5.4 boots fine, while 5.5 doesn't. The disc1 ISO is bigger than 600MB so I don't think comment #38 is valid in this case.
(In reply to comment #45) > I've tested booting from disc1 ISO images for RHEL5.5 and RHEL5.4. 5.4 boots > fine, while 5.5 doesn't. The disc1 ISO is bigger than 600MB so I don't think > comment #38 is valid in this case. It is still valid. It's all about boot.img (not boot.iso).
Agreed with Dan. The 5.4 iso's boot.img has 32 megabytes, so it is not over the limit and correctly boots. I also tried taking the failing image from 0201 (comment 44), unpacking it, removing intird.img completely, building a new boot.img (now way below 33 megas) and a boot.iso with it. This finds the kernel and starts booting fine. (then hangs on the part where ramdisk is supposed to be mounted of course). This is not a problem in buildinstall (or anaconda), there is something else going wrong when the machine tries to unpack the image. This should be assigned to someone who knows about firmware of the architecture or has some other ia64 expertise.
Adding Luming, Doug and Bill to this BZ in hopes that we can resolve this beta blocker ASAP.
Offhand, I don't have a clue what could be going wrong here... But anyone in need of an ia64 hvm-capable machine to test on is welcomed to hop onto mine, hp-rx2660-01.lab.bos.redhat.com.
It looks like the question has been answered, the boot.img is simply too big. I have seen this on physical hardware many times and in fact toward the end of the Fedora-ia64 life we even had to hack the initrd.img to remove unneeded stuff just to make the images bootable. Since in this case it appears the iso boots just fine on real hardware but fails under KVM it sounds like the virtual guest firmware has a lower limit on this. So the only fixes for this are to either remove some stuff (i.e. unneeded kernel modules) from initrd.img or get Intel to fix the firmware to increase this limit.
Adding ddugger, who might have inputs to guest firmware.
Hi Doug, > Since in this case it appears the iso boots just fine on real hardware but > fails under KVM it sounds like the virtual guest firmware has a lower limit on To be more specific we really know nothing about what happens when the boot.iso over 33 mb boots on real hardware. What Alexander talks about in comment 45 is rhel 5.4 CD ISO that contains another boot.iso in it that is BELOW 33 mb. So either that can be why it boots or it really does not matter for real hardware. Ales
Not by me.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Xen virtual machine under ia64 will not boot if the size of boot.img inside boot.iso is over 33 megabytes. This is most probably due to a faulty firmware on ia64.
(dummy comment to clear the needinfo)
(In reply to comment #50) > It looks like the question has been answered, the boot.img is simply too big. > I have seen this on physical hardware many times and in fact toward the end of > the Fedora-ia64 life we even had to hack the initrd.img to remove unneeded > stuff just to make the images bootable. > > Since in this case it appears the iso boots just fine on real hardware but > fails under KVM it sounds like the virtual guest firmware has a lower limit on > this. So the only fixes for this are to either remove some stuff (i.e. > unneeded kernel modules) from initrd.img or get Intel to fix the firmware to > increase this limit. Doug's right. makebootdisk truncates files when the boot.img gets too large. I also remember this happening on fedora-ia64. anaconda is going to have to do some magic to fix this issue. This is *not* a kernel issue. The kernel is doing the "right" thing by attempting to load the boot.img _which is broken_. This results in a boot failure. Doug (dchapman), IIRC we just dumped some non-ia64 related stuff from the boot.img upstream, correct? I don't think we can do that here. The anaconda team needs to handle this. P.
Oops. "Doug's right. makebootdisk truncates files when the boot.img gets too large. I also remember this happening on fedora-ia64." should have read "Doug's right. The FIRMWARE truncates files when the boot.img ...." Sorry for the confusion, P.
> Doug (dchapman), IIRC we just dumped some non-ia64 related stuff from the > boot.img upstream, correct? I don't think we can do that here. > > The anaconda team needs to handle this. Yes, that was the fix for Fedora. As a temporary fix toward the end we even had a script to hack the initrd.img and strip out stuff after everything was built (ugly but effective). There is a ton of stuff in initrd.img that will never get used during installation.
BTW, the image size could be easily shaved off by nearly 50% (which could suffice for a couple more RHEL5 releases at lesat) if we removed/linked the whole efi/boot to point to the files in the image's root. Those are in fact the same: $ md5sum vmlinuz efi/boot/vmlinuz 502dc3ce7793ccb101aeed1871d7207e vmlinuz 502dc3ce7793ccb101aeed1871d7207e efi/boot/vmlinuz $ md5sum initrd.img efi/boot/initrd.img 5dddd59c8ec1c0774305b6f320151134 initrd.img 5dddd59c8ec1c0774305b6f320151134 efi/boot/initrd.img Those files are referenced from elilo.conf, can someone WHO REALLY KNOWS INTERNALS OF THE IA64 BOOT PROCESS confirm that changing the efi/boot/elilo.conf file as the patch shows would work and boot correctly?
Created attachment 389707 [details] proposed fix in the elilo.conf
(In reply to comment #63) > BTW, the image size could be easily shaved off by nearly 50% (which could > suffice for a couple more RHEL5 releases at lesat) if we removed/linked the > whole efi/boot to point to the files in the image's root. Those are in fact the > same: > > $ md5sum vmlinuz efi/boot/vmlinuz > 502dc3ce7793ccb101aeed1871d7207e vmlinuz > 502dc3ce7793ccb101aeed1871d7207e efi/boot/vmlinuz > > $ md5sum initrd.img efi/boot/initrd.img > 5dddd59c8ec1c0774305b6f320151134 initrd.img > 5dddd59c8ec1c0774305b6f320151134 efi/boot/initrd.img > > Those files are referenced from elilo.conf, can someone WHO REALLY KNOWS > INTERNALS OF THE IA64 BOOT PROCESS confirm that changing the > efi/boot/elilo.conf file as the patch shows would work and boot correctly? This would make the .iso smaller but not the initrd.img. IIRC it is the initrd.img size that causes the issue. The way we fixed this in Fedora was remove some of the drivers from the kernel config for the install kernel.
(In reply to comment #65) > This would make the .iso smaller but not the initrd.img. IIRC it is the > initrd.img size that causes the issue. The way we fixed this in Fedora was > remove some of the drivers from the kernel config for the install kernel. The problem is that initrd.img is *twice* on boot.img. Removing one of them would reduce boot.img to approx. 16MB. The boot.iso size doesn't matter.
Yes, like Daniel says the issue is the size of boot.img. Why: because the machine complains about missing kernel which comes into play before we even touch initrd.img.
Can I get the test machine again please?
(In reply to comment #66) > (In reply to comment #65) > > This would make the .iso smaller but not the initrd.img. IIRC it is the > > initrd.img size that causes the issue. The way we fixed this in Fedora was > > remove some of the drivers from the kernel config for the install kernel. > > The problem is that initrd.img is *twice* on boot.img. > Removing one of them would reduce boot.img to approx. 16MB. > The boot.iso size doesn't matter. Good point, removing one copy of the initrd.img is probably the right thing to do. I didn't realize it was in there twice. However, iirc the core of the issue is more related to having a single file over a certain size. If that is the case the only fix is to reduce the size of initrd.img. IMHO this is not a bug in EFI, simply a limitation (which all other OS's including RHEL up to this point have had no issue with). The real problem is the bloated boot images which seem to continue to grow over time.
Doug, I wrote to anaconda list about my plan to remove clone of the initrd.img plus vmlinuz that reside in root: https://www.redhat.com/archives/anaconda-devel-list/2010-February/msg00106.html but Jeremy told me sometimes the first image "is/was" necessary: https://www.redhat.com/archives/anaconda-devel-list/2010-February/msg00114.html Is that still really the case? Or else can we split the image in two? Thank you, Ales
Doug, can you give us a list of modules that are in the installation initrd and are not being used. We can ask anaconda/rcm team to adjust their build scripts and remove those modules.
(In reply to comment #74) > Doug, > > I wrote to anaconda list about my plan to remove clone of the initrd.img plus > vmlinuz that reside in root: > https://www.redhat.com/archives/anaconda-devel-list/2010-February/msg00106.html > > but Jeremy told me sometimes the first image "is/was" necessary: > https://www.redhat.com/archives/anaconda-devel-list/2010-February/msg00114.html > > Is that still really the case? Or else can we split the image in two? > > Thank you, > Ales As I mentioned earlier I don't think this is the issue. What needs to be done is a little experimentation. Manually hack up a new boot.img with only one and do some testing to see if this even works.
(In reply to comment #75) > Doug, > can you give us a list of modules that are in the installation initrd and are > not being used. We can ask anaconda/rcm team to adjust their build scripts and > remove those modules. I will take a look and come up with some recommendations later today.
I'd like to ask our IA64 partners to test whether the boot.iso below correctly starts booting any IA64 machine: http://akozumpl.fedorapeople.org/bz556976/boot.iso for instance: virt-install --name ia64_hvm_guest --nonsparse --hvm --file /var/lib/xen/images/ia64_hvm_guest.img --cdrom /var/lib/xen/images/boot.iso -s 10 --debug --prompt --noreboot --vnc --ram 1024 Thank you, Ales
Further explanation to the comment above: the iso contains a standard boot.img but with the redundant kernel and ramdisk images removed. Some say this should work, others say it won't on certain EFIs, so let's test this. QE can you also please test this on the hardware you have available?
Just a reminder that Xen can not be used on Altix systems -- so Anaconda should not depend on it. George
(In reply to comment #78) > I'd like to ask our IA64 partners to test whether the boot.iso below correctly > starts booting any IA64 machine: > > http://akozumpl.fedorapeople.org/bz556976/boot.iso > > for instance: > virt-install --name ia64_hvm_guest --nonsparse --hvm --file > /var/lib/xen/images/ia64_hvm_guest.img --cdrom /var/lib/xen/images/boot.iso -s > 10 --debug --prompt --noreboot --vnc --ram 1024 > > Thank you, > Ales The xen component of this honestly is only on particular place where this fails. We are seeing this fail on some physical hardware which IMHO is a much bigger issue. Also, we really need to test this on every ia64 implementation to see if removing those files does cause issues with certain versions of EFI. However, I tested the boot.iso on a physical system that failed with the original boot.is and this does indeed fix the issue there. So, we are on the right track it appears.
I committed a patch that generates images from comment 78 during a proper rhel5 compose (c8a5d546044e9b24eb22023dc49f053a94d76d38). Future nightly images with anaconda-11.1.2.202-5 should contain the small version of boot.iso.
We have reproduced a similar issue on a physical machine. it is a HP integrity rx2660 server. We downloaded the RHEL 5.5 Beta Binary DVD image from : https://rhn.redhat.com/rhn/software/channel/downloads/Download.do?cid=6012 I have attached a screenshot for reference. When can we expect a fix for this issue? Thanks, Balaji
Created attachment 394536 [details] screenshot of installation issue of ia64 on a physical machine
Hi Sribalaji, we know this issue is present in 5.5 Beta, even on physical machines. You can try booting with the iso from comment 78 and see whether that helps you get past the kernel load. Thank you. Ales Kozumplik
*** Bug 565874 has been marked as a duplicate of this bug. ***
Using Ales boot.iso I successfully installed 5.5 nightly (via NFS from bigpapi) on altix6. George
boot.iso from snap #3 boots as fully virtualized xen guest. I was able to complete a default install. Testing with DVD ISO now.
booting from DVD also works. Moving to VERIFIED.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0194.html
Hi, RedHat, When I opened the link above, it seems that it doesn't exist.