Bug 439379

Summary: Broken bootloader config for Xen after system upgrade
Product: Red Hat Enterprise Linux 5 Reporter: Frank Arnold <frank.arnold>
Component: anacondaAssignee: Chris Lumens <clumens>
Status: CLOSED ERRATA QA Contact: James Laska <jlaska>
Severity: low Docs Contact:
Priority: low    
Version: 5.2CC: bnagendr, jlaska, jturner, osrc-sysint
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: RHBA-2008-0397 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-05-21 15:33:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/root/upgrade.log
none
anaconda-logs.tgz
none
anaconda-upgrade-new-bootloader.tar.gz none

Description Frank Arnold 2008-03-28 12:44:36 UTC
Description of problem:
I did upgrades from 5.2 Beta to Snapshot 1 and from Snapshot 1 to Snapshot 2
using Anaconda's upgrade functionality. In both cases the Xen bootloader
configuration was broken after the upgrade was finished.
I'm not sure if this is Anaconda's fault. Please reassign if this is not the
appropriate component.


How reproducible:
Every time.


This is how it looks like after upgrading:

title Red Hat Enterprise Linux Server (2.6.18-86.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-86.el5 ro root=LABEL=/1 rhgb quiet
        initrd /initrd-2.6.18-86.el5.img
title Red Hat Enterprise Linux Server (2.6.18-86.el5xen)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-86.el5xen ro root=LABEL=/1 rhgb quiet
        initrd /initrd-2.6.18-86.el5xen.img

But I'd expect it to look like this:

title Red Hat Enterprise Linux Server (2.6.18-86.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-86.el5 ro root=LABEL=/1 rhgb quiet
        initrd /initrd-2.6.18-86.el5.img
title Red Hat Enterprise Linux Server (2.6.18-86.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-86.el5
        module /vmlinuz-2.6.18-86.el5xen ro root=LABEL=/1 rhgb quiet
        module /initrd-2.6.18-86.el5xen.img

Comment 1 Chris Lumens 2008-03-28 13:57:42 UTC
Can you please attach /root/upgrade.log to this bug report?

Comment 2 Frank Arnold 2008-03-28 14:40:30 UTC
Created attachment 299475 [details]
/root/upgrade.log

This is from the upgrade of Snapshot 1 to Snapshot 2.
Unfortunately, it was created with German locale. So read 'wird aktualisiert'
as 'set to be updated' or some such.

Comment 3 Tim Burke 2008-03-28 15:39:23 UTC
What is the effect of this error?  How does it impact the user?  Is there a
manual workaround?

Comment 4 Frank Arnold 2008-03-28 15:59:58 UTC
(In reply to comment #3)
> What is the effect of this error?
System does not boot into Xen hypervisor.

> How does it impact the user?
Unusable system if there's no standard Linux kernel installed in parallel. At
least it's not possible to start up a Xen Dom0. 

> Is there a manual workaround?
Manually editing the bootloader config, either by booting into standard Linux
kernel if installed, or by using the a Rescue CD and edit /boot/grub/grub.conf
from there.

Comment 5 James Laska 2008-04-01 12:57:32 UTC
Created attachment 299885 [details]
anaconda-logs.tgz

1) Reproduced installing RHEL5.2 Beta and anaconda upgrading to 5.2 snap#1
2) Reproduced installing RHEL5.1 and anaconda upgrading to 5.2 nightly 20080401


Attaching log files from case#2 above:
-rw------- root/root	  1139 2008-03-31 12:57 anaconda-ks.cfg
-rw------- root/root	   940 2008-03-31 13:40 grub.conf-after
-rw------- root/root	   655 2008-03-31 13:25 grub.conf-before
-rw-r--r-- root/root	 16467 2008-03-31 12:57 install.log
-rw-r--r-- root/root	  2706 2008-03-31 12:56 install.log.syslog
-rw-r--r-- root/root	 12226 2008-03-31 16:16 upgrade.log
-rw-r--r-- root/root	   309 2008-03-31 15:19 upgrade.log.syslog
-rw------- root/root	227276 2008-03-31 16:20 var/log/anaconda.log

Comment 6 James Laska 2008-04-01 15:41:21 UTC
Also, on the tty1 console after upgrade (prior to reboot) I see:

FATAL: Could not open '/boot/System.map-2.6.18-53.el5xen': no such file or directory
No modules available for kernel "2.6.18-53.el5xen".
mkinitrd failed

Comment 7 James Laska 2008-04-01 15:44:24 UTC
Created attachment 299916 [details]
anaconda-upgrade-new-bootloader.tar.gz

Additional information:

When manually upgrading, and selecting to Create a *new* bootloader config, I
get:
 1) a working new kernel xen config
 2) a non-working old kernel xen config

I think option#2 failed due to the previous console FATAL message posted above.

Comment 8 James Laska 2008-04-01 20:20:18 UTC
FYI: Also hitting this error when upgrading from RHEL5.0 -> RHEL5.1 using
anaconda on a server-virt install

Comment 9 Chris Lumens 2008-04-01 21:48:38 UTC
James - can you reproduce this by yum upgrading from 5.1 to 5.2 or 5.0 to 5.1? 
We'd like to try to remove anaconda from this picture if at all possible.  Don
Zickus hasn't seen anything funny happening here in his test cases or
installing/upgrading kernel packages by hand with rpm.

Comment 10 James Laska 2008-04-02 13:40:31 UTC
Performing a yum upgrade from RHEL5.1 -> RHEL5.2-Server-20080402.nightly does
the right thing.

1) Only the installed kernel is updated (just kernel-xen, not kernel)

# rpm -qa | grep kernel
kernel-xen-2.6.18-53.el5
kernel-xen-2.6.18-87.el5

2) grub.conf shows correct entries

...
title Red Hat Enterprise Linux Server (2.6.18-87.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-87.el5
        module /vmlinuz-2.6.18-87.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-87.el5xen.img
title Red Hat Enterprise Linux Server (2.6.18-53.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-53.el5
        module /vmlinuz-2.6.18-53.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-53.el5xen.img


Comment 11 Chris Lumens 2008-04-02 18:47:53 UTC
Oh I see what's going on.  This is a side effect of the code we have that
rebuilds initrds in the case where you install extra modules from a driver disk.
 The test for whether to do that or not wasn't as smart as it should be.

Comment 13 Chris Lumens 2008-04-02 19:03:57 UTC
This will be fixed in anaconda-11.1.2.111-1.

Comment 15 Frank Arnold 2008-04-02 22:06:21 UTC
(In reply to comment #11)
> Oh I see what's going on.  This is a side effect of the code we have that
> rebuilds initrds in the case where you install extra modules from a driver disk.
>  The test for whether to do that or not wasn't as smart as it should be.

I looked at your patch, and now I'm just curious about what happens if someone
adds additional driver modules for the Xen Dom0 kernel... wouldn't this trigger
the same bug again?

Maybe that's unsupported, but I'm really curious about it ;)

Comment 16 Chris Lumens 2008-04-02 22:18:06 UTC
Unsure.  I am leaning towards yes, it would still hit the bug.  However I don't
have the setup here to be able to test that scenario.

Comment 17 James Laska 2008-04-03 17:48:19 UTC
Tested anaconda-11.1.2.111-1 (RHEL5.2-Server-20080402.0):

[PASSED] - anaconda upgrade from 5.1, update existing bootloader 
[PASSED] - anaconda upgrade from 5.1, write new bootloader
[PASSED] - anaconda upgrade from 5.1, bootloader untouched

As stated in comment#15 and comment#16, there may still be exposure in xen
environments where driver disks are needed.  That use case seems a bit obscure.
 But, it's there.

Comment 18 Frank Arnold 2008-04-04 09:09:23 UTC
(In reply to comment #17)
> As stated in comment#15 and comment#16, there may still be exposure in xen
> environments where driver disks are needed.  That use case seems a bit obscure.
>  But, it's there.

To fix this entirely one would need to refine recreateInitrd in packages.py. If
it is a xen0 kernel, it needs to pass the --multiboot option to
/sbin/new-kernel-package.
For details look at the kernel RPM spec, or just ask your favorite Xen package
maintainer...

Comment 20 errata-xmlrpc 2008-05-21 15:33:46 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2008-0397.html