Bug 594411 - Upgrade when / is on ISCSI failed
Summary: Upgrade when / is on ISCSI failed
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: anaconda
Version: 6.0
Hardware: All
OS: Linux
Target Milestone: rc
: ---
Assignee: Brian Lane
QA Contact: Release Test Team
Depends On:
Blocks: 617535
TreeView+ depends on / blocked
Reported: 2010-05-20 15:32 UTC by Alexander Todorov
Modified: 2010-11-10 19:44 UTC (History)
3 users (show)

Fixed In Version: anaconda-13.21.57-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 617535 (view as bug list)
Last Closed: 2010-11-10 19:44:19 UTC
Target Upstream Version:

Attachments (Terms of Use)
upgrade.log failing upgrade (7.17 KB, text/plain)
2010-06-17 08:23 UTC, Radek Vykydal
no flags Details
grub.conf (801 bytes, text/plain)
2010-06-30 08:55 UTC, Alexander Todorov
no flags Details

Description Alexander Todorov 2010-05-20 15:32:15 UTC
Description of problem:
When upgrading a system which has / on iSCSI the system can't boot after upgrade.

Version-Release number of selected component (if applicable):

How reproducible:
1/1, will re-test

Steps to Reproduce:
1. Install a system with older snapshot (I used 0428.3)
2. /boot and swap are on local disk, / is on iSCSI disk
3. Verify that system is able to boot
4. Start a second install
5. In stage2 select advances storage devices and add the iSCSI disk again. 
6. Anaconda will recognize that there's a previous installation and ask you to Upgrade or Re-install. 
7. Select Upgrade and proceed with default options (update bootloader config).
Actual results:
Install completes but system can't boot. There's the error message:
Error 15: File not found

Press any key to continue...

Thi slooks like a message from grub. Going into rescue mode to find out what's wrong. More info coming soon.

Expected results:
The system boots. 

Additional info:

Comment 1 Alexander Todorov 2010-05-20 16:08:31 UTC
Some observations:

1) After the screen where anaconda asks you if you want to do fresh install or upgrade there is a screen which asks the user to select which storage devices will be part of the installation. I've selected all (local disk and iSCSI disk) and added them to the list of install target devices.

2) Bootloader selection is correctly set to the local disk.

3) This is independent of disk configuration. I tried with defaults (Use all space) when I have a local disk and iSCSI disk and it still happens.

Comment 2 Alexander Todorov 2010-05-20 16:27:27 UTC
Running in rescue mode reveals that grub.conf is incorrect.

It points to 2.6.32-23 vmlinuz and initramfs while on disk are only 2.6.32-25 from the latest kernel package. Fixing grub.conf made the system boot.

1) grub.conf hasn't been updated like anaconda said it will do.

2) The previous kernel package is removed. Not sure if this has always been true for anaconda but yum doesn't do that.

Comment 3 Alexander Todorov 2010-05-20 16:28:37 UTC
(In reply to comment #2)
> Fixing grub.conf made the system boot.

Scratch that. System boots half-way then fails to find the root device. I guess iSCSI fails to start. Will investigate later.

Comment 4 RHEL Program Management 2010-05-20 17:46:47 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for

Comment 5 Hans de Goede 2010-05-21 12:20:41 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > Fixing grub.conf made the system boot.
> > 
> Scratch that. System boots half-way then fails to find the root device. I guess
> iSCSI fails to start. Will investigate later.    

Ok, putting this in need-info then till you're doie investigating.

Comment 6 Alexander Todorov 2010-05-21 12:37:58 UTC
Hi Hans,
there are at least two issues here. Let's fix the first one where anaconda isn't updating grub.conf properly. Fixing this will also help speed up testing for the second issue as I don't have to boot into rescue mode and manually update this file before proceeding further.

Comment 7 Hans de Goede 2010-05-21 13:18:04 UTC

Moving this over to rvykydal as he knows the grub updating booty code the best. Radek, I think this is related to having / and /boot on different disks or some such, at the time we are doing grub configuration / updating there nothing iscsi specific happening in the code, once past mounting filesystems the iscsi disk is just another disk.

Alexander, have you tried:
1) regular upgrades
2) upgrades with a similar partitioning screen using 2 regular disks ?



Comment 8 Radek Vykydal 2010-06-17 08:20:29 UTC
(In reply to comment #7)

> 1) regular upgrades
> 2) upgrades with a similar partitioning screen using 2 regular disks ?

I can confirm the same fail "Error 15: File not found" for both cases. Fixing
grub.conf described in comment #2 helps. Update of grub.conf is
not business of anaconda, it is done by grubby (sbin/new-kernel-pkg) when
updating kernel package.

I tested with 0610 rhel6 nightly upgrading rehl6-beta1 with updates.img setting upgradeany in anaconda (I also had to hack-out some ValueErrors when handling rpm errors in anaconda callback). 

upgrade.log says:

Upgrading openssh-server-5.3p1-17.el6.i686
grubby fatal error: unable to find a suitable template
grubby: doing this would leave no kernel entries. Not writing out new config.
groupdel: group 'saslauth' does not exist
warning: %postun(cyrus-sasl-2.1.23-4.1.el6.i686) scriptlet failed, exit status 6
/sbin/ldconfig: relative path `1' used to build cache
warning: %postun(dmraid-1.0.0.rc16-7.el6.i686) scriptlet failed, exit status 1
grubby fatal error: unable to find a suitable template

Assigning to grubby, I have no idea what is going on here, Peter - do you?

Comment 9 Radek Vykydal 2010-06-17 08:23:09 UTC
Created attachment 424719 [details]
upgrade.log failing upgrade

I can attach other anaconda logs if you need.

Comment 10 Peter Jones 2010-06-29 15:09:56 UTC
Please attach the original grub.conf .

Comment 11 Hans de Goede 2010-06-29 15:14:12 UTC
Hi atodorov,

To be clear, what Peter means is do you still have an image of the system in question, and could you get the grub.conf out of that image? If not can you reproduce this ? And if you can can you please attach the grub.conf of the installed system which fails to upgrade ?



Comment 13 Alexander Todorov 2010-06-30 08:45:03 UTC
I can reproduce using 2 local disks and default partitioning. grub.conf after initial installation looks like:

# grub.conf generated by anaconda
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_dhcp70206-lv_root
#          initrd /initrd-[generic-]version.img
title Red Hat Enterprise Linux (2.6.32-23.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-23.el6.x86_64 ro root=/dev/mapper/vg_dhcp70206-lv_root rd_LVM_LV=vg_dhcp70206/lv_root rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet
        initrd /initramfs-2.6.32-23.el6.x86_64.img

Comment 14 Alexander Todorov 2010-06-30 08:55:30 UTC
Created attachment 427915 [details]

Comment 15 Brian Lane 2010-07-01 00:31:13 UTC
Ends up the problem was more widespread. I duplicated it with a single disk and I'm pretty sure F13 is going to have the same problem.

We are clearing installonlypkgs, which lists the kernel packages to only install, on updates as well as new installs so the kernel install removes the old kernel, making grubby unhappy.

Comment 16 Brian Lane 2010-07-01 16:13:08 UTC
Tested on F13 and it works ok. On F13 the kernel update is not at the end, like it is in the RHEL6 test case. I watched it install the new kernel, then during the cleanup ('Finishing Upgrade Process' dialog) it removed the old kernel and re-updated grub.conf

I think what happens is that when the kernel is last there is a race condition with the cleanup, and the old kernel is cleaned up before grubby can update grub.conf (it needs the old kernel in place in order to detect the right grub.conf entry to use as a template).

committed 0783c488cdda8e7157511df1365ccc5ec95a2d33 to master branch of anaconda to fix this in the future.

Comment 19 Alexander Todorov 2010-07-23 11:39:33 UTC
with snap #8 (0722.0) the upgrade process completed and grub.conf now contains the new kernel. However comment #3 is still present, system doesn't boot. 

I'll move this to VERIFIED and open a new bug for the second issue.

Comment 20 releng-rhel@redhat.com 2010-11-10 19:44:19 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.