Red Hat Bugzilla – Bug 594411
Upgrade when / is on ISCSI failed
Last modified: 2010-11-10 14:44:19 EST
Description of problem:
When upgrading a system which has / on iSCSI the system can't boot after upgrade.
Version-Release number of selected component (if applicable):
1/1, will re-test
Steps to Reproduce:
1. Install a system with older snapshot (I used 0428.3)
2. /boot and swap are on local disk, / is on iSCSI disk
3. Verify that system is able to boot
4. Start a second install
5. In stage2 select advances storage devices and add the iSCSI disk again.
6. Anaconda will recognize that there's a previous installation and ask you to Upgrade or Re-install.
7. Select Upgrade and proceed with default options (update bootloader config).
Install completes but system can't boot. There's the error message:
Error 15: File not found
Press any key to continue...
Thi slooks like a message from grub. Going into rescue mode to find out what's wrong. More info coming soon.
The system boots.
1) After the screen where anaconda asks you if you want to do fresh install or upgrade there is a screen which asks the user to select which storage devices will be part of the installation. I've selected all (local disk and iSCSI disk) and added them to the list of install target devices.
2) Bootloader selection is correctly set to the local disk.
3) This is independent of disk configuration. I tried with defaults (Use all space) when I have a local disk and iSCSI disk and it still happens.
Running in rescue mode reveals that grub.conf is incorrect.
It points to 2.6.32-23 vmlinuz and initramfs while on disk are only 2.6.32-25 from the latest kernel package. Fixing grub.conf made the system boot.
1) grub.conf hasn't been updated like anaconda said it will do.
2) The previous kernel package is removed. Not sure if this has always been true for anaconda but yum doesn't do that.
(In reply to comment #2)
> Fixing grub.conf made the system boot.
Scratch that. System boots half-way then fails to find the root device. I guess iSCSI fails to start. Will investigate later.
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release. Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release. This request is not yet committed for
(In reply to comment #3)
> (In reply to comment #2)
> > Fixing grub.conf made the system boot.
> Scratch that. System boots half-way then fails to find the root device. I guess
> iSCSI fails to start. Will investigate later.
Ok, putting this in need-info then till you're doie investigating.
there are at least two issues here. Let's fix the first one where anaconda isn't updating grub.conf properly. Fixing this will also help speed up testing for the second issue as I don't have to boot into rescue mode and manually update this file before proceeding further.
Moving this over to rvykydal as he knows the grub updating booty code the best. Radek, I think this is related to having / and /boot on different disks or some such, at the time we are doing grub configuration / updating there nothing iscsi specific happening in the code, once past mounting filesystems the iscsi disk is just another disk.
Alexander, have you tried:
1) regular upgrades
2) upgrades with a similar partitioning screen using 2 regular disks ?
(In reply to comment #7)
> 1) regular upgrades
> 2) upgrades with a similar partitioning screen using 2 regular disks ?
I can confirm the same fail "Error 15: File not found" for both cases. Fixing
grub.conf described in comment #2 helps. Update of grub.conf is
not business of anaconda, it is done by grubby (sbin/new-kernel-pkg) when
updating kernel package.
I tested with 0610 rhel6 nightly upgrading rehl6-beta1 with updates.img setting upgradeany in anaconda (I also had to hack-out some ValueErrors when handling rpm errors in anaconda callback).
grubby fatal error: unable to find a suitable template
grubby: doing this would leave no kernel entries. Not writing out new config.
groupdel: group 'saslauth' does not exist
warning: %postun(cyrus-sasl-2.1.23-4.1.el6.i686) scriptlet failed, exit status 6
/sbin/ldconfig: relative path `1' used to build cache
warning: %postun(dmraid-1.0.0.rc16-7.el6.i686) scriptlet failed, exit status 1
grubby fatal error: unable to find a suitable template
*** FINISHED INSTALLING PACKAGES ***
Assigning to grubby, I have no idea what is going on here, Peter - do you?
Created attachment 424719 [details]
upgrade.log failing upgrade
I can attach other anaconda logs if you need.
Please attach the original grub.conf .
To be clear, what Peter means is do you still have an image of the system in question, and could you get the grub.conf out of that image? If not can you reproduce this ? And if you can can you please attach the grub.conf of the installed system which fails to upgrade ?
I can reproduce using 2 local disks and default partitioning. grub.conf after initial installation looks like:
# grub.conf generated by anaconda
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/vg_dhcp70206-lv_root
# initrd /initrd-[generic-]version.img
title Red Hat Enterprise Linux (2.6.32-23.el6.x86_64)
kernel /vmlinuz-2.6.32-23.el6.x86_64 ro root=/dev/mapper/vg_dhcp70206-lv_root rd_LVM_LV=vg_dhcp70206/lv_root rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet
Created attachment 427915 [details]
Ends up the problem was more widespread. I duplicated it with a single disk and I'm pretty sure F13 is going to have the same problem.
We are clearing installonlypkgs, which lists the kernel packages to only install, on updates as well as new installs so the kernel install removes the old kernel, making grubby unhappy.
Tested on F13 and it works ok. On F13 the kernel update is not at the end, like it is in the RHEL6 test case. I watched it install the new kernel, then during the cleanup ('Finishing Upgrade Process' dialog) it removed the old kernel and re-updated grub.conf
I think what happens is that when the kernel is last there is a race condition with the cleanup, and the old kernel is cleaned up before grubby can update grub.conf (it needs the old kernel in place in order to detect the right grub.conf entry to use as a template).
committed 0783c488cdda8e7157511df1365ccc5ec95a2d33 to master branch of anaconda to fix this in the future.
with snap #8 (0722.0) the upgrade process completed and grub.conf now contains the new kernel. However comment #3 is still present, system doesn't boot.
I'll move this to VERIFIED and open a new bug for the second issue.
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.