Bug 617535 - Upgrade when / is on ISCSI failed
Summary: Upgrade when / is on ISCSI failed
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: anaconda   
(Show other bugs)
Version: 6.0
Hardware: All
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: Ales Kozumplik
QA Contact: Release Test Team
URL:
Whiteboard:
Keywords:
Depends On: 594411
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-23 11:40 UTC by Alexander Todorov
Modified: 2014-09-30 23:39 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 594411
Environment:
Last Closed: 2010-12-06 16:10:39 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
grub.conf after 1st install (949 bytes, text/plain)
2010-07-23 12:24 UTC, Alexander Todorov
no flags Details
grub.conf after upgrade (1.92 KB, text/plain)
2010-07-23 12:25 UTC, Alexander Todorov
no flags Details
anaconda.log (13.90 KB, text/plain)
2010-07-23 12:25 UTC, Alexander Todorov
no flags Details
program.log (20.03 KB, text/plain)
2010-07-23 12:25 UTC, Alexander Todorov
no flags Details
storage.log (51.82 KB, text/plain)
2010-07-23 12:25 UTC, Alexander Todorov
no flags Details
syslog (37.96 KB, text/plain)
2010-07-23 12:26 UTC, Alexander Todorov
no flags Details
yum.log (2.79 KB, text/plain)
2010-07-23 12:26 UTC, Alexander Todorov
no flags Details

Comment 1 Alexander Todorov 2010-07-23 11:41:50 UTC
Steps to reproduce: 

1) Install a system where /boot is on local disk and / on iSCSI
2) Upgrade this system to newer snapshot

Actual result: 
System doesn't boot. root device can't be found. 

Will try a simpler reproducer and upload logs.

Comment 2 Alexander Todorov 2010-07-23 12:24:41 UTC
Created attachment 433935 [details]
grub.conf after 1st install

The system has /boot and swap on local disk and / on iSCSI disk. No LVM.

Comment 3 Alexander Todorov 2010-07-23 12:25:14 UTC
Created attachment 433936 [details]
grub.conf after upgrade

Comment 4 Alexander Todorov 2010-07-23 12:25:27 UTC
Created attachment 433937 [details]
anaconda.log

Comment 5 Alexander Todorov 2010-07-23 12:25:44 UTC
Created attachment 433939 [details]
program.log

Comment 6 Alexander Todorov 2010-07-23 12:25:53 UTC
Created attachment 433940 [details]
storage.log

Comment 7 Alexander Todorov 2010-07-23 12:26:07 UTC
Created attachment 433941 [details]
syslog

Comment 8 Alexander Todorov 2010-07-23 12:26:18 UTC
Created attachment 433942 [details]
yum.log

Comment 9 Alexander Todorov 2010-07-23 12:27:13 UTC
(In reply to comment #3)
> Created an attachment (id=433936) [details]
> grub.conf after upgrade    

The interesting part here is that there's an entry for the new kernel as well as additional entry for the already installed kernel.

Comment 10 Hans de Goede 2010-07-23 12:35:23 UTC
Hi,

(In reply to comment #9)
> (In reply to comment #3)
> > Created an attachment (id=433936) [details] [details]
> > grub.conf after upgrade    
> 
> The interesting part here is that there's an entry for the new kernel as well
> as additional entry for the already installed kernel.    

Which is I must admit a bit weird, but other then that everything looks fine. Will the system still boot when you select the old kernel ?

Are you doing this inside a kvm ?

Can you see if there is an initramfs-.... file for the new kernel, and how its size compares to the one for the old kernel ?

If the system boots with the old kernel, can you rebuild the initramfs for the old kernel (first move the old one to .bak) and see if the old kernel still boots with a re-generated (so new) initramfs ?

Thanks,

Hans

Comment 11 Alexander Todorov 2010-07-26 12:39:56 UTC
(In reply to comment #10)
> Hi,
> 
> (In reply to comment #9)
> > (In reply to comment #3)
> > > Created an attachment (id=433936) [details] [details] [details]
> > > grub.conf after upgrade    
> > 
> > The interesting part here is that there's an entry for the new kernel as well
> > as additional entry for the already installed kernel.    
> 
> Which is I must admit a bit weird, but other then that everything looks fine.
> Will the system still boot when you select the old kernel ?
> 

As far as I remember booting the old kernel failed as well. 


> Are you doing this inside a kvm ?
> 

Yes.

> Can you see if there is an initramfs-.... file for the new kernel, and how its
> size compares to the one for the old kernel ?
> 
> If the system boots with the old kernel, can you rebuild the initramfs for the
> old kernel (first move the old one to .bak) and see if the old kernel still
> boots with a re-generated (so new) initramfs ?
> 

I will try to compare both initramfs images as soon as I get a system in Beaker. There seems to be a problem now (bug #618123).

Comment 12 Hans de Goede 2010-07-26 14:05:46 UTC
Hi,

(In reply to comment #11)
<snip>

> > Are you doing this inside a kvm ?
> > 
> 
> Yes.
>

Hmm, there is a kernel bug in RHEL-5 and RHEL-6 kernel hosts, which get exposed by doing installs inside kvm with recent RHEL-6 builds. This bug is causing a problem with memory corruption, which is causing all kind of random errors. I've a feeling that you are hitting this here.

Can you please try to reproduce using real hardware ?

Regards,

Hans

Comment 13 Alexander Todorov 2010-07-27 14:27:43 UTC
(In reply to comment #10)
> 
> Are you doing this inside a kvm ?
>

Next results are from bare metal. 
 
> Can you see if there is an initramfs-.... file for the new kernel, and how its
> size compares to the one for the old kernel ?
> 

After upgrade completed but while still in anaconda I've switched to tty2 and the results are:

# ls -lh initramfs-2.6.32-*
-rw-r--r--. 1 root root 19M 2010-07-27 10:20 initramfs-2.6.32-37.el6.x86_64.img
-rw-r--r--. 1 root root 19M 2010-07-27 10:21 initramfs-2.6.32-52.el6.x86_64.img

The system was able to boot and find the root fs after upgrade.


There are two issues here: 
1) grub.conf will add additional config entry for the old kernel
2) a KVM domU will not boot after upgrade while bare metal will

Comment 14 Hans de Goede 2010-07-27 15:20:14 UTC
Hi,

(In reply to comment #13)
> (In reply to comment #10)
> > 
> > Are you doing this inside a kvm ?
> >
> 
> Next results are from bare metal. 
> 
> > Can you see if there is an initramfs-.... file for the new kernel, and how its
> > size compares to the one for the old kernel ?
> > 
> 
> After upgrade completed but while still in anaconda I've switched to tty2 and
> the results are:
> 
> # ls -lh initramfs-2.6.32-*
> -rw-r--r--. 1 root root 19M 2010-07-27 10:20 initramfs-2.6.32-37.el6.x86_64.img
> -rw-r--r--. 1 root root 19M 2010-07-27 10:21 initramfs-2.6.32-52.el6.x86_64.img
> 
> The system was able to boot and find the root fs after upgrade.
> 

Ok, good!

> 
> There are two issues here: 
> 1) grub.conf will add additional config entry for the old kernel

Yes that is a (grubby) bug I'll gladly admit that. But that seems something to fix for 6.1 to me.

> 2) a KVM domU will not boot after upgrade while bare metal will    

That is very very likely bug 607650, which is a kernel issue on the *host* side (and yes Fedora hosts have the same problem). To verify you need to install a fixed kernel on the *host* side.

Regards,

Hans

Comment 15 Denise Dumas 2010-07-29 21:03:22 UTC
Please note that we explicitly do not support upgrades from one snapshot to another, or from RHEL5 to RHEL6. Only upgrades from one minor release to the next.

Comment 16 Ales Kozumplik 2010-10-08 13:40:01 UTC
> There are two issues here: 
> 1) grub.conf will add additional config entry for the old kernel
> 2) a KVM domU will not boot after upgrade while bare metal will

Hi Alex,

OK to close this one given that 594411 is in VERIFIED now and other problem was caused by (now fixed 607650)?

If in doubt can you please retest with the latest RHEL6 RC? (note that you need to have a fix on the host machine too if you want to test this in KVM).

Thanks.
Ales

Comment 17 Ales Kozumplik 2010-12-06 16:10:39 UTC
Fixed already per comment 16.


Note You need to log in before you can comment on or make changes to this bug.