Bug 675744 - System fails to boot after upgrade, rootfs on LVM not found
Summary: System fails to boot after upgrade, rootfs on LVM not found
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.1
Hardware: s390x
OS: Linux
high
high
Target Milestone: beta
: ---
Assignee: Zdenek Kabelac
QA Contact: Corey Marthaler
URL:
Whiteboard:
Depends On:
Blocks: 670159
TreeView+ depends on / blocked
 
Reported: 2011-02-07 15:41 UTC by Jan Stodola
Modified: 2011-05-19 14:27 UTC (History)
11 users (show)

Fixed In Version: lvm2-2.02.83-1.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-05-19 14:27:20 UTC
Target Upstream Version:


Attachments (Terms of Use)
lsinitrd /boot/initramfs-2.6.32-112.el6.s390x.img (100.66 KB, text/plain)
2011-02-07 15:43 UTC, Jan Stodola
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0772 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2011-05-18 18:08:31 UTC

Description Jan Stodola 2011-02-07 15:41:55 UTC
Created attachment 477437 [details]
init.log

Description of problem:
I tried to upgrade RHEL6.0 GA (s390x arch) to the latest nightly build (RHEL6.1-20110207.n.0), but system fails to boot after the upgrade:
...
dracut: Scanning devices dasda2 dasdb1 dasdc1 dasdd1 dasde1 dasdf1 dasdg1 dasdh1
  for LVM logical volumes vg_rtt4/lv_root vg_rtt4/lv_swap  
dracut: No volume groups found 
dracut: Volume group "vg_rtt4" not found 
dracut: Skipping volume group vg_rtt4 
dracut Warning: LVM vg_rtt4/lv_root not found 
dracut Warning: LVM vg_rtt4/lv_swap not found 
  
Boot has failed, sleeping forever.  


There was default (LVM) partitioning on the system:
/dev/dasda1 /boot
/dev/dasda2 + dasdb1 + dasdc1 +..+ dasdh1: PVs
/ and swap on LVs

From initramfs:
dracut:/# cat /proc/cmdline
cat /proc/cmdline  
root=/dev/mapper/vg_rtt4-lv_root rd_DASD=0.0.3024 rd_DASD=0.0.3124 rd_DASD=0.0.3
224 rd_DASD=0.0.3324 rd_DASD=0.0.3424 rd_DASD=0.0.3524 rd_DASD=0.0.3624 rd_DASD=
0.0.3724 rd_LVM_LV=vg_rtt4/lv_root rd_LVM_LV=vg_rtt4/lv_swap rd_NO_LUKS rd_NO_MD
 rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us cio_ignore=all,
!0.0.0009 crashkernel=auto rdinitdebug rdshell BOOT_IMAGE=3  

dracut:/# ls -l /dev/mapper
ls -l /dev/mapper  
total 0  
crw-rw---- 1 0 root 10, 60 Feb  7 15:40 control

dracut:/# lvm pvs
lvm pvs  
File descriptor 9 (/.console_lock) leaked on lvm invocation. Parent PID 934: sh 
 
dracut:/# cat /proc/modules
cat /proc/modules  
dasd_eckd_mod 92983 24 - Live 0x000003c00090d000  
dasd_mod 106272 9 dasd_eckd_mod, Live 0x000003c0008ac000
dm_mod 106735 0 - Live 0x000003c000848000  

dracut:/# blkid
blkid  
/dev/dasda1: UUID="07f2648e-ad20-4fc9-a801-52a98596e79c" SEC_TYPE="ext2" TYPE="e
xt3"   
/dev/dasda2: UUID="Hgt8pJ-jVfc-QWoC-GcdU-0pdD-8JSt-zBO2Ah" TYPE="LVM2_member"   
/dev/dasdb1: UUID="21Pm3p-0Pd2-yLNx-2A0P-x1lg-esOn-znqlet" TYPE="LVM2_member"   
/dev/dasdc1: UUID="JfKfDc-15tO-BFvB-6m43-xSuM-2RSc-W3XjLp" TYPE="LVM2_member"   
/dev/dasdd1: UUID="lCFFnu-f1R1-YfMP-EMcC-qYmN-gKeT-HsDMJo" TYPE="LVM2_member"   
/dev/dasde1: UUID="pZL09K-Ty6N-Klha-YLEG-YFbT-zA6y-J1Vbw5" TYPE="LVM2_member"   
/dev/dasdf1: UUID="bQBS6z-30fq-vzH8-5zM1-adf5-GZQ5-GsxNPP" TYPE="LVM2_member"   
/dev/dasdg1: UUID="hJu1B3-2Q23-Kgy7-Zmhs-pIPq-1a7w-YHj5b5" TYPE="LVM2_member"   
/dev/dasdh1: UUID="jVokU6-3Ojp-Sses-Txcl-hZiu-NyJX-PNLhsU" TYPE="LVM2_member"   


See init.log in the attachment.

There were no errors during the upgrade.
Cannot reproduce on x86_64 - system booted successfully after the upgrade.

Version-Release number of selected component (if applicable):
RHEL6.1-20110207.n.0
dracut-004-37.el6
kernel-2.6.32-112.el6
lvm2-2.02.82-1.el6

How reproducible:
always

Steps to Reproduce:
1. on s390x, install RHEL6.0 (autopartitioning, default package set)
2. create /etc/yum.redpo.d/rhel.repo pointing to the latest nightly tree
3. yum update
4. reboot
  
Actual results:
System fails to boot, rootfs not found.

Expected results:
Successful reboot into upgraded system.

Comment 1 Jan Stodola 2011-02-07 15:43:40 UTC
Created attachment 477438 [details]
lsinitrd /boot/initramfs-2.6.32-112.el6.s390x.img

Comment 2 Jan Stodola 2011-02-08 08:15:36 UTC
This bug seems to be caused by lvm2 package:
lvm2-2.02.72-8.el6.s390x OK
lvm2-2.02.82-1.el6.s390x FAIL

[root@rtt6 ~]# rpm -qa |grep lvm
lvm2-libs-2.02.72-8.el6.s390x
lvm2-2.02.72-8.el6.s390x
lvm2-libs-2.02.72-8.el6.s390
[root@rtt6 ~]# lvm pvs
  PV          VG      Fmt  Attr PSize PFree
  /dev/dasda2 vg_rtt6 lvm2 --   1.80g    0 
  /dev/dasdb1 vg_rtt6 lvm2 --   2.29g    0 
  /dev/dasdc1 vg_rtt6 lvm2 --   2.29g    0 
  /dev/dasdd1 vg_rtt6 lvm2 --   2.29g    0 
  /dev/dasde1 vg_rtt6 lvm2 --   2.29g    0 
  /dev/dasdf1 vg_rtt6 lvm2 --   2.29g    0 
  /dev/dasdg1 vg_rtt6 lvm2 --   2.29g    0 
  /dev/dasdh1 vg_rtt6 lvm2 --   2.29g    0 
  /dev/dasdi1 vg_rtt6 lvm2 --   9.17g    0 
  /dev/dasdj1 vg_rtt6 lvm2 --   9.17g    0 

[root@rtt6 ~]# yum update lvm2
...
[root@rtt6 ~]# rpm -qa |grep lvm
lvm2-libs-2.02.82-1.el6.s390
lvm2-2.02.82-1.el6.s390x
lvm2-libs-2.02.82-1.el6.s390x
[root@rtt6 ~]# lvm pvs
[root@rtt6 ~]# 

Reassigning to lvm2

Comment 3 Zdenek Kabelac 2011-02-08 10:34:51 UTC
Problem is caused by big-endian incompatibility with newer crc32 algorithm.
Fix is in development.

Comment 4 Zdenek Kabelac 2011-02-08 13:03:09 UTC
Fix committed upstream:

http://www.redhat.com/archives/lvm-devel/2011-February/msg00040.html

Problem was introduced with version 2.02.75 where the speedup of crc32 has been implemented. This code was reading uint32 in machine order while the crc32 algorithm expected the little-endian style of byte array reading.  

LVM2  code between version 2.02.75 - 2.02.83  generated incompatible CRC32 checksum on big-endian machines - and upgrade to fixed version is highly recommended.

Any metadata generated by broken lvm2 code on big endian architecture are incompatible with  little-endian generated versions.

Comment 9 errata-xmlrpc 2011-05-19 14:27:20 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0772.html


Note You need to log in before you can comment on or make changes to this bug.