Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 625943 - [LSI 5.7 bug] [CR183068] lvm volume group and block devices get deleted after node shutdown causing the node to not come back up
[LSI 5.7 bug] [CR183068] lvm volume group and block devices get deleted after...
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: lvm2-cluster (Show other bugs)
5.5
x86_64 Linux
medium Severity medium
: rc
: 5.7
Assigned To: LVM and device-mapper development team
Cluster QE
: OtherQA
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-08-20 17:14 EDT by Abdel Jalal
Modified: 2011-05-03 04:55 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-03 04:55:11 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
VG_backup Cluster.conf host log (113.87 KB, application/x-zip-compressed)
2010-08-20 17:14 EDT, Abdel Jalal
no flags Details
Physical devices (4.17 KB, text/plain)
2010-09-22 16:37 EDT, Abdel Jalal
no flags Details
Console output - Devices discovery during the boot (35.84 KB, text/plain)
2010-09-22 18:02 EDT, Abdel Jalal
no flags Details

  None (edit)
Description Abdel Jalal 2010-08-20 17:14:53 EDT
Created attachment 440054 [details]
VG_backup Cluster.conf host log

Description of problem:

Configuration has 4 x64 nodes running RH5.5 and all nodes have RHCS configured. Node 1 was shutdown and tried to bring it back up but got stuck while trying to mount gfs. The only way to bring it up is to unplug all the cables then re-insert power cable only. When trying to start gfs service it shows that the paths are unavailable because the volume group (lvm_vg) directory has been deleted during the shutdown along with the block devices used by lvm. Host log shows a soft lockup


[root@kswc-marine ~]# service gfs start gfs
Mounting GFS filesystems:  /sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol0"
/sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol1"
/sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol2"
/sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol3"
/sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol4"
/sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol5"
/sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol6"
/sbin/mount.gfs: invalid device path "/dev/lvm_vg/lvol7"
                                                        [FAILED]

[root@kswc-marine ~]# chkconfig --list | egrep "cman|gfs|rgmanage|clvmd"
clvmd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
cman            0:off   1:off   2:on    3:on    4:on    5:on    6:off
gfs             0:off   1:off   2:on    3:on    4:on    5:on    6:off
gfs2            0:off   1:off   2:off   3:off   4:off   5:off   6:off
rgmanager       0:off   1:off   2:on    3:on    4:on    5:on    6:off


The content of /dev/mapper/ before node shutdown
crw------- 1 root root  10, 63 Aug 17 14:20 control
brw-rw---- 1 root disk 253,  2 Aug 17 14:29 lvm_vg-lvol0
brw-rw---- 1 root disk 253,  3 Aug 17 14:29 lvm_vg-lvol1
brw-rw---- 1 root disk 253,  4 Aug 17 14:29 lvm_vg-lvol2
brw-rw---- 1 root disk 253,  5 Aug 17 14:29 lvm_vg-lvol3
brw-rw---- 1 root disk 253,  6 Aug 17 14:29 lvm_vg-lvol4
brw-rw---- 1 root disk 253,  7 Aug 17 14:29 lvm_vg-lvol5
brw-rw---- 1 root disk 253,  8 Aug 17 14:29 lvm_vg-lvol6
brw-rw---- 1 root disk 253,  9 Aug 17 14:29 lvm_vg-lvol7
brw-rw---- 1 root disk 253,  0 Aug 17 14:20 VolGroup00-LogVol00
brw-rw---- 1 root disk 253,  1 Aug 17 14:20 VolGroup00-LogVol01

/dev/mapper/ after the shoutdown/back online
crw------- 1 root root  10, 63 Aug 19 14:04 control
brw-rw---- 1 root disk 253,  0 Aug 19 14:05 VolGroup00-LogVol00
brw-rw---- 1 root disk 253,  1 Aug 19 14:04 VolGroup00-LogVol01

/dev/lvm_vg/ Before shutdown
lrwxrwxrwx 1 root root 24 Aug 17 14:29 lvol0 -> /dev/mapper/lvm_vg-lvol0
lrwxrwxrwx 1 root root 24 Aug 17 14:29 lvol1 -> /dev/mapper/lvm_vg-lvol1
lrwxrwxrwx 1 root root 24 Aug 17 14:29 lvol2 -> /dev/mapper/lvm_vg-lvol2
lrwxrwxrwx 1 root root 24 Aug 17 14:29 lvol3 -> /dev/mapper/lvm_vg-lvol3
lrwxrwxrwx 1 root root 24 Aug 17 14:29 lvol4 -> /dev/mapper/lvm_vg-lvol4
lrwxrwxrwx 1 root root 24 Aug 17 14:29 lvol5 -> /dev/mapper/lvm_vg-lvol5
lrwxrwxrwx 1 root root 24 Aug 17 14:29 lvol6 -> /dev/mapper/lvm_vg-lvol6

After shutdown, lvm_vg got deleted

How reproducible:
Setup 4 nodes running rh5.5 with rhcs/gfs

Steps to Reproduce:
1.Shutdown one node and bring it back online
2.Notice it will stop at the mounting gfs ... and hangs there 
3.Unplug all cables and power the node back on and verify the volume group and the block devices used by lvm have been deleted
Comment 1 Abdel Jalal 2010-09-01 16:40:11 EDT
Any updates?
Comment 2 Robert Peterson 2010-09-14 15:42:39 EDT
GFS uses and has no control over the underlying lvm devices,
so the problem goes back to lvm and/or the hardware layers
below it.  Reassigning to lvm2.
Comment 4 Ben Marzinski 2010-09-16 14:50:18 EDT
If the phsyical volumes aren't there, then there is no way for LVM to assemble your logical volume.  You need to figure out why your physical volumes didn't get discovered by your system during bootup.  Do you have the proper device driver loaded?
Comment 5 Abdel Jalal 2010-09-22 16:37:45 EDT
Created attachment 449050 [details]
Physical devices
Comment 6 Abdel Jalal 2010-09-22 16:39:14 EDT
I was able to recreate the issue and all the physical devices are there after the host came back up.
Comment 7 Abdel Jalal 2010-09-22 18:02:03 EDT
Created attachment 449059 [details]
Console output  - Devices discovery during  the boot

Added console output showing devices discovery during the boot
Comment 8 Abdel Jalal 2010-10-04 00:07:06 EDT
Any update on this bug?
Comment 9 Milan Broz 2010-10-04 06:22:19 EDT
Please add rpm -q lvm2 lvm2-cluster.

Maybe it is dup of bug #592125 resp. bug #588903 which should be fixed in current z-stream (see bug #591524).
Comment 10 Abdel Jalal 2010-10-05 15:59:34 EDT
#rpm -q lvm2 lvm2-cluster
lvm2-2.02.56-8.el5
lvm2-cluster-2.02.56-7.el5
Comment 11 Andrius Benokraitis 2010-10-12 18:02:06 EDT
Milan - anything else needed from LSI?
Comment 12 Abdel Jalal 2010-10-18 17:18:58 EDT
Milan - Any update on this?
Comment 13 Milan Broz 2010-10-19 03:14:13 EDT
See log on bug #588903 - last respin contains another fix for 5.6, I hope this bug is just some side effect.

Is this bug reproducible with released z-stream 5.5 packages (it should already contain fix for init_info section)?
(Try lvm2-cluster-2.02.56-7.el5_5.4 + lvm2-2.02.56-8.el5_5.6 here.)
Comment 14 Abdel Jalal 2010-10-22 19:22:56 EDT
Hardware is being used for other testings. I will update this BZ once I have found other hardware. Can you please post the link to those lvm2?
Comment 15 Andrius Benokraitis 2010-11-10 11:03:40 EST
Abdel - just use RHEL 5.6 Alpha at this point.
Comment 17 Andrius Benokraitis 2010-12-02 21:06:45 EST
Deferring to RHEL 5.7 - out of time for 5.6.
Comment 18 Tom Coughlan 2011-02-01 09:25:32 EST
Abdel,

Any update on this?
Comment 19 Milan Broz 2011-05-03 04:55:11 EDT
I think that bug was already fixed in packages mentioned in comment #13.
Requested info not provided for 3 months, closing.

Please reopen the bug and provide requested info if you are able to reproduce it with released packages, thanks.

Note You need to log in before you can comment on or make changes to this bug.