Bug 608879 - clvmd causing problems in/after fault scenarios
clvmd causing problems in/after fault scenarios
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: Milan Broz
Corey Marthaler
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-06-28 15:51 EDT by Jonathan Earl Brassow
Modified: 2013-02-28 23:09 EST (History)
9 users (show)

See Also:
Fixed In Version: lvm2-2.02.69-2.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-10 16:08:16 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jonathan Earl Brassow 2010-06-28 15:51:10 EDT
I'm not sure if this is due to how clvmd is caching or perhaps some funny udev problem...  I know that it is causing a lot of problems with the automated testing.

The scenario is:
1) create cluster mirror
2) kill a device that a leg is on
3) write to device to force dmeventd to do repair
4) bring device back
5) attempt to convert back to mirror [fails]
*) repeating #5 will not help
**) restarting clvmd (on all machines) makes #5 work

I'm not sure if dmeventd is required in these steps.  Perhaps you could replace #3 with a simple call to 'lvconvert --repair'.

Here are the exact steps to reproduce (every time):
[root@bp-01 ~]# lvcreate -m1 -L 500M -n lv vg
  Logical volume "lv" created
[root@bp-01 ~]# devices
  LV            Copy%  Devices
  lv            100.00 lv_mimage_0(0),lv_mimage_1(0)
  [lv_mimage_0]        /dev/sdb1(0)
  [lv_mimage_1]        /dev/sdc1(0)
  [lv_mlog]            /dev/sdi1(0)
[root@bp-01 ~]# off.sh sdc <-- done on all machines
Turning off sdc
[root@bp-01 ~]# # 'dd if=/dev/zero of=/dev/vg/lv bs=4M count=1' on bp-02
[root@bp-01 ~]# devices
  Couldn't find device with uuid zwxlQu-3hG3-RhIu-izSu-z8gB-Onfy-hfwH77.
  LV      Copy%  Devices
  lv             /dev/sdb1(0)
[root@bp-01 ~]# on.sh sdc <-- done on all machines
Turning on sdc
[root@bp-01 ~]# pvscan
  WARNING: Volume Group vg is not consistent
  PV /dev/sdb1   VG vg        lvm2 [233.75 GiB / 233.27 GiB free]
  PV /dev/sdc1   VG vg        lvm2 [233.75 GiB / 233.75 GiB free]
  PV /dev/sdd1   VG vg        lvm2 [233.75 GiB / 233.75 GiB free]
  PV /dev/sde1   VG vg        lvm2 [233.75 GiB / 233.75 GiB free]
  PV /dev/sdf1   VG vg        lvm2 [233.75 GiB / 233.75 GiB free]
  PV /dev/sdg1   VG vg        lvm2 [233.75 GiB / 233.75 GiB free]
  PV /dev/sdh1   VG vg        lvm2 [233.75 GiB / 233.75 GiB free]
  PV /dev/sdi1   VG vg        lvm2 [233.75 GiB / 233.75 GiB free]
  PV /dev/sda2   VG vg_bp01   lvm2 [148.52 GiB / 0    free]
  Total: 9 [1.97 TiB] / in use: 9 [1.97 TiB] / in no VG: 0 [0   ]
[root@bp-01 ~]# lvconvert -m1 vg/lv
  WARNING: Inconsistent metadata found for VG vg - updating to use version 19
  Missing device /dev/sdc1 reappeared, updating metadata for VG vg to version 19.
  /dev/vg/lv_mlog: not found: device not cleared
  Aborting. Failed to wipe mirror log.
  Failed to initialise mirror log.
[root@bp-01 ~]# lvconvert -m1 vg/lv
  /dev/vg/lv_mlog: not found: device not cleared
  Aborting. Failed to wipe mirror log.
  Failed to initialise mirror log.
[root@bp-01 ~]# killall clvmd
[root@bp-01 ~]# clvmd
[root@bp-01 ~]# lvconvert -m1 vg/lv
  vg/lv: Converted: 1.6%
  vg/lv: Converted: 95.2%
  vg/lv: Converted: 100.0%
[root@bp-01 ~]# devices
  LV            Copy%  Devices
  lv            100.00 lv_mimage_0(0),lv_mimage_1(0)
  [lv_mimage_0]        /dev/sdb1(0)
  [lv_mimage_1]        /dev/sdc1(0)
  [lv_mlog]            /dev/sdi1(0)
Comment 2 RHEL Product and Program Management 2010-06-28 16:02:57 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 3 Milan Broz 2010-06-29 06:23:50 EDT
I think this is dup of ug #595523 (still unresolved).

Which version of lvm rpm is that?
Comment 4 Milan Broz 2010-06-29 10:58:37 EDT
Ok, I can reproduce it. (with lvm2-2.02.68-1.el6)

It works properly with local mirror, so the problem is somewhere in updating metadata cache on remote nodes after PV reappears.
Comment 5 Milan Broz 2010-07-01 17:05:02 EDT
Fixed upstream, not balanced memlock caused clvmd to avoid rescans.
Comment 7 Corey Marthaler 2010-08-13 17:05:55 EDT
VG reconfiguration test cases (same as in bug 595523) now pass. Marking this verified.
Comment 8 releng-rhel@redhat.com 2010-11-10 16:08:16 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.