RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 606528 - lvm device scan needed in order for re-enabled mirror device to show up
Summary: lvm device scan needed in order for re-enabled mirror device to show up
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.0
Hardware: All
OS: Linux
medium
high
Target Milestone: rc
: ---
Assignee: Petr Rockai
QA Contact: Corey Marthaler
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-06-21 20:57 UTC by Corey Marthaler
Modified: 2010-11-10 21:08 UTC (History)
9 users (show)

Fixed In Version: lvm2-2.02.72-2.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-11-10 21:08:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
/etc/lvm/backup/helter_skelter (3.71 KB, text/plain)
2010-06-25 21:52 UTC, Corey Marthaler
no flags Details
/etc/lvm/archive/helter_skelter_00000.vg (2.29 KB, text/plain)
2010-06-25 21:52 UTC, Corey Marthaler
no flags Details
/etc/lvm/archive/helter_skelter_00001.vg (2.31 KB, text/plain)
2010-06-25 21:53 UTC, Corey Marthaler
no flags Details
/etc/lvm/archive/helter_skelter_00002.vg (3.73 KB, text/plain)
2010-06-25 21:54 UTC, Corey Marthaler
no flags Details
lvconvert -vvvv (74.90 KB, text/plain)
2010-06-30 16:04 UTC, Corey Marthaler
no flags Details

Description Corey Marthaler 2010-06-21 20:57:29 UTC
Description of problem:
After a successful device failure and mirror repair, helter_skelter re-enables the failed device(s) and then assumes it's automatically added back into the VG before using them in an upconvert. This assumption appears to have regressed. Now lvm realizes they've returned but doesn't allow them to be used again until a pvscan is run. This used to be the case in early RHEL5, but then that changed later in RHEL5, and now appears to have changed again making it difficult to keep tests up to date.

[root@taft-01 ~]# lvscan
  WARNING: Inconsistent metadata found for VG helter_skelter - updating to use version 8
  Missing device /dev/sdh1 reappeared, updating metadata for VG helter_skelter to version 8.
  Missing device /dev/sdg1 reappeared, updating metadata for VG helter_skelter to version 8.
  ACTIVE            '/dev/helter_skelter/syncd_primary_log_2legs_1' [600.00 MiB] inherit
  ACTIVE            '/dev/vg_taft01/lv_root' [32.30 GiB] inherit
  ACTIVE            '/dev/vg_taft01/lv_home' [25.62 GiB] inherit
  ACTIVE            '/dev/vg_taft01/lv_swap' [9.83 GiB] inherit

Version-Release number of selected component (if applicable):
Version 2.02.67+ (custom build by Brassow)

How reproducible:
Everytime

Comment 2 Petr Rockai 2010-06-23 14:53:14 UTC
Actually, I am not sure. Treatment of inconsistent metadata should be the same as it used to be in later RHEL5 -- at least I don't remember changing any of this. We have a rudimentary test for this that is not catching any change in behaviour in past 12 months, but maybe we need to augment the test to cover some further scenarios.

I'll look into reproducing the problem. Corey, can you maybe provide the metadata from the system after the repair but before the inconsistent metadata is corrected? Thanks! (It should be available in the metadata backup directory.)

Comment 3 Corey Marthaler 2010-06-25 21:50:46 UTC
Posting everything found in /etc/lvm/[backup|cache], before a pvscan.

Comment 4 Corey Marthaler 2010-06-25 21:52:04 UTC
Created attachment 427002 [details]
/etc/lvm/backup/helter_skelter

Comment 5 Corey Marthaler 2010-06-25 21:52:59 UTC
Created attachment 427004 [details]
/etc/lvm/archive/helter_skelter_00000.vg

Comment 6 Corey Marthaler 2010-06-25 21:53:35 UTC
Created attachment 427006 [details]
/etc/lvm/archive/helter_skelter_00001.vg

Comment 7 Corey Marthaler 2010-06-25 21:54:06 UTC
Created attachment 427007 [details]
/etc/lvm/archive/helter_skelter_00002.vg

Comment 8 Petr Rockai 2010-06-28 19:54:07 UTC
Corey, what command do you use to upconvert the mirror? I have written following script:

aux prepare_vg 3
lvcreate -m 1 --ig -L 1 -n 2way $vg $dev1 $dev2 $dev3:0
disable_dev $dev2
echo n | lvconvert --repair $vg/2way 2>&1 | tee 2way.out
lvs -a -o +devices | not grep unknown
# the device is linear at this point
enable_dev $dev2
lvconvert -m 1 $vg/2way $dev1 $dev2 $dev3:0
check mirror $vg 2way $dev3

and it seems to work as expected with current CVS: the last lvconvert is saying this:

+ lvconvert -m 1 LVMTEST28808vg/2way /srv/build/lvm2/cvs-upstream/default/test/LVMTEST28808.YJdmEzPsP6/dev/mapper/LVMTEST28808pv1 /srv/build/lvm2/cvs-upstream/default/test/LVMTEST28808.YJdmEzPsP6/dev/mapper/LVMTEST28808pv2 /srv/build/lvm2/cvs-upstream/default/test/LVMTEST28808.YJdmEzPsP6/dev/mapper/LVMTEST28808pv3:0
  WARNING: Inconsistent metadata found for VG LVMTEST28808vg - updating to use version 8
  Missing device /srv/build/lvm2/cvs-upstream/default/test/LVMTEST28808.YJdmEzPsP6/dev/mapper/LVMTEST28808pv2 reappeared, updating metadata for VG LVMTEST28808vg to version 8.
  WARNING: This metadata update is NOT backed up
  WARNING: This metadata update is NOT backed up

the volume is mirrored again after the lvconvert, as checked by the last line of the script.

Thanks again.

Comment 9 Corey Marthaler 2010-06-28 22:03:01 UTC
'lvconvert -m $legnum -b $vg/$mirror @pvlist'

The pvlist includes the devices that were just failed/re-enabled. So that appears to be the only difference between our two cmds.

Comment 10 Petr Rockai 2010-06-29 22:06:18 UTC
Hm, could you get the output (ideally with -vvvv) of the failing lvconvert? I.e. your

lvconvert -m $legnum -b $vg/$mirror @pvlist

(with -vvvv added) on a volume group that has inconsistent metadata after the disabled device returned. Something is going wrong with that command -- it could be failing to get a lock or something like that maybe, or some other environment dependency is tripping the code. Also, by any chance, are you running in a cluster, or just locally?

Ta.

Comment 11 Corey Marthaler 2010-06-30 16:04:59 UTC
Created attachment 428028 [details]
lvconvert -vvvv

Comment 12 Petr Rockai 2010-06-30 20:34:02 UTC
I see. This shows up in the logs:

#metadata/metadata.c:3626   Cannot change VG helter_skelter while PVs are missing.
#metadata/metadata.c:3627   Consider vgreduce --removemissing.

this means, that your VG is incomplete at this point and you cannot upconvert the mirror without first fixing it. It would be interesting to know how you got into this situation.

What we have:
- /dev/sdg is failed
- a mirror write happens, which trips dmeventd, which
  - runs lvconvert --repair --use-policies ... this recovers the mirror
  - which in turn does (kind of) vgreduce --removemissing ... this removes /dev/sdg from helter_skelter

If you run vgextend at this point, it should notice that /dev/sdg disappeared from helter_skelter, update the inconsistent metadata and all should be well. This is assuming that all of the above worked.

So what I have found is that the second bullet under dmeventd, that is removal of /dev/sdg never happens with current code. This is a regression, and a likely cause for this bug. I will shortly submit a patch upstream. I have also corrected our upstream tests so this does not happen again...

Comment 13 Petr Rockai 2010-07-27 20:08:24 UTC
Fixed upstream in Version 2.02.70 - 6th July 2010: Restore the removemissing behaviour of lvconvert --repair --use-policies.

Comment 15 Corey Marthaler 2010-07-29 20:17:34 UTC
Fix verified in the latest build.

2.6.32-52.el6.x86_64

lvm2-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
lvm2-libs-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
lvm2-cluster-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
udev-147-2.21.el6    BUILT: Mon Jul 12 04:55:00 CDT 2010
device-mapper-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
device-mapper-libs-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
device-mapper-event-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
device-mapper-event-libs-1.02.53-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010
cmirror-2.02.72-3.el6    BUILT: Wed Jul 28 15:39:43 CDT 2010

Comment 16 releng-rhel@redhat.com 2010-11-10 21:08:07 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.