Red Hat Bugzilla – Bug 957692
LVM-on-SW-RAID is not activated on boot if using lvmetad and automatic volume activation
Last modified: 2013-05-21 23:15:37 EDT
Description of problem:
Since I installed updates on thu 25 apr 2013, my LVM metadata is destroyed on boot. This only happens to LVM-on-SW-RAID.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. create sw-raid array, raid 5
2. create a pv+vg+lv on it
3. mount the new lv by default
destroyed lvm metadata on vg_home, other lvm (those not on sw-raid) unaffected.
I can manually restore the metadata and have a correct vg/lv again (even fsck says it's ok)
/etc/fstab (I've switched /home to noauto)
/dev/mapper/vg_paul-lv_root / ext4 defaults 1 1
UUID=078909b0-991a-4197-ba9e-7dabded585e8 /boot ext4 defaults 1 2
/dev/mapper/vg_paul-lv_swap swap swap defaults 0 0
/dev/mapper/vg_home-lv_home /home ext4 defaults,noauto 1 2
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1 sdd1 sda1 sdc1
937316352 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/3 pages [0KB], 65536KB chunk
unused devices: <none>
# Generated by LVM2 version 2.02.98(2) (2012-10-15): Mon Apr 29 11:04:56 2013
contents = "Text Format Volume Group"
version = 1
description = "Created *after* executing 'vgs'"
creation_host = "paul.internal.Hupie.com" # Linux paul.internal.Hupie.com 3.9.0-0.rc8.git0.2.fc20.x86_64 #1 SMP Mon Apr 22 21:26:31 UTC 2013 x86_64
creation_time = 1367226296 # Mon Apr 29 11:04:56 2013
id = "zFrPhC-6TYC-uCaj-28N1-BDHH-M5qx-IQIgif"
seqno = 4
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = 
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0
metadata_copies = 0
id = "dsTImS-53ru-LCmJ-SfO9-70PV-QtFN-6gM8N7"
device = "/dev/md0" # Hint only
status = ["ALLOCATABLE"]
flags = 
dev_size = 1874632704 # 893.895 Gigabytes
pe_start = 3072
pe_count = 228836 # 893.891 Gigabytes
id = "yw5W2T-fnpb-jJ99-Zjab-FgSn-JKpO-1xuYbo"
status = ["READ", "WRITE", "VISIBLE"]
flags = 
creation_host = "paul.internal.Hupie.com"
creation_time = 1352899831 # 2012-11-14 14:30:31 +0100
segment_count = 1
start_extent = 0
extent_count = 228836 # 893.891 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
metadata is destroyed even when the volume was not mounted when rebooting.
manual restore procedure
pvcreate --uuid "dsTImS-53ru-LCmJ-SfO9-70PV-QtFN-6gM8N7" \
--restorefile /etc/lvm/backup/vg_home \
vgchange -ay vg_home
fsck.ext4 /dev/vg_home/lv_home -f
What leads you to think it is 'destroyed'?
Please provide the actual boot messages and error messages you are seeing when you try to look for it manually after booting and activate it.
Also give the versions of other packages involved like lvm2, udev, mdadm, dracut etc. and provide a copy of lvm.conf. Is lvmetad running?
(In reply to comment #2)
> What leads you to think it is 'destroyed'?
Because on _every_ reboot the PV on /dev/md0 is gone
and a pvscan /dev/md0 doesn't bring it back.
> Please provide the actual boot messages and error messages you are seeing
> when you try to look for it manually after booting and activate it.
> Also give the versions of other packages involved like lvm2, udev, mdadm,
> dracut etc. and provide a copy of lvm.conf. Is lvmetad running?
Going to attach it. The attached files are generated by a boot that has /home mounted on boot.
lvmetad is running
Created attachment 741478 [details]
Created attachment 741479 [details]
Created attachment 741480 [details]
Created attachment 741481 [details]
Created attachment 741482 [details]
Created attachment 741483 [details]
hdparm sda - sde
Try the pvscan with --cache and add -vvvv to watch what it is doing.
If still no luck, eliminate lvmetad: set use_lvmetad to 0 instead of 1 in lvm.conf and kill the daemon.
Created attachment 741500 [details]
(In reply to comment #10)
> Try the pvscan with --cache and add -vvvv to watch what it is doing.
Ok, that works.
the restore procedure now is
pvscan --cache -vvv /dev/md0
vgchange -ay vg_home
fsck.ext4 /dev/vg_home/lv_home -f
Pretty likely just another instance of Bug 952782 bug-family.
Disabling lvmetad is a workaround as confirmed here:
Shout please if not working for you.
My /home is not encrypted and I have lvm2 2.02.98-7.fc19.
So my report looks entirely different IMHO
(In reply to comment #14)
> Disabling lvmetad is a workaround as confirmed here:
> Shout please if not working for you.
I does work :-)
So you have a workaround, but you are using the latest package and so we still need to understand why it's not working out-of-the-box.
Powered on the system after being away for the weekend and *BOOM*, same problem :-(
Even though being on the latest lvm2 and having use_lvmetad = 0
Restore procedure as mentioned in comment 12 still works
Then ran updates, got kernel 3.10.rc0-nodebug, rebooted, and it works ok.
OK, we've nailed down the problem - we need to recognize the CHANGE uevent that reports the MD device coming from inactive to active state.
LVM2 volumes are now activated automatically as PVs appear in the system (very similar to MD's incremental mode). These volumes are autoactivated with the help of lvmetad that collects metadata information from newly appeared devices. Fedora 19 will use this new activation scheme by default.
With that LVM2 update, the LVM2 udev rules were modified a bit, so the autoactivation happens only on ADD events for any devices other than device-mapper ones (it was activated on ADD and CHANGE before the update). So for MD, we expected the device to be usable after ADD event as well, which needs to be fixed!
That update was necessary, as otherwise, the LVM2 volumes would be activated even after the CHANGE event that is triggered because of the WATCH udev rule (which happens on each close of the device opened for read-write) or any other spurious CHANGE event in general. So the LVM2 volumes were activated when they should not actually! This update fixed that.
However, the MD is special and we need to recognize the exact CHANGE event that makes the device usable/active from other CHANGE events. We need to add this hook to lvm2-lvmetad udev rules... I'll try to provide the update with the fix asap (this might probably require some coordination with md udev rules).
Changing the description of the problem to "activation" instead of "metadata loss" problem as metadata are not destroyed, only the LVM volumes are not activated if lvmetad is used (and so the automatic/event-based activation of LVM2 volumes).
lvm2-2.02.98-8.fc19 has been submitted as an update for Fedora 19.
I've done an update which should resolve this issue. If you still encounter a problem with MD+LVM with lvmetad, feel free to reopen this bug report. Thanks.
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing lvm2-2.02.98-8.fc19'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
installed the koji rpms and re-enabled use_lvmetad, a few reboots later and everything looks good.
fixed for me.
lvm2-2.02.98-8.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.