Bug 995440
Summary: | lvmetad unable to update metadata after failed PV has returned to mirror. | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Nenad Peric <nperic> |
Component: | lvm2 | Assignee: | Petr Rockai <prockai> |
Status: | CLOSED ERRATA | QA Contact: | Cluster QE <mspqa-list> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 6.5 | CC: | agk, cmarthal, dwysocha, heinzm, jbrassow, msnitzer, prajnoha, prockai, thornber, tlavigne, zkabelac |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | lvm2-2.02.100-5.el6 | Doc Type: | Bug Fix |
Doc Text: |
Cause: Repair of inconsistent metadata used a different code path depending on whether lvmetad was running and enabled.
Consequence: The lvmetad version of metadata repair failed to actually correct the metadata and a warning was printed repeatedly by every command (until the problem was manually fixed).
Fix: The code paths have been reconciled.
Result: Metadata inconsistencies are automatically repaired as appropriate, whether lvmetad is enabled or not.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2013-11-21 23:26:47 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Nenad Peric
2013-08-09 11:27:11 UTC
I can still reproduce this easily by running revolution_9 test: revolution_9 -i 5 -o virt-012 -e kill_random_legs,kill_random_devices it is 100% reproducible for me (lvmetad on) [root@virt-012 ~]# vgs PV LpymAu-GeS1-jD9f-OHgn-keyq-kPcv-Jj1qO8 not recognised. Is the device missing? Missing device /dev/sdj1 reappeared, updating metadata for VG revolution_9 to version 55. Missing device /dev/sdb1 reappeared, updating metadata for VG revolution_9 to version 55. PV LpymAu-GeS1-jD9f-OHgn-keyq-kPcv-Jj1qO8 not recognised. Is the device missing? Missing device /dev/sdj1 reappeared, updating metadata for VG revolution_9 to version 55. Missing device /dev/sdb1 reappeared, updating metadata for VG revolution_9 to version 55. VG #PV #LV #SN Attr VSize VFree revolution_9 6 1 0 wz-pn- 59.95g 55.95g vg_virt012 1 2 0 wz--n- 7.51g 0 [root@virt-012 ~]# lvs PV LpymAu-GeS1-jD9f-OHgn-keyq-kPcv-Jj1qO8 not recognised. Is the device missing? Missing device /dev/sdj1 reappeared, updating metadata for VG revolution_9 to version 55. Missing device /dev/sdb1 reappeared, updating metadata for VG revolution_9 to version 55. PV LpymAu-GeS1-jD9f-OHgn-keyq-kPcv-Jj1qO8 not recognised. Is the device missing? Missing device /dev/sdj1 reappeared, updating metadata for VG revolution_9 to version 55. Missing device /dev/sdb1 reappeared, updating metadata for VG revolution_9 to version 55. LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert mirror_1 revolution_9 mwi-aom--- 2.00g mirror_1_mlog 100.00 lv_root vg_virt012 -wi-ao---- 6.71g lv_swap vg_virt012 -wi-ao---- 816.00m ================================================================================ ================================================================================ An easy way to reproduce WITHOUT this revolution test would be: Turn off lvmetad and set use_lvmetad to 0. Have these set in lvm.conf: mirror_log_fault_policy="allocate" mirror_image_fault_policy="allocate" mirror_segtype_default="mirror" Create a VG, and a mirrored LV, wait for sync, then fail a device: - vgcreate newvg /dev/sd{a..f}1 - lvcreate -m1 -L3G -n mirror newvg - echo 1 >/sys/block/sda/device/delete do some I/O to force repair and replacement of the device wait for the sync to finish 9so that LVM is not doing anything on the LV anymore) When device is replaced and sync is done, turn on lvmetad by changing use_lvmetad to 1 in lvm.conf and starting lvm2-lvmetad daemon (/etc/init.d/lvm2-lcmetad start). Return the failing device by rescanning the scsi bus for example: echo "- - -" >/sys/class/scsi_host/host6/scan Now try any lvm command, here are the results: [root@virt-012 ~]# lvs vg Missing device /dev/sda1 reappeared, updating metadata for VG newvg to version 12. Missing device /dev/sda1 reappeared, updating metadata for VG newvg to version 12. LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert mirror newvg mwi-a-m--- 3.00g mirror_mlog 100.00 lv_root vg_virt012 -wi-ao---- 6.71g lv_swap vg_virt012 -wi-ao---- 816.00m [root@virt-012 ~]# vgs Missing device /dev/sda1 reappeared, updating metadata for VG newvg to version 12. Missing device /dev/sda1 reappeared, updating metadata for VG newvg to version 12. VG #PV #LV #SN Attr VSize VFree newvg 6 1 0 wz--n- 59.95g 53.95g vg_virt012 1 2 0 wz--n- 7.51g 0 [root@virt-012 ~]# pvscan --cache WARNING: Inconsistent metadata found for VG newvg [root@virt-012 ~]# vgs Missing device /dev/sda1 reappeared, updating metadata for VG newvg to version 12. Missing device /dev/sda1 reappeared, updating metadata for VG newvg to version 12. VG #PV #LV #SN Attr VSize VFree newvg 6 1 0 wz--n- 59.95g 53.95g vg_virt012 1 2 0 wz--n- 7.51g 0 The number just stays the same. The problem here seems to be with lvmetad updating its cache. I've upstreamed test case for internal lvm test suite for such case: https://www.redhat.com/archives/lvm-devel/2013-October/msg00021.html Fix for lvmetad case needs to be added. This should be fixed upstream in 0decd7553ac9dcf4a7d81f5b10b1f4ca053ae9a5. (We have cleared up the matter about lvconvert --repair: without lvmetad, it accidentally repairs the metadata even though it's not supposed to do that. It's not a major issue, although it might be surprising that PVs are re-integrated from by dmeventd. With "normal" commands, things now work as expected both with and without lvmetad.) The messages no longer repeat and metadata is updated as it should be. [root@virt-011 ~]# vgs PV SRfdXr-cr5q-1lgz-UFi3-7uRd-dJhd-vcKszr not recognised. Is the device missing? PV SRfdXr-cr5q-1lgz-UFi3-7uRd-dJhd-vcKszr not recognised. Is the device missing? VG #PV #LV #SN Attr VSize VFree newvg 4 1 0 wz-pn- 39.97g 33.96g vg_virt011 1 2 0 wz--n- 7.51g 0 [root@virt-011 ~]# echo "- - -" >/sys/class/scsi_host/host9/scan [root@virt-011 ~]# lvs -a Missing device /dev/sdb1 reappeared, updating metadata for VG newvg to version 10. LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert mirror newvg mwi-a-m--- 3.00g mirror_mlog 100.00 [mirror_mimage_0] newvg iwi-aom--- 3.00g [mirror_mimage_1] newvg iwi-aom--- 3.00g [mirror_mlog] newvg lwi-aom--- 4.00m lv_root vg_virt011 -wi-ao---- 6.71g lv_swap vg_virt011 -wi-ao---- 816.00m [root@virt-011 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert mirror newvg mwi-a-m--- 3.00g mirror_mlog 100.00 [mirror_mimage_0] newvg iwi-aom--- 3.00g [mirror_mimage_1] newvg iwi-aom--- 3.00g [mirror_mlog] newvg lwi-aom--- 4.00m lv_root vg_virt011 -wi-ao---- 6.71g lv_swap vg_virt011 -wi-ao---- 816.00m marking VERIFIED with: lvm2-2.02.100-5.el6.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1704.html |