Bug 598886

Summary: LVM2: lvchange --refresh returns without error when pvs are missing
Product: Red Hat Enterprise Linux 5 Reporter: Ofer Schreiber <oschreib>
Component: lvm2Assignee: Milan Broz <mbroz>
Status: CLOSED ERRATA QA Contact: Corey Marthaler <cmarthal>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.5CC: abaron, agk, bazulay, cpelland, dwysocha, heinzm, iheim, jbrassow, joe.thornber, jturner, mbroz, prockai, pvrabec
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.56-12.el5 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-01-13 22:41:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 601078    

Description Ofer Schreiber 2010-06-02 09:45:09 UTC
Description of problem:
lvchange --refresh return code is 0 when some pvs are not available, even though we did not pass -p

Version-Release number of selected component (if applicable):
lvm2-2.02.56-8.el5_5.4

Additional info:
[root@white-vdsb vdsm]# lvchange --refresh --config " devices { preferred_names = [\"^/dev/mapper/\"] write_cache_state=0 filter = [ \"a%/dev/mapper/14f504e46494c4500444b764169322d6e485a662d6962596d|/dev/mapper/14f504e46494c45005076427150362d416b79452d4e4c355a|/dev/mapper/14f504e46494c4500524331396b412d65444b762d67304733|/dev/mapper/14f504e46494c45006a536b7079312d4231615a2d744d4734%\", \"r%.*%\" ] }" 52c06b7f-01fa-4a36-b49a-f65c3197a894/1974d431-7545-45da-af2b-de993a3a525d
  Couldn't find device with uuid hoHngo-RIcz-NqLm-cNCa-0TYT-sSXt-VVon3V.
  Couldn't find device with uuid hoHngo-RIcz-NqLm-cNCa-0TYT-sSXt-VVon3V.
[root@white-vdsb vdsm]# echo $?
0


root@white-vdsb vdsm]# lvs
  /dev/mapper/52c06b7f--01fa--4a36--b49a--f65c3197a894-1974d431--7545--45da--af2b--de993a3a525d-missing_2_0: read failed after 0 of 4096 at 0: Input/output error
  LV                                   VG                                   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  1974d431-7545-45da-af2b-de993a3a525d 52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-ao   3.00G                                      
  7a084870-9dfc-41c4-afaf-5c566e036360 52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-ao  15.00G                                      
  ids                                  52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-a- 128.00M                                      
  inbox                                52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-a- 128.00M                                      
  leases                               52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-a-   2.00G                                      
  master                               52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-a-   1.00G                                      
  metadata                             52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-a- 512.00M                                      
  outbox                               52c06b7f-01fa-4a36-b49a-f65c3197a894 -wi-a- 128.00M                                      
  root                                 vg0                                  -wi-ao  19.53G                                      
  swap                                 vg0                                  -wi-ao  19.53G                                      
  test                                 vg0                                  -wi-a-   1.00G                                      
[root@white-vdsb vdsm]# dmsetup table | grep 197
52c06b7f--01fa--4a36--b49a--f65c3197a894-1974d431--7545--45da--af2b--de993a3a525d: 0 2097152 linear 253:5 8331392
52c06b7f--01fa--4a36--b49a--f65c3197a894-1974d431--7545--45da--af2b--de993a3a525d: 2097152 3145728 linear 253:5 41885824
52c06b7f--01fa--4a36--b49a--f65c3197a894-1974d431--7545--45da--af2b--de993a3a525d: 5242880 1048576 linear 253:16 0
52c06b7f--01fa--4a36--b49a--f65c3197a894-inbox: 0 262144 linear 253:5 5709952
52c06b7f--01fa--4a36--b49a--f65c3197a894-leases: 0 4194304 linear 253:5 1253504
52c06b7f--01fa--4a36--b49a--f65c3197a894-7a084870--9dfc--41c4--afaf--5c566e036360: 0 31457280 linear 253:5 10428544
52c06b7f--01fa--4a36--b49a--f65c3197a894-ids: 0 262144 linear 253:5 5447808
52c06b7f--01fa--4a36--b49a--f65c3197a894-master: 0 2097152 linear 253:5 6234240
52c06b7f--01fa--4a36--b49a--f65c3197a894-outbox: 0 262144 linear 253:5 5972096
52c06b7f--01fa--4a36--b49a--f65c3197a894-1974d431--7545--45da--af2b--de993a3a525d-missing_2_0: 0 1048576 error 
52c06b7f--01fa--4a36--b49a--f65c3197a894-metadata: 0 1048576 linear 253:5 204928

Comment 2 Milan Broz 2010-06-02 14:30:15 UTC
Please can we have full debug log from the lvchange --refrech command? (add -vvvv)

Comment 4 Milan Broz 2010-06-04 10:13:43 UTC
Fixed upstream, for refreshing partial LVs it requires -P switch, otherwise it fails woth non-zero return code.

Comment 7 Milan Broz 2010-06-07 10:37:31 UTC
Fixed in lvm2-2.02.56-12.el5.

Comment 8 Corey Marthaler 2010-06-29 20:24:08 UTC
Is this all we need to verify, that the cmd returns non zero without the --partial and zero with the --partial?

[root@taft-01 tmp]# pvscan
  Couldn't find device with uuid mtlSmT-JNPp-2ro8-xEb2-XDDe-Vz3I-mdyJ76.
  PV /dev/sdb1        VG VG              lvm2 [135.66 GiB / 135.57 GiB free]
  PV unknown device   VG VG              lvm2 [135.66 GiB / 135.57 GiB free]
  PV /dev/sdd1        VG VG              lvm2 [135.66 GiB / 135.66 GiB free]
  PV /dev/sde1                           lvm2 [135.67 GiB]
  PV /dev/sdf1                           lvm2 [135.67 GiB]
  PV /dev/sdg1                           lvm2 [135.67 GiB]
  PV /dev/sdh1                           lvm2 [135.67 GiB]
  Total: 8 [1017.40 GiB] / in use: 4 [474.74 GiB] / in no VG: 4 [542.66 GiB]

[root@taft-01 tmp]# lvs -a -o +devices
  Couldn't find device with uuid mtlSmT-JNPp-2ro8-xEb2-XDDe-Vz3I-mdyJ76.
  LV                VG        Attr   LSize   Move Log         Copy%  Convert Devices
  mirror            VG        mwi-a- 100.00m      mirror_mlog 100.00         mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] VG        iwi-ao 100.00m                                 /dev/sdb1(0)
  [mirror_mimage_1] VG        iwi-ao 100.00m                                 unknown device(0)
  [mirror_mlog]     VG        lwi-ao   4.00m                                 /dev/sdd1(0)

[root@taft-01 tmp]# lvchange --refresh VG/mirror
  Couldn't find device with uuid mtlSmT-JNPp-2ro8-xEb2-XDDe-Vz3I-mdyJ76.
  Refusing refresh of partial LV mirror. Use --partial to override.
[root@taft-01 tmp]# echo $?
5

[root@taft-01 tmp]# lvchange --refresh --partial VG/mirror
  Partial mode. Incomplete volume groups will be activated read-only.
  Couldn't find device with uuid mtlSmT-JNPp-2ro8-xEb2-XDDe-Vz3I-mdyJ76.
[root@taft-01 tmp]# echo $?
0

Comment 9 Milan Broz 2010-06-29 20:37:31 UTC
With --partial it should really remap part of that LV to error target (or to missing filer specified in lvm.conf). This must not happen without --partial specified (see dmsetup table after the command is run).

Comment 11 Corey Marthaler 2010-11-09 00:09:50 UTC
Milan, in response to comment #9, you say the error target shouldn't exist w/o the --partial specified? but it does show up after running w/o --partial. 

Can you check the following output and verify if that's expected.

[root@grant-01 tmp]# lvs  -a -o +devices
  LV                VG         Attr   LSize   Log         Copy%  Devices
  mirror            grant      mwi-a- 500.00M mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] grant      iwi-ao 500.00M                    /dev/sdb1(0)
  [mirror_mimage_1] grant      iwi-ao 500.00M                    /dev/sdb2(0)
  [mirror_mlog]     grant      lwi-ao   4.00M                    /dev/sdc2(0)

[root@grant-01 tmp]# echo offline > /sys/block/sdc/device/state

[root@grant-01 tmp]# pvscan
  /dev/sdc2: read failed after 0 of 1024 at 58514538496: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 58514632704: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 0: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 4096: Input/output error
  /dev/sdc2: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid ZTAokb-74em-zwuH-dmfv-N2g7-QBma-JwQTbd.
  Couldn't find device with uuid aG8agq-id0n-nwB8-WT3f-pkm6-HVMN-i3Wlcj.
  PV /dev/sdb1        VG grant           lvm2 [40.87 GB / 40.38 GB free]
  PV /dev/sdb2        VG grant           lvm2 [40.87 GB / 40.38 GB free]
  PV unknown device   VG grant           lvm2 [54.49 GB / 54.49 GB free]
  PV unknown device   VG grant           lvm2 [54.49 GB / 54.49 GB free]

[root@grant-01 tmp]# lvs -a -o +devices
  /dev/sdc2: read failed after 0 of 2048 at 0: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 58514538496: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 58514632704: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 0: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 4096: Input/output error
  Couldn't find device with uuid ZTAokb-74em-zwuH-dmfv-N2g7-QBma-JwQTbd.
  Couldn't find device with uuid aG8agq-id0n-nwB8-WT3f-pkm6-HVMN-i3Wlcj.
  LV                VG         Attr   LSize   Log         Copy%  Devices
  mirror            grant      mwi-a- 500.00M mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] grant      iwi-ao 500.00M                    /dev/sdb1(0)
  [mirror_mimage_1] grant      iwi-ao 500.00M                    /dev/sdb2(0)
  [mirror_mlog]     grant      lwi-ao   4.00M                    unknown device(0)

[root@grant-01 tmp]# lvchange --refresh grant/mirror
  /dev/sdc2: read failed after 0 of 2048 at 0: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 58514538496: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 58514632704: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 0: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 4096: Input/output error
  Couldn't find device with uuid ZTAokb-74em-zwuH-dmfv-N2g7-QBma-JwQTbd.
  Couldn't find device with uuid aG8agq-id0n-nwB8-WT3f-pkm6-HVMN-i3Wlcj.
  Refusing refresh of partial LV mirror. Use --partial to override.
[root@grant-01 tmp]# echo $?
5

[root@grant-01 tmp]# dmsetup table
grant-mirror: 0 1024000 mirror disk 3 253:2 1024 block_on_error 2 253:3 0 253:4 0
grant-mirror_mlog: 0 8192 linear 8:34 384
grant-mirror_mimage_1: 0 1024000 linear 8:18 384
grant-mirror_mimage_0: 0 1024000 linear 8:17 384

[root@grant-01 tmp]# lvchange --refresh --partial grant/mirror
  Partial mode. Incomplete logical volumes will be processed.
  /dev/sdc2: read failed after 0 of 2048 at 0: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 58514538496: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 58514632704: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 0: Input/output error
  /dev/sdc2: read failed after 0 of 1024 at 4096: Input/output error
  Couldn't find device with uuid ZTAokb-74em-zwuH-dmfv-N2g7-QBma-JwQTbd.
  Couldn't find device with uuid aG8agq-id0n-nwB8-WT3f-pkm6-HVMN-i3Wlcj.
[root@grant-01 tmp]# echo $?
0

[root@grant-01 tmp]# dmsetup table
grant-mirror: 0 1024000 mirror core 2 1024 block_on_error 2 253:3 0 253:4 0
grant-mirror_mimage_1: 0 1024000 linear 8:18 384
grant-mirror_mimage_0: 0 1024000 linear 8:17 384

Comment 12 Milan Broz 2010-11-09 15:25:30 UTC
Yes, comment #11 seems to me to be correct:

- device with mirror log failed -> mirror LV is not in partial state -> lvchange --refresh is NOOP without -P flag, command returns error code

- with -P, the missing parts of LV are replaced with error taget, here it is mirror log, so code switch to core log instead (replacing log device with error target is nonsense) -> comand returns 0

Comment 13 Milan Broz 2010-11-09 15:26:25 UTC
(In reply to comment #12)
> - mirror LV _IS_ in partial state ...

Comment 14 Corey Marthaler 2010-11-09 17:21:16 UTC
Marking verified with the latest rpms.

2.6.18-227.el5

lvm2-2.02.74-1.el5    BUILT: Fri Oct 15 10:26:21 CDT 2010
lvm2-cluster-2.02.74-2.el5    BUILT: Fri Oct 29 07:48:11 CDT 2010
device-mapper-1.02.55-1.el5    BUILT: Fri Oct 15 06:15:55 CDT 2010
cmirror-1.1.39-10.el5    BUILT: Wed Sep  8 16:32:05 CDT 2010
kmod-cmirror-0.1.22-3.el5    BUILT: Tue Dec 22 13:39:47 CST 2009

Comment 16 errata-xmlrpc 2011-01-13 22:41:50 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0052.html