Bug 521973 - recovered mirror may never lose its tmp image
Summary: recovered mirror may never lose its tmp image
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: lvm2
Version: 5.4
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Petr Rockai
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-09-08 21:55 UTC by Corey Marthaler
Modified: 2010-11-09 12:59 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-10-13 13:59:44 UTC


Attachments (Terms of Use)

Description Corey Marthaler 2009-09-08 21:55:00 UTC
Description of problem:
I've reproduced this a couple times now where after a cluster containing a fully synced cmirror is completely rebooted (due to one hang or another), that mirror will show up as 100% synced, yet have tmp images that are syncing as well. Once those tmp images are finished (up to 100%), they never leave. and if you deactivate and then reactivate that mirror, the sync restarts again at 0.00%.

[root@taft-01 ~]# lvs -a -o +devices
  LV                   VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert            Devices                                                 
  mirror               TAFT       cwi-a- 25.00G                                 97.23 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0)                
  [mirror_mimage_1]    TAFT       iwi-ao 25.00G                                                          /dev/sdb1(0)                                            
  [mirror_mimage_2]    TAFT       iwi-ao 25.00G                                                          /dev/sdc1(0)                                            
  [mirror_mimage_3]    TAFT       iwi-ao 25.00G                                                          /dev/sdd1(0)                                            
  [mirror_mimage_4]    TAFT       Iwi-ao 25.00G                                                          /dev/sde1(0)                                            
  [mirror_mimagetmp_4] TAFT       mwi-ao 25.00G                    mirror_mlog 100.00                    mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0)
  [mirror_mlog]        TAFT       lwi-ao  4.00M                                                          /dev/sdg1(0)                                            
  LogVol00             VolGroup00 -wi-ao 58.38G                                                          /dev/sda2(0)                                            
  LogVol01             VolGroup00 -wi-ao  9.75G                                                          /dev/sda2(1868)                                         
[root@taft-01 ~]# lvs -a -o +devices
  LV                   VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert            Devices                                                 
  mirror               TAFT       cwi-a- 25.00G                                100.00 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0)                
  [mirror_mimage_1]    TAFT       iwi-ao 25.00G                                                          /dev/sdb1(0)                                            
  [mirror_mimage_2]    TAFT       iwi-ao 25.00G                                                          /dev/sdc1(0)                                            
  [mirror_mimage_3]    TAFT       iwi-ao 25.00G                                                          /dev/sdd1(0)                                            
  [mirror_mimage_4]    TAFT       iwi-ao 25.00G                                                          /dev/sde1(0)                                            
  [mirror_mimagetmp_4] TAFT       mwi-ao 25.00G                    mirror_mlog 100.00                    mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0)
  [mirror_mlog]        TAFT       lwi-ao  4.00M                                                          /dev/sdg1(0)                                            
  LogVol00             VolGroup00 -wi-ao 58.38G                                                          /dev/sda2(0)                                            
  LogVol01             VolGroup00 -wi-ao  9.75G                                                          /dev/sda2(1868)                                         
[root@taft-01 ~]# lvs
  LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert           
  mirror   TAFT       cwi-a- 25.00G                        100.00 mirror_mimagetmp_4
  LogVol00 VolGroup00 -wi-ao 58.38G                                                 
  LogVol01 VolGroup00 -wi-ao  9.75G                                                 
[root@taft-01 ~]# vgchange -an TAFT
  0 logical volume(s) in volume group "TAFT" now active
[root@taft-01 ~]# vgchange -ay TAFT
  1 logical volume(s) in volume group "TAFT" now active
[root@taft-01 ~]# lvs
  LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert           
  mirror   TAFT       cwi-a- 25.00G                          0.28 mirror_mimagetmp_4
  LogVol00 VolGroup00 -wi-ao 58.38G                                                 
  LogVol01 VolGroup00 -wi-ao  9.75G                                                 
[root@taft-01 ~]# lvs -a -o +devices
  LV                   VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert            Devices                                                 
  mirror               TAFT       cwi-a- 25.00G                                  0.80 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0)                
  [mirror_mimage_1]    TAFT       iwi-ao 25.00G                                                          /dev/sdb1(0)                                            
  [mirror_mimage_2]    TAFT       iwi-ao 25.00G                                                          /dev/sdc1(0)                                            
  [mirror_mimage_3]    TAFT       iwi-ao 25.00G                                                          /dev/sdd1(0)                                            
  [mirror_mimage_4]    TAFT       Iwi-ao 25.00G                                                          /dev/sde1(0)                                            
  [mirror_mimagetmp_4] TAFT       mwi-ao 25.00G                    mirror_mlog 100.00                    mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0)
  [mirror_mlog]        TAFT       lwi-ao  4.00M                                                          /dev/sdg1(0)                                            
  LogVol00             VolGroup00 -wi-ao 58.38G                                                          /dev/sda2(0)                                            
  LogVol01             VolGroup00 -wi-ao  9.75G                                                          /dev/sda2(1868)                                         


Version-Release number of selected component (if applicable):
2.6.18-160.el5

lvm2-2.02.46-8.el5    BUILT: Thu Jun 18 08:06:12 CDT 2009
lvm2-cluster-2.02.46-8.el5    BUILT: Thu Jun 18 08:05:27 CDT 2009
cmirror-1.1.39-2.el5    BUILT: Mon Jul 27 15:39:05 CDT 2009
kmod-cmirror-0.1.22-1.el5    BUILT: Mon Jul 27 15:28:46 CDT 2009

Comment 1 Jonathan Earl Brassow 2009-10-14 15:23:30 UTC
This happens because the process monitoring the conversion gets killed off.  The process should be completed by simply running 'lvconvert <mirror device>'.  Since one of the layers uses a 'corelog', if the process dies and the mirrors are deactivated, there would be no way to tell upon activation if the mirror had been previously sync'ed.

I suppose the way to fix this would be to have a disk log for each layer... but it still might be necessary to run 'lvconvert' - I'll have to test that.

Comment 2 Corey Marthaler 2009-12-10 21:52:15 UTC
I'm able to reproduce this issue during single machine failure testing in the latest rpms. After the mirror fully syncs, the tmp image is never removed.

[root@taft-01 ~]# lvs -a -o +devices
  LV                                  VG             Attr   LSize    Log                        Copy%  Convert                           Devices  syncd_primary_3legs_1               helter_skelter cwi-ao 600.00M                             100.00 syncd_primary_3legs_1_mimagetmp_3 syncd_primary_3legs_1_mimagetmp_3(0),syncd_primary_3legs_1_mimage_3(0)
  [syncd_primary_3legs_1_mimage_1]    helter_skelter iwi-ao 600.00M                                                                      /dev/sdd1(0)  [syncd_primary_3legs_1_mimage_2]    helter_skelter iwi-ao 600.00M                                                                      /dev/sdh1(0)  [syncd_primary_3legs_1_mimage_3]    helter_skelter iwi-ao 600.00M                                                                      /dev/sdg1(0)  [syncd_primary_3legs_1_mimagetmp_3] helter_skelter mwi-ao 600.00M  syncd_primary_3legs_1_mlog 100.00                                   syncd_primary_3legs_1_mimage_1(0),syncd_primary_3legs_1_mimage_2(0)
  [syncd_primary_3legs_1_mlog]        helter_skelter lwi-ao   4.00M                                                                      /dev/sde1(0)


2.6.18-160.el5

lvm2-2.02.56-2.el5    BUILT: Thu Dec 10 09:38:13 CST 2009
lvm2-cluster-2.02.56-2.el5    BUILT: Thu Dec 10 09:38:41 CST 2009
device-mapper-1.02.39-1.el5    BUILT: Wed Nov 11 12:31:44 CST 2009
cmirror-1.1.39-2.el5    BUILT: Mon Jul 27 15:39:05 CDT 2009
kmod-cmirror-0.1.22-1.el5    BUILT: Mon Jul 27 15:28:46 CDT 2009

Comment 3 Petr Rockai 2009-12-17 10:11:29 UTC
I agree with Jon -- if a reboot intervenes, there is no particularly good way to finish off the conversion automatically. It should also be fairly harmless and is remedied by, as Jon pointed out, running lvconvert on the offending mirror.

Comment 6 Ludek Smid 2010-03-11 12:20:54 UTC
Since it is too late to address this issue in RHEL 5.5, it has been proposed for RHEL 5.6.  Contact your support representative if you need to escalate this issue.

Comment 10 Petr Rockai 2010-10-13 13:59:44 UTC
I'll tentatively close this as not a bug, if people feel that a technical note is needed, please reopen and I'll write something up.


Note You need to log in before you can comment on or make changes to this bug.