Bug 521973

Summary: recovered mirror may never lose its tmp image
Product: Red Hat Enterprise Linux 5 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Petr Rockai <prockai>
Status: CLOSED NOTABUG QA Contact: Cluster QE <mspqa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.4CC: agk, ccaulfie, dwysocha, heinzm, iannis, jbrassow, mbroz, prockai
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-13 13:59:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2009-09-08 21:55:00 UTC
Description of problem:
I've reproduced this a couple times now where after a cluster containing a fully synced cmirror is completely rebooted (due to one hang or another), that mirror will show up as 100% synced, yet have tmp images that are syncing as well. Once those tmp images are finished (up to 100%), they never leave. and if you deactivate and then reactivate that mirror, the sync restarts again at 0.00%.

[root@taft-01 ~]# lvs -a -o +devices
  LV                   VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert            Devices                                                 
  mirror               TAFT       cwi-a- 25.00G                                 97.23 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0)                
  [mirror_mimage_1]    TAFT       iwi-ao 25.00G                                                          /dev/sdb1(0)                                            
  [mirror_mimage_2]    TAFT       iwi-ao 25.00G                                                          /dev/sdc1(0)                                            
  [mirror_mimage_3]    TAFT       iwi-ao 25.00G                                                          /dev/sdd1(0)                                            
  [mirror_mimage_4]    TAFT       Iwi-ao 25.00G                                                          /dev/sde1(0)                                            
  [mirror_mimagetmp_4] TAFT       mwi-ao 25.00G                    mirror_mlog 100.00                    mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0)
  [mirror_mlog]        TAFT       lwi-ao  4.00M                                                          /dev/sdg1(0)                                            
  LogVol00             VolGroup00 -wi-ao 58.38G                                                          /dev/sda2(0)                                            
  LogVol01             VolGroup00 -wi-ao  9.75G                                                          /dev/sda2(1868)                                         
[root@taft-01 ~]# lvs -a -o +devices
  LV                   VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert            Devices                                                 
  mirror               TAFT       cwi-a- 25.00G                                100.00 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0)                
  [mirror_mimage_1]    TAFT       iwi-ao 25.00G                                                          /dev/sdb1(0)                                            
  [mirror_mimage_2]    TAFT       iwi-ao 25.00G                                                          /dev/sdc1(0)                                            
  [mirror_mimage_3]    TAFT       iwi-ao 25.00G                                                          /dev/sdd1(0)                                            
  [mirror_mimage_4]    TAFT       iwi-ao 25.00G                                                          /dev/sde1(0)                                            
  [mirror_mimagetmp_4] TAFT       mwi-ao 25.00G                    mirror_mlog 100.00                    mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0)
  [mirror_mlog]        TAFT       lwi-ao  4.00M                                                          /dev/sdg1(0)                                            
  LogVol00             VolGroup00 -wi-ao 58.38G                                                          /dev/sda2(0)                                            
  LogVol01             VolGroup00 -wi-ao  9.75G                                                          /dev/sda2(1868)                                         
[root@taft-01 ~]# lvs
  LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert           
  mirror   TAFT       cwi-a- 25.00G                        100.00 mirror_mimagetmp_4
  LogVol00 VolGroup00 -wi-ao 58.38G                                                 
  LogVol01 VolGroup00 -wi-ao  9.75G                                                 
[root@taft-01 ~]# vgchange -an TAFT
  0 logical volume(s) in volume group "TAFT" now active
[root@taft-01 ~]# vgchange -ay TAFT
  1 logical volume(s) in volume group "TAFT" now active
[root@taft-01 ~]# lvs
  LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert           
  mirror   TAFT       cwi-a- 25.00G                          0.28 mirror_mimagetmp_4
  LogVol00 VolGroup00 -wi-ao 58.38G                                                 
  LogVol01 VolGroup00 -wi-ao  9.75G                                                 
[root@taft-01 ~]# lvs -a -o +devices
  LV                   VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert            Devices                                                 
  mirror               TAFT       cwi-a- 25.00G                                  0.80 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0)                
  [mirror_mimage_1]    TAFT       iwi-ao 25.00G                                                          /dev/sdb1(0)                                            
  [mirror_mimage_2]    TAFT       iwi-ao 25.00G                                                          /dev/sdc1(0)                                            
  [mirror_mimage_3]    TAFT       iwi-ao 25.00G                                                          /dev/sdd1(0)                                            
  [mirror_mimage_4]    TAFT       Iwi-ao 25.00G                                                          /dev/sde1(0)                                            
  [mirror_mimagetmp_4] TAFT       mwi-ao 25.00G                    mirror_mlog 100.00                    mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0)
  [mirror_mlog]        TAFT       lwi-ao  4.00M                                                          /dev/sdg1(0)                                            
  LogVol00             VolGroup00 -wi-ao 58.38G                                                          /dev/sda2(0)                                            
  LogVol01             VolGroup00 -wi-ao  9.75G                                                          /dev/sda2(1868)                                         


Version-Release number of selected component (if applicable):
2.6.18-160.el5

lvm2-2.02.46-8.el5    BUILT: Thu Jun 18 08:06:12 CDT 2009
lvm2-cluster-2.02.46-8.el5    BUILT: Thu Jun 18 08:05:27 CDT 2009
cmirror-1.1.39-2.el5    BUILT: Mon Jul 27 15:39:05 CDT 2009
kmod-cmirror-0.1.22-1.el5    BUILT: Mon Jul 27 15:28:46 CDT 2009

Comment 1 Jonathan Earl Brassow 2009-10-14 15:23:30 UTC
This happens because the process monitoring the conversion gets killed off.  The process should be completed by simply running 'lvconvert <mirror device>'.  Since one of the layers uses a 'corelog', if the process dies and the mirrors are deactivated, there would be no way to tell upon activation if the mirror had been previously sync'ed.

I suppose the way to fix this would be to have a disk log for each layer... but it still might be necessary to run 'lvconvert' - I'll have to test that.

Comment 2 Corey Marthaler 2009-12-10 21:52:15 UTC
I'm able to reproduce this issue during single machine failure testing in the latest rpms. After the mirror fully syncs, the tmp image is never removed.

[root@taft-01 ~]# lvs -a -o +devices
  LV                                  VG             Attr   LSize    Log                        Copy%  Convert                           Devices  syncd_primary_3legs_1               helter_skelter cwi-ao 600.00M                             100.00 syncd_primary_3legs_1_mimagetmp_3 syncd_primary_3legs_1_mimagetmp_3(0),syncd_primary_3legs_1_mimage_3(0)
  [syncd_primary_3legs_1_mimage_1]    helter_skelter iwi-ao 600.00M                                                                      /dev/sdd1(0)  [syncd_primary_3legs_1_mimage_2]    helter_skelter iwi-ao 600.00M                                                                      /dev/sdh1(0)  [syncd_primary_3legs_1_mimage_3]    helter_skelter iwi-ao 600.00M                                                                      /dev/sdg1(0)  [syncd_primary_3legs_1_mimagetmp_3] helter_skelter mwi-ao 600.00M  syncd_primary_3legs_1_mlog 100.00                                   syncd_primary_3legs_1_mimage_1(0),syncd_primary_3legs_1_mimage_2(0)
  [syncd_primary_3legs_1_mlog]        helter_skelter lwi-ao   4.00M                                                                      /dev/sde1(0)


2.6.18-160.el5

lvm2-2.02.56-2.el5    BUILT: Thu Dec 10 09:38:13 CST 2009
lvm2-cluster-2.02.56-2.el5    BUILT: Thu Dec 10 09:38:41 CST 2009
device-mapper-1.02.39-1.el5    BUILT: Wed Nov 11 12:31:44 CST 2009
cmirror-1.1.39-2.el5    BUILT: Mon Jul 27 15:39:05 CDT 2009
kmod-cmirror-0.1.22-1.el5    BUILT: Mon Jul 27 15:28:46 CDT 2009

Comment 3 Petr Rockai 2009-12-17 10:11:29 UTC
I agree with Jon -- if a reboot intervenes, there is no particularly good way to finish off the conversion automatically. It should also be fairly harmless and is remedied by, as Jon pointed out, running lvconvert on the offending mirror.

Comment 6 Ludek Smid 2010-03-11 12:20:54 UTC
Since it is too late to address this issue in RHEL 5.5, it has been proposed for RHEL 5.6.  Contact your support representative if you need to escalate this issue.

Comment 10 Petr Rockai 2010-10-13 13:59:44 UTC
I'll tentatively close this as not a bug, if people feel that a technical note is needed, please reopen and I'll write something up.