Description of problem: I've reproduced this a couple times now where after a cluster containing a fully synced cmirror is completely rebooted (due to one hang or another), that mirror will show up as 100% synced, yet have tmp images that are syncing as well. Once those tmp images are finished (up to 100%), they never leave. and if you deactivate and then reactivate that mirror, the sync restarts again at 0.00%. [root@taft-01 ~]# lvs -a -o +devices LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices mirror TAFT cwi-a- 25.00G 97.23 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0) [mirror_mimage_1] TAFT iwi-ao 25.00G /dev/sdb1(0) [mirror_mimage_2] TAFT iwi-ao 25.00G /dev/sdc1(0) [mirror_mimage_3] TAFT iwi-ao 25.00G /dev/sdd1(0) [mirror_mimage_4] TAFT Iwi-ao 25.00G /dev/sde1(0) [mirror_mimagetmp_4] TAFT mwi-ao 25.00G mirror_mlog 100.00 mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0) [mirror_mlog] TAFT lwi-ao 4.00M /dev/sdg1(0) LogVol00 VolGroup00 -wi-ao 58.38G /dev/sda2(0) LogVol01 VolGroup00 -wi-ao 9.75G /dev/sda2(1868) [root@taft-01 ~]# lvs -a -o +devices LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices mirror TAFT cwi-a- 25.00G 100.00 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0) [mirror_mimage_1] TAFT iwi-ao 25.00G /dev/sdb1(0) [mirror_mimage_2] TAFT iwi-ao 25.00G /dev/sdc1(0) [mirror_mimage_3] TAFT iwi-ao 25.00G /dev/sdd1(0) [mirror_mimage_4] TAFT iwi-ao 25.00G /dev/sde1(0) [mirror_mimagetmp_4] TAFT mwi-ao 25.00G mirror_mlog 100.00 mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0) [mirror_mlog] TAFT lwi-ao 4.00M /dev/sdg1(0) LogVol00 VolGroup00 -wi-ao 58.38G /dev/sda2(0) LogVol01 VolGroup00 -wi-ao 9.75G /dev/sda2(1868) [root@taft-01 ~]# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert mirror TAFT cwi-a- 25.00G 100.00 mirror_mimagetmp_4 LogVol00 VolGroup00 -wi-ao 58.38G LogVol01 VolGroup00 -wi-ao 9.75G [root@taft-01 ~]# vgchange -an TAFT 0 logical volume(s) in volume group "TAFT" now active [root@taft-01 ~]# vgchange -ay TAFT 1 logical volume(s) in volume group "TAFT" now active [root@taft-01 ~]# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert mirror TAFT cwi-a- 25.00G 0.28 mirror_mimagetmp_4 LogVol00 VolGroup00 -wi-ao 58.38G LogVol01 VolGroup00 -wi-ao 9.75G [root@taft-01 ~]# lvs -a -o +devices LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices mirror TAFT cwi-a- 25.00G 0.80 mirror_mimagetmp_4 mirror_mimagetmp_4(0),mirror_mimage_4(0) [mirror_mimage_1] TAFT iwi-ao 25.00G /dev/sdb1(0) [mirror_mimage_2] TAFT iwi-ao 25.00G /dev/sdc1(0) [mirror_mimage_3] TAFT iwi-ao 25.00G /dev/sdd1(0) [mirror_mimage_4] TAFT Iwi-ao 25.00G /dev/sde1(0) [mirror_mimagetmp_4] TAFT mwi-ao 25.00G mirror_mlog 100.00 mirror_mimage_1(0),mirror_mimage_2(0),mirror_mimage_3(0) [mirror_mlog] TAFT lwi-ao 4.00M /dev/sdg1(0) LogVol00 VolGroup00 -wi-ao 58.38G /dev/sda2(0) LogVol01 VolGroup00 -wi-ao 9.75G /dev/sda2(1868) Version-Release number of selected component (if applicable): 2.6.18-160.el5 lvm2-2.02.46-8.el5 BUILT: Thu Jun 18 08:06:12 CDT 2009 lvm2-cluster-2.02.46-8.el5 BUILT: Thu Jun 18 08:05:27 CDT 2009 cmirror-1.1.39-2.el5 BUILT: Mon Jul 27 15:39:05 CDT 2009 kmod-cmirror-0.1.22-1.el5 BUILT: Mon Jul 27 15:28:46 CDT 2009
This happens because the process monitoring the conversion gets killed off. The process should be completed by simply running 'lvconvert <mirror device>'. Since one of the layers uses a 'corelog', if the process dies and the mirrors are deactivated, there would be no way to tell upon activation if the mirror had been previously sync'ed. I suppose the way to fix this would be to have a disk log for each layer... but it still might be necessary to run 'lvconvert' - I'll have to test that.
I'm able to reproduce this issue during single machine failure testing in the latest rpms. After the mirror fully syncs, the tmp image is never removed. [root@taft-01 ~]# lvs -a -o +devices LV VG Attr LSize Log Copy% Convert Devices syncd_primary_3legs_1 helter_skelter cwi-ao 600.00M 100.00 syncd_primary_3legs_1_mimagetmp_3 syncd_primary_3legs_1_mimagetmp_3(0),syncd_primary_3legs_1_mimage_3(0) [syncd_primary_3legs_1_mimage_1] helter_skelter iwi-ao 600.00M /dev/sdd1(0) [syncd_primary_3legs_1_mimage_2] helter_skelter iwi-ao 600.00M /dev/sdh1(0) [syncd_primary_3legs_1_mimage_3] helter_skelter iwi-ao 600.00M /dev/sdg1(0) [syncd_primary_3legs_1_mimagetmp_3] helter_skelter mwi-ao 600.00M syncd_primary_3legs_1_mlog 100.00 syncd_primary_3legs_1_mimage_1(0),syncd_primary_3legs_1_mimage_2(0) [syncd_primary_3legs_1_mlog] helter_skelter lwi-ao 4.00M /dev/sde1(0) 2.6.18-160.el5 lvm2-2.02.56-2.el5 BUILT: Thu Dec 10 09:38:13 CST 2009 lvm2-cluster-2.02.56-2.el5 BUILT: Thu Dec 10 09:38:41 CST 2009 device-mapper-1.02.39-1.el5 BUILT: Wed Nov 11 12:31:44 CST 2009 cmirror-1.1.39-2.el5 BUILT: Mon Jul 27 15:39:05 CDT 2009 kmod-cmirror-0.1.22-1.el5 BUILT: Mon Jul 27 15:28:46 CDT 2009
I agree with Jon -- if a reboot intervenes, there is no particularly good way to finish off the conversion automatically. It should also be fairly harmless and is remedied by, as Jon pointed out, running lvconvert on the offending mirror.
Since it is too late to address this issue in RHEL 5.5, it has been proposed for RHEL 5.6. Contact your support representative if you need to escalate this issue.
I'll tentatively close this as not a bug, if people feel that a technical note is needed, please reopen and I'll write something up.