Bug 464933

Summary:	pvmove is broken in latest 5.3
Product:	Red Hat Enterprise Linux 5	Reporter:	Corey Marthaler <cmarthal>
Component:	lvm2-cluster	Assignee:	Milan Broz <mbroz>
Status:	CLOSED DUPLICATE	QA Contact:	Cluster QE <mspqa-list>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.3	CC:	agk, ccaulfie, dwysocha, edamato, heinzm, jbrassow, mbroz, prockai, pvrabec
Target Milestone:	rc	Keywords:	Regression
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2008-10-06 20:18:50 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Corey Marthaler 2008-10-01 14:43:32 UTC

Description of problem:
I noticed that pvmove was either stuck or really slow, so I let one run over night and found that it's broke. This may be a related to or a result of bz 464481. It stayed stuck all night, never getting beyond 34.22% done.

Test Output:

creating lvm devices...
create PV(s) out of all avail partitions on grant-01
create VG(s) on grant-01
         pv_shuffle

creating linear on grant-01
creating stripe on grant-01

Iteration 1 of 10000 started on grant-03 at Tue Sep 30 17:06:26 CDT 2008
FREE PVS: /dev/sdb2 /dev/sdb3 /dev/sdb4
INUSE PVS: /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4
NOT INUSE PVS: /dev/sdb1

Adding /dev/sdb2 to the volume group on grant-03
Moving data from /dev/sdc1 to /dev/sdb2 on grant-03
    Finding volume group "pv_shuffle"
    Executing: /sbin/modprobe dm-log-clustered
    Archiving volume group "pv_shuffle" metadata (seqno 4).
    Creating logical volume pvmove0
    Moving 1280 extents of logical volume pv_shuffle/linear
    Updating volume group metadata
    Creating volume group backup "/etc/lvm/backup/pv_shuffle" (seqno 5).
    Checking progress every 15 seconds


Wed Oct  1 09:38:44 CDT 2008
  LV        VG         Attr   LSize  Origin Snap%  Move      Log Copy%  Convert Devices                               
  LogVol00  VolGroup00 -wi-ao 72.44G                                            /dev/sda2(0)                          
  LogVol01  VolGroup00 -wi-ao  1.94G                                            /dev/sda2(2318)                       
  linear    pv_shuffle -wI-a-  5.00G                                            pvmove0(0)                            
  [pvmove0] pv_shuffle p-C-ao  5.00G               /dev/sdc1      34.22         /dev/sdc1(0),/dev/sdb2(0)             
  stripe    pv_shuffle -wi---  5.00G                                            /dev/sdc2(0),/dev/sdc3(0),/dev/sdc4(0)


Version-Release number of selected component (if applicable):
2.6.18-116.el5

lvm2-2.02.40-3.el5    BUILT: Thu Sep 25 14:59:07 CDT 2008
lvm2-cluster-2.02.40-3.el5    BUILT: Thu Sep 25 15:00:54 CDT 2008
device-mapper-1.02.28-2.el5    BUILT: Fri Sep 19 02:50:32 CDT 2008
cmirror-1.1.28-1.el5    BUILT: Tue Sep 30 15:48:54 CDT 2008
kmod-cmirror-0.1.18-1.el5    BUILT: Mon Sep 29 16:20:21 CDT 2008


How reproducible:
Everytime

Comment 1 Corey Marthaler 2008-10-01 15:08:22 UTC

This same test works over and over in single machine lvm.

creating lvm devices...
create PV(s) out of all avail partitions on grant-03
create VG(s) on grant-03                            
         pv_shuffle                                 

creating linear on grant-03
creating stripe on grant-03

Iteration 1 of 10000 started on grant-03 at Wed Oct  1 10:04:43 CDT 2008
FREE PVS: /dev/sdb2 /dev/sdb3 /dev/sdb4                                 
INUSE PVS: /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4                      
NOT INUSE PVS: /dev/sdb1                                                

Adding /dev/sdb2 to the volume group on grant-03
Moving data from /dev/sdc2 to /dev/sdb2 on grant-03
    Finding volume group "pv_shuffle"              
    Archiving volume group "pv_shuffle" metadata (seqno 4).
    Creating logical volume pvmove0                        
    Moving 427 extents of logical volume pv_shuffle/stripe 
    Found volume group "pv_shuffle"                        
    Creating pv_shuffle-stripe                             
    Loading pv_shuffle-stripe table                        
    Resuming pv_shuffle-stripe (253:2)                     
    Updating volume group metadata                         
    Creating volume group backup "/etc/lvm/backup/pv_shuffle" (seqno 5).
    Found volume group "pv_shuffle"                                     
    Found volume group "pv_shuffle"                                     
    Suspending pv_shuffle-stripe (253:2) with device flush              
    Found volume group "pv_shuffle"                                     
    Creating pv_shuffle-pvmove0                                         
    Loading pv_shuffle-pvmove0 table                                    
    Resuming pv_shuffle-pvmove0 (253:3)                                 
    Found volume group "pv_shuffle"                                     
    Loading pv_shuffle-pvmove0 table                                    
    Suppressed pv_shuffle-pvmove0 identical table reload.               
    Loading pv_shuffle-stripe table                                     
    Resuming pv_shuffle-stripe (253:2)                                  
    Checking progress every 15 seconds                                  
    Found volume group "pv_shuffle"                                     
    Found volume group "pv_shuffle"                                     
    Loading pv_shuffle-stripe table                                     
    Suspending pv_shuffle-stripe (253:2) with device flush              
    Suspending pv_shuffle-pvmove0 (253:3) with device flush             
    Found volume group "pv_shuffle"                                     
    Found volume group "pv_shuffle"                                     
    Found volume group "pv_shuffle"                                     
    Resuming pv_shuffle-pvmove0 (253:3)                                 
    Found volume group "pv_shuffle"                                     
    Resuming pv_shuffle-stripe (253:2)                                  
    Found volume group "pv_shuffle"                                     
    Removing pv_shuffle-pvmove0 (253:3)                                 
    Found volume group "pv_shuffle"                                     
    Removing temporary pvmove LV                                        
    Writing out final volume group after pvmove                         
    Creating volume group backup "/etc/lvm/backup/pv_shuffle" (seqno 7).
Removing /dev/sdb1 from the volume group on grant-03

Comment 2 Milan Broz 2008-10-06 10:40:14 UTC

Is it still lproblem with the latest cmirror builds?
It seems to me like problem with cmirror status update and not base lvm2.

Please attach output of pvmove ... -vvvv to check what's happening when pvmove repeatedly check for status.

Comment 3 Corey Marthaler 2008-10-06 20:10:38 UTC

This is fixed with the latest cmirror build cmirror-1.1.29-1.el5 which means it was related to bz 464550.

Comment 4 Milan Broz 2008-10-06 20:18:50 UTC

Great, I'll mark this as duplicate of bug 464550 to avoid requesting of all these acks needed for attaching to errata bug list. Anyway the core bug wasn't in lvm2 code.

Please reopen if you hit it again.

*** This bug has been marked as a duplicate of bug 464550 ***