Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1535012

Summary: pvmove fails with non-exclusive lvs in clvmd cluster
Product: Red Hat Enterprise Linux 7 Reporter: Roman Bednář <rbednar>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
lvm2 sub component: Command-line tools QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: agk, cmarthal, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, prockai, rhandlin, thornber, zkabelac
Version: 7.5Keywords: Regression, TestBlocker
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.177-4.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 15:23:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
pvmove_vvvv none

Description Roman Bednář 2018-01-16 13:15:24 UTC 
Created attachment 1382001 [details]
pvmove_vvvv

pvmove attempt with non-exclusively active lvs does no longer work. Also there is a separate bug for pvmove not working without background (-b) flag (BZ#1476408)
Attaching verbose output of pvmove attempt.

See also BZ#1476408 but this BZ should track a different issue.

Odd are this is not a supported feature, although it seems functional in RHEL7.4 thus adding regression flag.

Reproducible always.



# lvs -a -o +devices
  LV   VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root rhel_virt-170 -wi-ao----  <6.20g                                                     /dev/vda2(205)
  swap rhel_virt-170 -wi-ao---- 820.00m                                                     /dev/vda2(0)  
  lv   vg            -wi-a-----   1.00g                                                     /dev/sda1(0)  
  lv2  vg            -wi-a-----   1.00g                                                     /dev/sda1(256)
  lv3  vg            -wi-a-----   1.00g                                                     /dev/sda1(512)

# pvmove -v -b /dev/sda1 /dev/sdb1
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "vg" metadata (seqno 16).
    Creating logical volume pvmove0
    Moving 256 extents of logical volume vg/lv
    Moving 256 extents of logical volume vg/lv2
    Moving 256 extents of logical volume vg/lv3
  Increasing mirror region size from 0    to 2.00 KiB
    Setting up pvmove in on-disk volume group metadata.
    Creating volume group backup "/etc/lvm/backup/vg" (seqno 17).
    Checking progress before waiting every 15 seconds.

# echo $?
0

# lvs -a -o +devices
  LV   VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root rhel_virt-170 -wi-ao----  <6.20g                                                     /dev/vda2(205)
  swap rhel_virt-170 -wi-ao---- 820.00m                                                     /dev/vda2(0)  
  lv   vg            -wi-a-----   1.00g                                                     /dev/sdb1(0)  
  lv2  vg            -wi-a-----   1.00g                                                     /dev/sdb1(256)
  lv3  vg            -wi-a-----   1.00g                                                     /dev/sdb1(512)

=================================
RHEL 7.5:

# lvs -a -o +devices
  LV   VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root rhel_virt-387 -wi-ao----  <6.20g                                                     /dev/vda2(205)
  swap rhel_virt-387 -wi-ao---- 820.00m                                                     /dev/vda2(0)  
  lv1  vg            -wi-a-----   1.00g                                                     /dev/sda1(0)  
  lv2  vg            -wi-a-----   1.00g                                                     /dev/sda1(256)
  lv3  vg            -wi-a-----   1.00g                                                     /dev/sda1(512)

# pvmove -v -b /dev/sda1 /dev/sdb1
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "vg" metadata (seqno 4).
    Creating logical volume pvmove0
    Moving 256 extents of logical volume vg/lv1.
    Moving 256 extents of logical volume vg/lv2.
    Moving 256 extents of logical volume vg/lv3.
  Increasing mirror region size from 0    to 2.00 KiB
  Error locking on node 1: Device or resource busy
  Failed to activate vg/lv1

# echo $?
5

# lvs -a -o +devices
  LV   VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root rhel_virt-387 -wi-ao----  <6.20g                                                     /dev/vda2(205)
  swap rhel_virt-387 -wi-ao---- 820.00m                                                     /dev/vda2(0)  
  lv1  vg            -wi-a-----   1.00g                                                     /dev/sda1(0)  
  lv2  vg            -wi-a-----   1.00g                                                     /dev/sda1(256)
  lv3  vg            -wi-a-----   1.00g                                                     /dev/sda1(512)



3.10.0-826.el7.x86_64

lvm2-2.02.176-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
lvm2-libs-2.02.176-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
lvm2-cluster-2.02.176-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
lvm2-python-boom-0.8.1-5.el7    BUILT: Wed Dec  6 11:15:40 CET 2017
cmirror-2.02.176-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
device-mapper-1.02.145-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
device-mapper-libs-1.02.145-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
device-mapper-event-1.02.145-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
device-mapper-event-libs-1.02.145-5.el7    BUILT: Wed Dec  6 11:13:07 CET 2017
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 12:07:18 CET 2017
vdo-6.1.0.106-13    BUILT: Thu Dec 21 16:00:07 CET 2017
kmod-kvdo-6.1.0.106-11.el7    BUILT: Thu Dec 21 17:09:12 CET 2017
Comment 4 Zdenek Kabelac 2018-01-30 10:46:06 UTC 
These issues are already fixed  (assuming minor fixes are included in last build for activation during conversion)

In basic logic - lvm2 cannot PVMOVE lvs active on multiple nodes - thus LVs that are part of pvmove must be always active exclusively.

There are 2 issues:

1.) lvm2 does not support 'lock upgrade' - so whenever user has just 'local' activation - it has to be deactivated and activated locally exclusively.


2.) lvm2 does not support 'clustered' mirroring with pvmove (which would in fact required rather obscure cmirrord)


Thus from lvm2 perspective existing behavior is way more correct to the previous state - so it's not a regression rather a bugfix.

It's unclear whether scheduling support for those 2 issue is worth to work on ATM.
Comment 5 Jonathan Earl Brassow 2018-01-30 13:36:18 UTC 
> 2.) lvm2 does not support 'clustered' mirroring with pvmove (which would in
> fact required rather obscure cmirrord)

pretty sure that cmirrord /was/ used in pvmove in a cluster.  You sure about that?
Comment 6 Jonathan Earl Brassow 2018-01-30 16:23:05 UTC 
moving back to ASSIGNED.  I've tested this on 7.4 and found it working just fine.
Comment 8 Zdenek Kabelac 2018-02-01 22:39:37 UTC 
Two patches try to address the issue:

https://www.redhat.com/archives/lvm-devel/2018-February/msg00001.html
https://www.redhat.com/archives/lvm-devel/2018-February/msg00000.html

(may need some slight changes if the sources for 7.5 already diverted)
Comment 12 Roman Bednář 2018-02-13 14:16:40 UTC 
Verified.


3.10.0-847.el7.x86_64

lvm2-2.02.177-2.el7    
lvm2-libs-2.02.177-2.el7    
lvm2-cluster-2.02.177-2.el7  



SCENARIO - [segmented_pvmove]
Create a couple small segmented lvs and then pvmove them
 Running lv_seg on virt-364 to create the segmented volumes
 virt-364: /usr/tests/sts-rhel7.5/lvm2/bin/lv_seg -v mirror_sanity -n segment
 Deactivating segment0 mirror
 Moving data from pv /dev/sdc1 (-n mirror_sanity/segment0) on virt-364
     Executing: /usr/sbin/modprobe dm-log-userspace
     Wiping internal VG cache
     Wiping cache of LVM-capable devices
   Increasing mirror region size from 0    to 512 B
     Archiving volume group "mirror_sanity" metadata (seqno 27).
     Creating logical volume pvmove0
     Moving 27 extents of logical volume mirror_sanity/segment0.
     Creating volume group backup "/etc/lvm/backup/mirror_sanity" (seqno 28).
     Checking progress before waiting every 15 seconds.
   /dev/sdc1: Moved: 25.93%
   /dev/sdc1: Moved: 100.00%
 Device does not exist.
 Command failed
 Unable to get copy percent, pvmove most likely finished.
 Ignore 'Command failed' messages ;)
     Polling finished successfully.
 Device does not exist.
 Command failed
 Device does not exist.
 Command failed
 Device does not exist.
 Command failed
 Device does not exist.
 Command failed
 Device does not exist.
 Command failed
 Device does not exist.
 Command failed
 Quick verification that pvmove is finished on other nodes as well
 Deactivating mirror segment0... and removing
 Deactivating mirror segment1... and removing
Comment 14 Marian Csontos 2018-02-16 08:18:36 UTC 
Kabi, is this complete?

a2d2fe3a8cf840fcfcd23fb0e706c3699b79b5fa
552e60b3a1e35329a47d6112c548ada124b5a4e3

I would do the build today or Monday, so QE can get their hands on it ASAP.
Comment 16 Roman Bednář 2018-02-22 10:54:41 UTC 
Verified.

lvm2-2.02.177-4.el7.x86_64
kernel-3.10.0-854.el7.x86_64


SCENARIO - [segmented_pvmove]
Create a couple small segmented lvs and then pvmove them
Running lv_seg on virt-386 to create the segmented volumes
virt-386: /usr/tests/sts-rhel7.5/lvm2/bin/lv_seg -v mirror_sanity -n segment
Deactivating segment0 mirror
Moving data from pv /dev/sdb1 (-n mirror_sanity/segment0) on virt-386
    Executing: /usr/sbin/modprobe dm-log-userspace
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "mirror_sanity" metadata (seqno 27).
    Creating logical volume pvmove0
  Increasing mirror region size from 0    to 512 B
    Moving 105 extents of logical volume mirror_sanity/segment0.
    Creating volume group backup "/etc/lvm/backup/mirror_sanity" (seqno 28).
    Checking progress before waiting every 15 seconds.
  /dev/sdb1: Moved: 7.62%
  /dev/sdb1: Moved: 77.14%
  /dev/sdb1: Moved: 80.95%
  /dev/sdb1: Moved: 84.76%
  /dev/sdb1: Moved: 88.57%
  /dev/sdb1: Moved: 92.38%
  /dev/sdb1: Moved: 96.19%
  /dev/sdb1: Moved: 100.00%
    Polling finished successfully.
Device does not exist.
Command failed
Unable to get copy percent, pvmove most likely finished.
Ignore 'Command failed' messages ;)
Device does not exist.
Command failed
Device does not exist.
Command failed
Device does not exist.
Command failed
Device does not exist.
Command failed
Device does not exist.
Command failed
Device does not exist.
Command failed
Quick verification that pvmove is finished on other nodes as well
Deactivating mirror segment0... and removing
Deactivating mirror segment1... and removing
Comment 19 errata-xmlrpc 2018-04-10 15:23:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0853