Red Hat Bugzilla – Bug 1492897
lvmlockd needs to safeguard against pvmoves on shared activated volumes with no cmirrord option
Last modified: 2018-04-10 11:23:08 EDT
Description of problem: This appears to be a current side effect of bug 1488592 as pvmove uses cmirrord when pvmove'ing shared activated linear/stripe volumes. Without a safeguard corruption can occur as shown here. [root@host-113 ~]# pcs status Cluster name: STSRHTS3525 Stack: corosync Current DC: host-115 (version 1.1.16-12.el7-94ff4df) - partition with quorum Last updated: Mon Sep 18 17:42:51 2017 Last change: Mon Sep 18 16:06:58 2017 by root via cibadmin on host-132 3 nodes configured 9 resources configured Online: [ host-113 host-115 host-132 ] Full list of resources: fence-host-115 (stonith:fence_xvm): Started host-113 fence-host-132 (stonith:fence_xvm): Started host-115 fence-host-113 (stonith:fence_xvm): Started host-132 Clone Set: dlm-clone [dlm] Started: [ host-113 host-115 host-132 ] Clone Set: lvmlockd-clone [lvmlockd] Started: [ host-113 host-115 host-132 ] ## Test setup: creating lvm devices... host-115: pvcreate /dev/sdg3 /dev/sdg2 /dev/sdg1 /dev/sdh3 /dev/sdh2 /dev/sdh1 host-115: vgcreate --shared pv_shuffle_A /dev/sdg3 /dev/sdg2 /dev/sdg1 /dev/sdh3 /dev/sdh2 /dev/sdh1 host-115: vgchange --lock-start pv_shuffle_A host-132: vgchange --lock-start pv_shuffle_A host-113: vgchange --lock-start pv_shuffle_A host-115: pvcreate /dev/sdc3 /dev/sdc2 /dev/sdc1 /dev/sdf3 /dev/sdf2 /dev/sdf1 host-115: vgcreate --shared pv_shuffle_B /dev/sdc3 /dev/sdc2 /dev/sdc1 /dev/sdf3 /dev/sdf2 /dev/sdf1 host-115: vgchange --lock-start pv_shuffle_B host-132: vgchange --lock-start pv_shuffle_B host-113: vgchange --lock-start pv_shuffle_B Create 4 free PV(s) on host-115 volume creation on pv_shuffle_A host-115 lvcreate --activate sy -n linear -L 2G pv_shuffle_A host-115 lvcreate --activate sy -i 3 -n stripe -L 2G pv_shuffle_A Activating shared lockd volume remotely on all nodes WARNING: lvmetad is being updated, retrying (setup) for 10 more seconds. volume creation on pv_shuffle_B host-115 lvcreate --activate sy -n linear -L 2G pv_shuffle_B host-115 lvcreate --activate sy -i 3 -n stripe -L 2G pv_shuffle_B Activating shared lockd volume remotely on all nodes filesystem creation on pv_shuffle_A mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:85637 /dev/pv_shuffle_A/linear -O mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:32249 /dev/pv_shuffle_A/stripe -O filesystem creation on pv_shuffle_B mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:80203 /dev/pv_shuffle_B/linear -O mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:53283 /dev/pv_shuffle_B/stripe -O ## Started IO load [cmarthal@dhcp80-218 ~]$ nannyctl -n shuffle_io_load_21645 ps host-115_pv_shuffle_A_linear [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_A_linear/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-132_pv_shuffle_A_linear [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_A_linear/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-113_pv_shuffle_A_linear [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_A_linear/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-115_pv_shuffle_A_stripe [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_A_stripe/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-132_pv_shuffle_A_stripe [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_A_stripe/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-113_pv_shuffle_A_stripe [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_A_stripe/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-115_pv_shuffle_B_linear [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_B_linear/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-132_pv_shuffle_B_linear [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_B_linear/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-113_pv_shuffle_B_linear [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_B_linear/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-115_pv_shuffle_B_stripe [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_B_stripe/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-132_pv_shuffle_B_stripe [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_B_stripe/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running host-113_pv_shuffle_B_stripe [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000 -F 1000000:/mnt/pv_shuffle_B_stripe/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"] Running [root@host-113 ~]# vgextend pv_shuffle_B /dev/sde1 /dev/sda3 Volume group "pv_shuffle_B" successfully extended [root@host-113 ~]# lvs -a -o +devices LV VG Attr LSize Devices linear pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(0) stripe pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0) linear pv_shuffle_B -wi-ao---- 2.00g /dev/sdc3(0) stripe pv_shuffle_B -wi-ao---- 2.00g /dev/sdc3(512),/dev/sdc2(0),/dev/sdc1(0) ## Attempt pvmoves of the same device from different nodes in the cluster (note the 'Cluster mirror log daemon is not running': [root@host-113 ~]# pvmove -v /dev/sdc3 /dev/sda3 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices Archiving volume group "pv_shuffle_B" metadata (seqno 5). Creating logical volume pvmove0 Moving 512 extents of logical volume pv_shuffle_B/linear Moving 171 extents of logical volume pv_shuffle_B/stripe activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear. activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe. Setting up pvmove in on-disk volume group metadata. Creating pv_shuffle_B-pvmove0 Loading pv_shuffle_B-pvmove0 table (253:6) Loading pv_shuffle_B-linear table (253:4) Loading pv_shuffle_B-stripe table (253:5) Suspending pv_shuffle_B-linear (253:4) with device flush Suspending pv_shuffle_B-stripe (253:5) with device flush activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0. Resuming pv_shuffle_B-pvmove0 (253:6) Loading pv_shuffle_B-pvmove0 table (253:6) Suppressed pv_shuffle_B-pvmove0 (253:6) identical table reload. Resuming pv_shuffle_B-linear (253:4) Resuming pv_shuffle_B-stripe (253:5) Creating volume group backup "/etc/lvm/backup/pv_shuffle_B" (seqno 6). Checking progress before waiting every 15 seconds. /dev/sdc3: Moved: 1.02% /dev/sdc3: Moved: 40.41% /dev/sdc3: Moved: 74.96% /dev/sdc3: Moved: 100.00% Polling finished successfully. ## This was started w/ the above pvmove still in progress. "Detected pvmove in progress for /dev/sdc3" [root@host-115 ~]# lvs -a -o +devices LV VG Attr LSize Move Devices linear pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(0) stripe pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0) linear pv_shuffle_B -wI-ao---- 2.00g pvmove0(0) [pvmove0] pv_shuffle_B p-C---m--- <2.67g /dev/sdc3 /dev/sdc3(0),/dev/sda3(0) [pvmove0] pv_shuffle_B p-C---m--- <2.67g /dev/sdc3 /dev/sdc3(512),/dev/sda3(512) stripe pv_shuffle_B -wI-ao---- 2.00g pvmove0(0),/dev/sdc2(0),/dev/sdc1(0) [root@host-115 ~]# pvmove -v /dev/sdc3 /dev/sda2 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices Detected pvmove in progress for /dev/sdc3 Ignoring remaining command line arguments activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0. Creating pv_shuffle_B-pvmove0 Loading pv_shuffle_B-pvmove0 table (253:6) Resuming pv_shuffle_B-pvmove0 (253:6) activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear. activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe. Checking progress before waiting every 15 seconds. /dev/sdc3: Moved: 75.70% /dev/sdc3: Moved: 100.00% Polling finished successfully. [root@host-115 ~]# lvs -a -o +devices LV VG Attr LSize Devices linear pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(0) stripe pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0) linear pv_shuffle_B -wi-ao---- 2.00g /dev/sda3(0) stripe pv_shuffle_B -wi-ao---- 2.00g /dev/sda3(512),/dev/sdc2(0),/dev/sdc1(0) [root@host-113 ~]# lvs -a -o +devices LV VG Attr LSize Devices linear pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(0) stripe pv_shuffle_A -wi-ao---- 2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0) linear pv_shuffle_B -wi-ao---- 2.00g /dev/sda3(0) stripe pv_shuffle_B -wi-ao---- 2.00g /dev/sda3(512),/dev/sdc2(0),/dev/sdc1(0) [root@host-115 ~]# dmsetup ls pv_shuffle_A-stripe (253:3) pv_shuffle_A-linear (253:2) pv_shuffle_B-pvmove0 (253:6) <- still existing pvmove left over from above attempt? pv_shuffle_B-stripe (253:5) pv_shuffle_B-linear (253:4) [root@host-115 ~]# pvmove -v /dev/sda3 /dev/sdc1 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices Archiving volume group "pv_shuffle_B" metadata (seqno 9). Creating logical volume pvmove0 Moving 512 extents of logical volume pv_shuffle_B/linear Moving 171 extents of logical volume pv_shuffle_B/stripe Insufficient suitable allocatable extents for logical volume : 171 more required Unable to allocate mirror extents for pv_shuffle_B/pvmove0. Failed to convert pvmove LV to mirrored [root@host-115 ~]# pvmove -v /dev/sda3 /dev/sdf3 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices Archiving volume group "pv_shuffle_B" metadata (seqno 9). Creating logical volume pvmove0 Moving 512 extents of logical volume pv_shuffle_B/linear Moving 171 extents of logical volume pv_shuffle_B/stripe activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear. activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe. Setting up pvmove in on-disk volume group metadata. Creating pv_shuffle_B-pvmove0 device-mapper: create ioctl on pv_shuffle_B-pvmove0 LVM-glzRa4uPoEr5obUcCkHlxamnnjo2Vm4RFCH4UH3oE6wuuDNQih2EbcUO4U0ko8E1 failed: Device or resource busy Failed to suspend linear ABORTING: Temporary pvmove mirror activation failed. [root@host-115 ~]# dmsetup remove pv_shuffle_B-pvmove0 [root@host-115 ~]# pvmove -v /dev/sda3 /dev/sdf3 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices Archiving volume group "pv_shuffle_B" metadata (seqno 9). Creating logical volume pvmove0 Moving 512 extents of logical volume pv_shuffle_B/linear Moving 171 extents of logical volume pv_shuffle_B/stripe activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear. activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe. Setting up pvmove in on-disk volume group metadata. Creating pv_shuffle_B-pvmove0 Loading pv_shuffle_B-pvmove0 table (253:6) Loading pv_shuffle_B-linear table (253:4) Loading pv_shuffle_B-stripe table (253:5) Suspending pv_shuffle_B-linear (253:4) with device flush Suspending pv_shuffle_B-stripe (253:5) with device flush activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0. Resuming pv_shuffle_B-pvmove0 (253:6) Loading pv_shuffle_B-pvmove0 table (253:6) Suppressed pv_shuffle_B-pvmove0 (253:6) identical table reload. Resuming pv_shuffle_B-linear (253:4) Resuming pv_shuffle_B-stripe (253:5) Creating volume group backup "/etc/lvm/backup/pv_shuffle_B" (seqno 10). Checking progress before waiting every 15 seconds. /dev/sda3: Moved: 0.73% /dev/sda3: Moved: 34.85% /dev/sda3: Moved: 60.18% /dev/sda3: Moved: 74.96% /dev/sda3: Moved: 100.00% Polling finished successfully. ## Filesystem appears gone now [root@host-113 ~]# ls -lrt /mnt/pv_shuffle_B_linear ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error total 1000 -?????????? ? ? ? ? ? host-132_load -rw-rw-rw-. 1 root root 1000000 Sep 15 10:21 host-115_load -rw-rw-rw-. 1 root root 1000000 Sep 18 17:32 host-113_load [root@host-115 ~]# ls -lrt /mnt/pv_shuffle_B_linear ls: cannot access /mnt/pv_shuffle_B_linear/host-113_load: Input/output error ls: cannot access /mnt/pv_shuffle_B_linear/host-115_load: Input/output error ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error total 0 -????????? ? ? ? ? ? host-132_load -????????? ? ? ? ? ? host-115_load -????????? ? ? ? ? ? host-113_load [root@host-132 ~]# ls -lrt /mnt/pv_shuffle_B_linear ls: cannot access /mnt/pv_shuffle_B_linear/host-113_load: Input/output error ls: cannot access /mnt/pv_shuffle_B_linear/host-115_load: Input/output error ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error total 0 -????????? ? ? ? ? ? host-132_load -????????? ? ? ? ? ? host-115_load -????????? ? ? ? ? ? host-113_load Sep 18 17:32:06 host-113 crmd[3033]: notice: High CPU load detected: 4.310000 Sep 18 17:32:36 host-113 crmd[3033]: notice: High CPU load detected: 4.250000 Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: fatal: invalid metadata block#012GFS2: fsid=STSRHTS3525:80203.2: bh = 159360 (magic number)#012GFS2: fsid=STSRHTS3525:80203.2: function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 437 Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: about to withdraw this file system Sep 18 17:32:47 host-113 qarshd[4033]: Sending child 4038 signal 2 Sep 18 17:32:47 host-113 qarshd[4035]: Sending child 4040 signal 2 Sep 18 17:32:47 host-113 qarshd[4034]: Sending child 4039 signal 2 Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: telling LM to unmount Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: withdrawn Sep 18 17:32:47 host-113 kernel: CPU: 0 PID: 3999 Comm: kworker/0:2H Not tainted 3.10.0-693.1.1.el7.x86_64 #1 Sep 18 17:32:47 host-113 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 Sep 18 17:32:47 host-113 kernel: Workqueue: glock_workqueue glock_work_func [gfs2] Sep 18 17:32:47 host-113 kernel: ffff880020113000 000000006ad32300 ffff88002009fbf0 ffffffff816a3d91 Sep 18 17:32:47 host-113 kernel: ffff88002009fc68 ffffffffc03cb456 0000000000000030 ffff88002009fc98 Sep 18 17:32:47 host-113 kernel: ffff88002009fc20 000000006ad32300 ffff88002009fcb0 ffff88002009fc38 Sep 18 17:32:47 host-113 kernel: Call Trace: Sep 18 17:32:47 host-113 kernel: [<ffffffff816a3d91>] dump_stack+0x19/0x1b Sep 18 17:32:47 host-113 kernel: [<ffffffffc03cb456>] gfs2_lm_withdraw+0x146/0x180 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffffc03cb745>] gfs2_meta_check_ii+0x45/0x50 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffffc03b2b59>] gfs2_meta_indirect_buffer+0xe9/0x150 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffff810bd504>] ? __wake_up+0x44/0x50 Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aeb64>] gfs2_inode_refresh+0x34/0x90 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aec38>] inode_go_lock+0x78/0xf0 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aae74>] do_promote+0x1b4/0x310 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffffc03ac7f2>] finish_xmote+0x172/0x4c0 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffffc03ad192>] glock_work_func+0x122/0x150 [gfs2] Sep 18 17:32:47 host-113 kernel: [<ffffffff810a881a>] process_one_work+0x17a/0x440 Sep 18 17:32:47 host-113 kernel: [<ffffffff810a94e6>] worker_thread+0x126/0x3c0 Sep 18 17:32:47 host-113 kernel: [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0 Sep 18 17:32:47 host-113 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0 Sep 18 17:32:47 host-113 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40 Sep 18 17:32:47 host-113 kernel: [<ffffffff816b4f18>] ret_from_fork+0x58/0x90 Sep 18 17:32:47 host-113 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40 Sep 18 17:33:06 host-113 crmd[3033]: notice: High CPU load detected: 3.120000 Version-Release number of selected component (if applicable): 3.10.0-693.1.1.el7.x86_64 lvm2-2.02.171-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 lvm2-libs-2.02.171-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 lvm2-cluster-2.02.171-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 device-mapper-1.02.140-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 device-mapper-libs-1.02.140-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 device-mapper-event-1.02.140-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 device-mapper-event-libs-1.02.140-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 10:15:46 CDT 2017 cmirror-2.02.171-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017 sanlock-3.5.0-1.el7 BUILT: Wed Apr 26 09:37:30 CDT 2017 sanlock-lib-3.5.0-1.el7 BUILT: Wed Apr 26 09:37:30 CDT 2017 lvm2-lockd-2.02.171-8.el7 BUILT: Wed Jun 28 13:28:58 CDT 2017
Two commits addressing mirror and pvmove issues in a shared VG: https://sourceware.org/git/?p=lvm2.git;a=commit;h=518a8e8cfbb672c2bf5e3455f1fe7cd8d94eb5b0 https://sourceware.org/git/?p=lvm2.git;a=commit;h=f2ee0e7aca6020cf273f22dc5bca20bfa3c89051 LVs with 'mirror' type can now be activated with a shared lock (-asy) from multiple hosts, and lvm correctly uses cmirrord for them. This only applies to shared VGs with lock_type dlm (sanlock is excluded since the combination of sanlock and corosync is not useful.) pvmove in a shared VG is restricted to cases where an LV name is specified (pvmove -n LV). A further restriction is that the node running pvmove must be able to exclusively activate the LV. This means a shared VG does not support pvmove of an entire PV, and does not support pvmove of an LV active on multiple nodes, e.g. under gfs.
Verified that pvmoves on shared locked LVs (w/ or w/o a given -n lv name arg) is no longer allowed. LVs need to be exclusively locked (or converted to ex lock by being the only node w/ a shared lock) before a pvmove can take place. limitations of lockd VGs Things that do not yet work in lockd VGs: [...] · pvmove of entire PVs, or under LVs activated with shared locks [root@host-026 ~]# lvs -a -o +devices pv_shuffle_A/stripe LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices stripe pv_shuffle_A -wi-ao---- 2.00g /dev/sda3(512),/dev/sda2(0),/dev/sda1(0) [root@host-026 ~]# mount | grep stripe /dev/mapper/pv_shuffle_A-stripe on /mnt/pv_shuffle_A_stripe type gfs2 (rw,relatime,seclabel) # Unnamed attempt: [root@host-026 ~]# pvmove -b -v /dev/sda2 /dev/sdh1 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices pvmove in a shared VG requires a named LV. # Shared lock w/ lv name attempt: [root@host-026 ~]# pvmove -b -v -n pv_shuffle_A/stripe /dev/sda2 /dev/sdh1 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices LV locked by other host: pv_shuffle_A/stripe pvmove in a shared VG requires exclusive lock on named LV. [root@host-026 ~]# lvchange -an pv_shuffle_A/stripe [root@host-026 ~]# lvchange -aye pv_shuffle_A/stripe [root@host-026 ~]# pvmove -b -v /dev/sda2 /dev/sdh1 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices pvmove in a shared VG requires a named LV. [root@host-026 ~]# pvmove -b -v -n pv_shuffle_A/stripe /dev/sda2 /dev/sdd2 Cluster mirror log daemon is not running. Wiping internal VG cache Wiping cache of LVM-capable devices Archiving volume group "pv_shuffle_A" metadata (seqno 27). Creating logical volume pvmove0 Moving 171 extents of logical volume pv_shuffle_A/stripe activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_A/stripe. Setting up pvmove in on-disk volume group metadata. Creating pv_shuffle_A-pvmove0 Loading pv_shuffle_A-pvmove0 table (253:6) Loading pv_shuffle_A-stripe table (253:3) Suspending pv_shuffle_A-stripe (253:3) with device flush activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_A/pvmove0. Resuming pv_shuffle_A-pvmove0 (253:6) Loading pv_shuffle_A-pvmove0 table (253:6) Suppressed pv_shuffle_A-pvmove0 (253:6) identical table reload. Resuming pv_shuffle_A-stripe (253:3) Creating volume group backup "/etc/lvm/backup/pv_shuffle_A" (seqno 28). Checking progress before waiting every 15 seconds. [root@host-026 ~]# lvs -a -o +devices pv_shuffle_A/stripe LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices stripe pv_shuffle_A -wi-a----- 2.00g /dev/sda3(512),/dev/sdd2(257),/dev/sda1(0) lvm2-2.02.175-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 lvm2-libs-2.02.175-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 lvm2-cluster-2.02.175-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 device-mapper-1.02.144-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 device-mapper-libs-1.02.144-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 device-mapper-event-1.02.144-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 device-mapper-event-libs-1.02.144-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 10:15:46 CDT 2017 cmirror-2.02.175-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017 sanlock-3.5.0-1.el7 BUILT: Wed Apr 26 09:37:30 CDT 2017 sanlock-lib-3.5.0-1.el7 BUILT: Wed Apr 26 09:37:30 CDT 2017 lvm2-lockd-2.02.175-2.el7 BUILT: Fri Oct 13 06:31:22 CDT 2017
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0853