1492897 – lvmlockd needs to safeguard against pvmoves on shared activated volumes with no cmirrord option

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1492897 - lvmlockd needs to safeguard against pvmoves on shared activated volumes with no cmirrord option

Summary: lvmlockd needs to safeguard against pvmoves on shared activated volumes with ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.4
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	LVM and device-mapper development team
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-09-18 23:08 UTC by Corey Marthaler
Modified:	2021-09-03 12:39 UTC (History)
CC List:	7 users (show)
Fixed In Version:	lvm2-2.02.175-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-04-10 15:22:16 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2018:0853	0	None	None	None	2018-04-10 15:23:07 UTC

Description Corey Marthaler 2017-09-18 23:08:29 UTC

Description of problem:
This appears to be a current side effect of bug 1488592 as pvmove uses cmirrord when pvmove'ing shared activated linear/stripe volumes. Without a safeguard corruption can occur as shown here.


[root@host-113 ~]# pcs status
Cluster name: STSRHTS3525
Stack: corosync
Current DC: host-115 (version 1.1.16-12.el7-94ff4df) - partition with quorum
Last updated: Mon Sep 18 17:42:51 2017
Last change: Mon Sep 18 16:06:58 2017 by root via cibadmin on host-132

3 nodes configured
9 resources configured

Online: [ host-113 host-115 host-132 ]

Full list of resources:

 fence-host-115 (stonith:fence_xvm):    Started host-113
 fence-host-132 (stonith:fence_xvm):    Started host-115
 fence-host-113 (stonith:fence_xvm):    Started host-132
 Clone Set: dlm-clone [dlm]
     Started: [ host-113 host-115 host-132 ]
 Clone Set: lvmlockd-clone [lvmlockd]
     Started: [ host-113 host-115 host-132 ]


## Test setup:
creating lvm devices...
host-115: pvcreate /dev/sdg3 /dev/sdg2 /dev/sdg1 /dev/sdh3 /dev/sdh2 /dev/sdh1
host-115: vgcreate  --shared pv_shuffle_A /dev/sdg3 /dev/sdg2 /dev/sdg1 /dev/sdh3 /dev/sdh2 /dev/sdh1
host-115: vgchange --lock-start pv_shuffle_A
host-132: vgchange --lock-start pv_shuffle_A
host-113: vgchange --lock-start pv_shuffle_A

host-115: pvcreate /dev/sdc3 /dev/sdc2 /dev/sdc1 /dev/sdf3 /dev/sdf2 /dev/sdf1
host-115: vgcreate  --shared pv_shuffle_B /dev/sdc3 /dev/sdc2 /dev/sdc1 /dev/sdf3 /dev/sdf2 /dev/sdf1
host-115: vgchange --lock-start pv_shuffle_B
host-132: vgchange --lock-start pv_shuffle_B
host-113: vgchange --lock-start pv_shuffle_B

Create 4 free PV(s) on host-115
volume creation on pv_shuffle_A
        host-115 lvcreate --activate sy -n linear -L 2G pv_shuffle_A
        host-115 lvcreate --activate sy -i 3 -n stripe -L 2G pv_shuffle_A
Activating shared lockd volume remotely on all nodes
  WARNING: lvmetad is being updated, retrying (setup) for 10 more seconds.
volume creation on pv_shuffle_B
        host-115 lvcreate --activate sy -n linear -L 2G pv_shuffle_B
        host-115 lvcreate --activate sy -i 3 -n stripe -L 2G pv_shuffle_B

Activating shared lockd volume remotely on all nodes
filesystem creation on pv_shuffle_A
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:85637 /dev/pv_shuffle_A/linear -O
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:32249 /dev/pv_shuffle_A/stripe -O

filesystem creation on pv_shuffle_B
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:80203 /dev/pv_shuffle_B/linear -O
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:53283 /dev/pv_shuffle_B/stripe -O


## Started IO load
[cmarthal@dhcp80-218 ~]$ nannyctl -n shuffle_io_load_21645 ps
host-115_pv_shuffle_A_linear  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_linear/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_A_linear  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_linear/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_A_linear  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_linear/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-115_pv_shuffle_A_stripe  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_stripe/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_A_stripe  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_stripe/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_A_stripe  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_stripe/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-115_pv_shuffle_B_linear  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_linear/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_B_linear  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_linear/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_B_linear  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_linear/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-115_pv_shuffle_B_stripe  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_stripe/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_B_stripe  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_stripe/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_B_stripe  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_stripe/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running


[root@host-113 ~]# vgextend pv_shuffle_B /dev/sde1 /dev/sda3
  Volume group "pv_shuffle_B" successfully extended

[root@host-113 ~]# lvs -a -o +devices
  LV     VG            Attr       LSize   Devices                                 
  linear pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(0)                            
  stripe pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear pv_shuffle_B  -wi-ao----   2.00g /dev/sdc3(0)                            
  stripe pv_shuffle_B  -wi-ao----   2.00g /dev/sdc3(512),/dev/sdc2(0),/dev/sdc1(0)

## Attempt pvmoves of the same device from different nodes in the cluster (note the 'Cluster mirror log daemon is not running':
[root@host-113 ~]# pvmove -v /dev/sdc3 /dev/sda3
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 5).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_B-pvmove0
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Loading pv_shuffle_B-linear table (253:4)
    Loading pv_shuffle_B-stripe table (253:5)
    Suspending pv_shuffle_B-linear (253:4) with device flush
    Suspending pv_shuffle_B-stripe (253:5) with device flush
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0.
    Resuming pv_shuffle_B-pvmove0 (253:6)
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Suppressed pv_shuffle_B-pvmove0 (253:6) identical table reload.
    Resuming pv_shuffle_B-linear (253:4)
    Resuming pv_shuffle_B-stripe (253:5)
    Creating volume group backup "/etc/lvm/backup/pv_shuffle_B" (seqno 6).
    Checking progress before waiting every 15 seconds.
  /dev/sdc3: Moved: 1.02%
  /dev/sdc3: Moved: 40.41%
  /dev/sdc3: Moved: 74.96%
  /dev/sdc3: Moved: 100.00%
    Polling finished successfully.


## This was started w/ the above pvmove still in progress. "Detected pvmove in progress for /dev/sdc3"
[root@host-115 ~]# lvs -a -o +devices
  LV        VG            Attr       LSize    Move       Devices                                 
  linear    pv_shuffle_A  -wi-ao----   2.00g             /dev/sdh3(0)                            
  stripe    pv_shuffle_A  -wi-ao----   2.00g             /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear    pv_shuffle_B  -wI-ao----   2.00g             pvmove0(0)                              
  [pvmove0] pv_shuffle_B  p-C---m---  <2.67g  /dev/sdc3  /dev/sdc3(0),/dev/sda3(0)               
  [pvmove0] pv_shuffle_B  p-C---m---  <2.67g  /dev/sdc3  /dev/sdc3(512),/dev/sda3(512)           
  stripe    pv_shuffle_B  -wI-ao----   2.00g             pvmove0(0),/dev/sdc2(0),/dev/sdc1(0)    

[root@host-115 ~]# pvmove -v /dev/sdc3 /dev/sda2
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  Detected pvmove in progress for /dev/sdc3
  Ignoring remaining command line arguments
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0.
    Creating pv_shuffle_B-pvmove0
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Resuming pv_shuffle_B-pvmove0 (253:6)
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Checking progress before waiting every 15 seconds.
  /dev/sdc3: Moved: 75.70%
  /dev/sdc3: Moved: 100.00%
    Polling finished successfully.

[root@host-115 ~]# lvs -a -o +devices
  LV     VG            Attr       LSize   Devices                                 
  linear pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(0)                            
  stripe pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(0)                            
  stripe pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(512),/dev/sdc2(0),/dev/sdc1(0)

[root@host-113 ~]# lvs -a -o +devices
  LV     VG            Attr       LSize   Devices                                 
  linear pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(0)                            
  stripe pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(0)                            
  stripe pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(512),/dev/sdc2(0),/dev/sdc1(0)

[root@host-115 ~]# dmsetup ls
pv_shuffle_A-stripe     (253:3)
pv_shuffle_A-linear     (253:2)
pv_shuffle_B-pvmove0    (253:6)  <- still existing pvmove left over from above attempt?
pv_shuffle_B-stripe     (253:5)
pv_shuffle_B-linear     (253:4)


[root@host-115 ~]#  pvmove -v /dev/sda3 /dev/sdc1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 9).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
  Insufficient suitable allocatable extents for logical volume : 171 more required
  Unable to allocate mirror extents for pv_shuffle_B/pvmove0.
  Failed to convert pvmove LV to mirrored

[root@host-115 ~]#  pvmove -v /dev/sda3 /dev/sdf3
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 9).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_B-pvmove0
  device-mapper: create ioctl on pv_shuffle_B-pvmove0 LVM-glzRa4uPoEr5obUcCkHlxamnnjo2Vm4RFCH4UH3oE6wuuDNQih2EbcUO4U0ko8E1 failed: Device or resource busy
  Failed to suspend linear
  ABORTING: Temporary pvmove mirror activation failed.

[root@host-115 ~]# dmsetup remove pv_shuffle_B-pvmove0

[root@host-115 ~]#  pvmove -v /dev/sda3 /dev/sdf3
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 9).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_B-pvmove0
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Loading pv_shuffle_B-linear table (253:4)
    Loading pv_shuffle_B-stripe table (253:5)
    Suspending pv_shuffle_B-linear (253:4) with device flush
    Suspending pv_shuffle_B-stripe (253:5) with device flush
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0.
    Resuming pv_shuffle_B-pvmove0 (253:6)
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Suppressed pv_shuffle_B-pvmove0 (253:6) identical table reload.
    Resuming pv_shuffle_B-linear (253:4)
    Resuming pv_shuffle_B-stripe (253:5)
    Creating volume group backup "/etc/lvm/backup/pv_shuffle_B" (seqno 10).
    Checking progress before waiting every 15 seconds.
  /dev/sda3: Moved: 0.73%
  /dev/sda3: Moved: 34.85%
  /dev/sda3: Moved: 60.18%
  /dev/sda3: Moved: 74.96%
  /dev/sda3: Moved: 100.00%
    Polling finished successfully.


## Filesystem appears gone now
[root@host-113 ~]# ls -lrt /mnt/pv_shuffle_B_linear
ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error
ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error
total 1000
-?????????? ? ?    ?          ?            ? host-132_load
-rw-rw-rw-. 1 root root 1000000 Sep 15 10:21 host-115_load
-rw-rw-rw-. 1 root root 1000000 Sep 18 17:32 host-113_load

[root@host-115 ~]# ls -lrt /mnt/pv_shuffle_B_linear
ls: cannot access /mnt/pv_shuffle_B_linear/host-113_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-115_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error
ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error
total 0
-????????? ? ? ? ?            ? host-132_load
-????????? ? ? ? ?            ? host-115_load
-????????? ? ? ? ?            ? host-113_load

[root@host-132 ~]# ls -lrt /mnt/pv_shuffle_B_linear
ls: cannot access /mnt/pv_shuffle_B_linear/host-113_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-115_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error
ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error
total 0
-????????? ? ? ? ?            ? host-132_load
-????????? ? ? ? ?            ? host-115_load
-????????? ? ? ? ?            ? host-113_load


Sep 18 17:32:06 host-113 crmd[3033]:  notice: High CPU load detected: 4.310000
Sep 18 17:32:36 host-113 crmd[3033]:  notice: High CPU load detected: 4.250000
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: fatal: invalid metadata block#012GFS2: fsid=STSRHTS3525:80203.2:   bh = 159360 (magic number)#012GFS2: fsid=STSRHTS3525:80203.2:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 437
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: about to withdraw this file system
Sep 18 17:32:47 host-113 qarshd[4033]: Sending child 4038 signal 2
Sep 18 17:32:47 host-113 qarshd[4035]: Sending child 4040 signal 2
Sep 18 17:32:47 host-113 qarshd[4034]: Sending child 4039 signal 2
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: telling LM to unmount
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: withdrawn
Sep 18 17:32:47 host-113 kernel: CPU: 0 PID: 3999 Comm: kworker/0:2H Not tainted 3.10.0-693.1.1.el7.x86_64 #1
Sep 18 17:32:47 host-113 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
Sep 18 17:32:47 host-113 kernel: Workqueue: glock_workqueue glock_work_func [gfs2]
Sep 18 17:32:47 host-113 kernel: ffff880020113000 000000006ad32300 ffff88002009fbf0 ffffffff816a3d91
Sep 18 17:32:47 host-113 kernel: ffff88002009fc68 ffffffffc03cb456 0000000000000030 ffff88002009fc98
Sep 18 17:32:47 host-113 kernel: ffff88002009fc20 000000006ad32300 ffff88002009fcb0 ffff88002009fc38
Sep 18 17:32:47 host-113 kernel: Call Trace:
Sep 18 17:32:47 host-113 kernel: [<ffffffff816a3d91>] dump_stack+0x19/0x1b
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03cb456>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03cb745>] gfs2_meta_check_ii+0x45/0x50 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03b2b59>] gfs2_meta_indirect_buffer+0xe9/0x150 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffff810bd504>] ? __wake_up+0x44/0x50
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aeb64>] gfs2_inode_refresh+0x34/0x90 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aec38>] inode_go_lock+0x78/0xf0 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aae74>] do_promote+0x1b4/0x310 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03ac7f2>] finish_xmote+0x172/0x4c0 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03ad192>] glock_work_func+0x122/0x150 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffff810a881a>] process_one_work+0x17a/0x440
Sep 18 17:32:47 host-113 kernel: [<ffffffff810a94e6>] worker_thread+0x126/0x3c0
Sep 18 17:32:47 host-113 kernel: [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
Sep 18 17:32:47 host-113 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Sep 18 17:32:47 host-113 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Sep 18 17:32:47 host-113 kernel: [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
Sep 18 17:32:47 host-113 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Sep 18 17:33:06 host-113 crmd[3033]:  notice: High CPU load detected: 3.120000




Version-Release number of selected component (if applicable):
3.10.0-693.1.1.el7.x86_64

lvm2-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
lvm2-libs-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
lvm2-cluster-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-libs-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-event-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-event-libs-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 10:15:46 CDT 2017
cmirror-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
sanlock-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
sanlock-lib-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
lvm2-lockd-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017

Comment 2 David Teigland 2017-09-20 17:00:59 UTC

Two commits addressing mirror and pvmove issues in a shared VG:

https://sourceware.org/git/?p=lvm2.git;a=commit;h=518a8e8cfbb672c2bf5e3455f1fe7cd8d94eb5b0

https://sourceware.org/git/?p=lvm2.git;a=commit;h=f2ee0e7aca6020cf273f22dc5bca20bfa3c89051

LVs with 'mirror' type can now be activated with a shared lock (-asy) from multiple hosts, and lvm correctly uses cmirrord for them.  This only applies to shared VGs with lock_type dlm (sanlock is excluded since the combination of sanlock and corosync is not useful.)

pvmove in a shared VG is restricted to cases where an LV name is specified (pvmove -n LV).  A further restriction is that the node running pvmove must be able to exclusively activate the LV.  This means a shared VG does not support pvmove of an entire PV, and does not support pvmove of an LV active on multiple nodes, e.g. under gfs.

Comment 5 Corey Marthaler 2017-10-16 15:23:10 UTC

Verified that pvmoves on shared locked LVs (w/ or w/o a given -n lv name arg) is no longer allowed. LVs need to be exclusively locked (or converted to ex lock by being the only node w/ a shared lock) before a pvmove can take place.

   limitations of lockd VGs                                                                                                                                                                                  
       Things that do not yet work in lockd VGs:                                                                                                                                                             
       [...]                                                                                                                                                                                                 
       Â· pvmove of entire PVs, or under LVs activated with shared locks                                                                                                                                     
                                                                                                                                                                                                             
[root@host-026 ~]# lvs -a -o +devices pv_shuffle_A/stripe                                                                                                                                                    
  LV     VG           Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                                                                                           
  stripe pv_shuffle_A -wi-ao---- 2.00g                                                     /dev/sda3(512),/dev/sda2(0),/dev/sda1(0)                                                                          
                                                                                                                                                                                                             
[root@host-026 ~]# mount | grep stripe                                                                                                                                                                       
/dev/mapper/pv_shuffle_A-stripe on /mnt/pv_shuffle_A_stripe type gfs2 (rw,relatime,seclabel)

# Unnamed attempt:
[root@host-026 ~]# pvmove -b -v /dev/sda2 /dev/sdh1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  pvmove in a shared VG requires a named LV.

# Shared lock w/ lv name attempt:
[root@host-026 ~]# pvmove -b -v -n pv_shuffle_A/stripe /dev/sda2 /dev/sdh1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  LV locked by other host: pv_shuffle_A/stripe
  pvmove in a shared VG requires exclusive lock on named LV.

[root@host-026 ~]# lvchange -an pv_shuffle_A/stripe
[root@host-026 ~]# lvchange -aye pv_shuffle_A/stripe

[root@host-026 ~]# pvmove -b -v /dev/sda2 /dev/sdh1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  pvmove in a shared VG requires a named LV.

[root@host-026 ~]#  pvmove -b -v -n pv_shuffle_A/stripe /dev/sda2 /dev/sdd2
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_A" metadata (seqno 27).
    Creating logical volume pvmove0
    Moving 171 extents of logical volume pv_shuffle_A/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_A/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_A-pvmove0
    Loading pv_shuffle_A-pvmove0 table (253:6)
    Loading pv_shuffle_A-stripe table (253:3)
    Suspending pv_shuffle_A-stripe (253:3) with device flush
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_A/pvmove0.
    Resuming pv_shuffle_A-pvmove0 (253:6)
    Loading pv_shuffle_A-pvmove0 table (253:6)
    Suppressed pv_shuffle_A-pvmove0 (253:6) identical table reload.
    Resuming pv_shuffle_A-stripe (253:3)
    Creating volume group backup "/etc/lvm/backup/pv_shuffle_A" (seqno 28).
    Checking progress before waiting every 15 seconds.

[root@host-026 ~]# lvs -a -o +devices pv_shuffle_A/stripe
  LV     VG           Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                   
  stripe pv_shuffle_A -wi-a----- 2.00g                                                     /dev/sda3(512),/dev/sdd2(257),/dev/sda1(0)


lvm2-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
lvm2-libs-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
lvm2-cluster-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-libs-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-event-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-event-libs-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 10:15:46 CDT 2017
cmirror-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
sanlock-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
sanlock-lib-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
lvm2-lockd-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017

Comment 8 errata-xmlrpc 2018-04-10 15:22:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0853

Note You need to log in before you can comment on or make changes to this bug.