Bug 1492897

Summary: lvmlockd needs to safeguard against pvmoves on shared activated volumes with no cmirrord option
Product: Red Hat Enterprise Linux 7 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
lvm2 sub component: LVM lock daemon / lvmlockd QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified CC: agk, heinzm, jbrassow, mcsontos, prajnoha, teigland, zkabelac
Version: 7.4   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.175-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 15:22:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2017-09-18 23:08:29 UTC
Description of problem:
This appears to be a current side effect of bug 1488592 as pvmove uses cmirrord when pvmove'ing shared activated linear/stripe volumes. Without a safeguard corruption can occur as shown here.


[root@host-113 ~]# pcs status
Cluster name: STSRHTS3525
Stack: corosync
Current DC: host-115 (version 1.1.16-12.el7-94ff4df) - partition with quorum
Last updated: Mon Sep 18 17:42:51 2017
Last change: Mon Sep 18 16:06:58 2017 by root via cibadmin on host-132

3 nodes configured
9 resources configured

Online: [ host-113 host-115 host-132 ]

Full list of resources:

 fence-host-115 (stonith:fence_xvm):    Started host-113
 fence-host-132 (stonith:fence_xvm):    Started host-115
 fence-host-113 (stonith:fence_xvm):    Started host-132
 Clone Set: dlm-clone [dlm]
     Started: [ host-113 host-115 host-132 ]
 Clone Set: lvmlockd-clone [lvmlockd]
     Started: [ host-113 host-115 host-132 ]


## Test setup:
creating lvm devices...
host-115: pvcreate /dev/sdg3 /dev/sdg2 /dev/sdg1 /dev/sdh3 /dev/sdh2 /dev/sdh1
host-115: vgcreate  --shared pv_shuffle_A /dev/sdg3 /dev/sdg2 /dev/sdg1 /dev/sdh3 /dev/sdh2 /dev/sdh1
host-115: vgchange --lock-start pv_shuffle_A
host-132: vgchange --lock-start pv_shuffle_A
host-113: vgchange --lock-start pv_shuffle_A

host-115: pvcreate /dev/sdc3 /dev/sdc2 /dev/sdc1 /dev/sdf3 /dev/sdf2 /dev/sdf1
host-115: vgcreate  --shared pv_shuffle_B /dev/sdc3 /dev/sdc2 /dev/sdc1 /dev/sdf3 /dev/sdf2 /dev/sdf1
host-115: vgchange --lock-start pv_shuffle_B
host-132: vgchange --lock-start pv_shuffle_B
host-113: vgchange --lock-start pv_shuffle_B

Create 4 free PV(s) on host-115
volume creation on pv_shuffle_A
        host-115 lvcreate --activate sy -n linear -L 2G pv_shuffle_A
        host-115 lvcreate --activate sy -i 3 -n stripe -L 2G pv_shuffle_A
Activating shared lockd volume remotely on all nodes
  WARNING: lvmetad is being updated, retrying (setup) for 10 more seconds.
volume creation on pv_shuffle_B
        host-115 lvcreate --activate sy -n linear -L 2G pv_shuffle_B
        host-115 lvcreate --activate sy -i 3 -n stripe -L 2G pv_shuffle_B

Activating shared lockd volume remotely on all nodes
filesystem creation on pv_shuffle_A
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:85637 /dev/pv_shuffle_A/linear -O
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:32249 /dev/pv_shuffle_A/stripe -O

filesystem creation on pv_shuffle_B
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:80203 /dev/pv_shuffle_B/linear -O
   mkfs.gfs2 -j 3 -p lock_dlm -t STSRHTS3525:53283 /dev/pv_shuffle_B/stripe -O


## Started IO load
[cmarthal@dhcp80-218 ~]$ nannyctl -n shuffle_io_load_21645 ps
host-115_pv_shuffle_A_linear  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_linear/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_A_linear  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_linear/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_A_linear  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_linear/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-115_pv_shuffle_A_stripe  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_stripe/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_A_stripe  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_stripe/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_A_stripe  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_A_stripe/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-115_pv_shuffle_B_linear  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_linear/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_B_linear  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_linear/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_B_linear  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_linear/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-115_pv_shuffle_B_stripe  [qarsh root@host-115 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_stripe/host-115_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-132_pv_shuffle_B_stripe  [qarsh root@host-132 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_stripe/host-132_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running 
host-113_pv_shuffle_B_stripe  [qarsh root@host-113 "/usr/tests/sts-rhel7.4/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/pv_shuffle_B_stripe/host-113_load | /usr/tests/sts-rhel7.4/bin/xdoio -v"]  Running


[root@host-113 ~]# vgextend pv_shuffle_B /dev/sde1 /dev/sda3
  Volume group "pv_shuffle_B" successfully extended

[root@host-113 ~]# lvs -a -o +devices
  LV     VG            Attr       LSize   Devices                                 
  linear pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(0)                            
  stripe pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear pv_shuffle_B  -wi-ao----   2.00g /dev/sdc3(0)                            
  stripe pv_shuffle_B  -wi-ao----   2.00g /dev/sdc3(512),/dev/sdc2(0),/dev/sdc1(0)

## Attempt pvmoves of the same device from different nodes in the cluster (note the 'Cluster mirror log daemon is not running':
[root@host-113 ~]# pvmove -v /dev/sdc3 /dev/sda3
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 5).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_B-pvmove0
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Loading pv_shuffle_B-linear table (253:4)
    Loading pv_shuffle_B-stripe table (253:5)
    Suspending pv_shuffle_B-linear (253:4) with device flush
    Suspending pv_shuffle_B-stripe (253:5) with device flush
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0.
    Resuming pv_shuffle_B-pvmove0 (253:6)
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Suppressed pv_shuffle_B-pvmove0 (253:6) identical table reload.
    Resuming pv_shuffle_B-linear (253:4)
    Resuming pv_shuffle_B-stripe (253:5)
    Creating volume group backup "/etc/lvm/backup/pv_shuffle_B" (seqno 6).
    Checking progress before waiting every 15 seconds.
  /dev/sdc3: Moved: 1.02%
  /dev/sdc3: Moved: 40.41%
  /dev/sdc3: Moved: 74.96%
  /dev/sdc3: Moved: 100.00%
    Polling finished successfully.


## This was started w/ the above pvmove still in progress. "Detected pvmove in progress for /dev/sdc3"
[root@host-115 ~]# lvs -a -o +devices
  LV        VG            Attr       LSize    Move       Devices                                 
  linear    pv_shuffle_A  -wi-ao----   2.00g             /dev/sdh3(0)                            
  stripe    pv_shuffle_A  -wi-ao----   2.00g             /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear    pv_shuffle_B  -wI-ao----   2.00g             pvmove0(0)                              
  [pvmove0] pv_shuffle_B  p-C---m---  <2.67g  /dev/sdc3  /dev/sdc3(0),/dev/sda3(0)               
  [pvmove0] pv_shuffle_B  p-C---m---  <2.67g  /dev/sdc3  /dev/sdc3(512),/dev/sda3(512)           
  stripe    pv_shuffle_B  -wI-ao----   2.00g             pvmove0(0),/dev/sdc2(0),/dev/sdc1(0)    

[root@host-115 ~]# pvmove -v /dev/sdc3 /dev/sda2
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  Detected pvmove in progress for /dev/sdc3
  Ignoring remaining command line arguments
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0.
    Creating pv_shuffle_B-pvmove0
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Resuming pv_shuffle_B-pvmove0 (253:6)
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Checking progress before waiting every 15 seconds.
  /dev/sdc3: Moved: 75.70%
  /dev/sdc3: Moved: 100.00%
    Polling finished successfully.

[root@host-115 ~]# lvs -a -o +devices
  LV     VG            Attr       LSize   Devices                                 
  linear pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(0)                            
  stripe pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(0)                            
  stripe pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(512),/dev/sdc2(0),/dev/sdc1(0)

[root@host-113 ~]# lvs -a -o +devices
  LV     VG            Attr       LSize   Devices                                 
  linear pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(0)                            
  stripe pv_shuffle_A  -wi-ao----   2.00g /dev/sdh3(512),/dev/sdg2(0),/dev/sdg1(0)
  linear pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(0)                            
  stripe pv_shuffle_B  -wi-ao----   2.00g /dev/sda3(512),/dev/sdc2(0),/dev/sdc1(0)

[root@host-115 ~]# dmsetup ls
pv_shuffle_A-stripe     (253:3)
pv_shuffle_A-linear     (253:2)
pv_shuffle_B-pvmove0    (253:6)  <- still existing pvmove left over from above attempt?
pv_shuffle_B-stripe     (253:5)
pv_shuffle_B-linear     (253:4)


[root@host-115 ~]#  pvmove -v /dev/sda3 /dev/sdc1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 9).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
  Insufficient suitable allocatable extents for logical volume : 171 more required
  Unable to allocate mirror extents for pv_shuffle_B/pvmove0.
  Failed to convert pvmove LV to mirrored

[root@host-115 ~]#  pvmove -v /dev/sda3 /dev/sdf3
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 9).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_B-pvmove0
  device-mapper: create ioctl on pv_shuffle_B-pvmove0 LVM-glzRa4uPoEr5obUcCkHlxamnnjo2Vm4RFCH4UH3oE6wuuDNQih2EbcUO4U0ko8E1 failed: Device or resource busy
  Failed to suspend linear
  ABORTING: Temporary pvmove mirror activation failed.

[root@host-115 ~]# dmsetup remove pv_shuffle_B-pvmove0

[root@host-115 ~]#  pvmove -v /dev/sda3 /dev/sdf3
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_B" metadata (seqno 9).
    Creating logical volume pvmove0
    Moving 512 extents of logical volume pv_shuffle_B/linear
    Moving 171 extents of logical volume pv_shuffle_B/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/linear.
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_B-pvmove0
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Loading pv_shuffle_B-linear table (253:4)
    Loading pv_shuffle_B-stripe table (253:5)
    Suspending pv_shuffle_B-linear (253:4) with device flush
    Suspending pv_shuffle_B-stripe (253:5) with device flush
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_B/pvmove0.
    Resuming pv_shuffle_B-pvmove0 (253:6)
    Loading pv_shuffle_B-pvmove0 table (253:6)
    Suppressed pv_shuffle_B-pvmove0 (253:6) identical table reload.
    Resuming pv_shuffle_B-linear (253:4)
    Resuming pv_shuffle_B-stripe (253:5)
    Creating volume group backup "/etc/lvm/backup/pv_shuffle_B" (seqno 10).
    Checking progress before waiting every 15 seconds.
  /dev/sda3: Moved: 0.73%
  /dev/sda3: Moved: 34.85%
  /dev/sda3: Moved: 60.18%
  /dev/sda3: Moved: 74.96%
  /dev/sda3: Moved: 100.00%
    Polling finished successfully.


## Filesystem appears gone now
[root@host-113 ~]# ls -lrt /mnt/pv_shuffle_B_linear
ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error
ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error
total 1000
-?????????? ? ?    ?          ?            ? host-132_load
-rw-rw-rw-. 1 root root 1000000 Sep 15 10:21 host-115_load
-rw-rw-rw-. 1 root root 1000000 Sep 18 17:32 host-113_load

[root@host-115 ~]# ls -lrt /mnt/pv_shuffle_B_linear
ls: cannot access /mnt/pv_shuffle_B_linear/host-113_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-115_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error
ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error
total 0
-????????? ? ? ? ?            ? host-132_load
-????????? ? ? ? ?            ? host-115_load
-????????? ? ? ? ?            ? host-113_load

[root@host-132 ~]# ls -lrt /mnt/pv_shuffle_B_linear
ls: cannot access /mnt/pv_shuffle_B_linear/host-113_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-115_load: Input/output error
ls: cannot access /mnt/pv_shuffle_B_linear/host-132_load: Input/output error
ls: reading directory /mnt/pv_shuffle_B_linear: Input/output error
total 0
-????????? ? ? ? ?            ? host-132_load
-????????? ? ? ? ?            ? host-115_load
-????????? ? ? ? ?            ? host-113_load


Sep 18 17:32:06 host-113 crmd[3033]:  notice: High CPU load detected: 4.310000
Sep 18 17:32:36 host-113 crmd[3033]:  notice: High CPU load detected: 4.250000
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: fatal: invalid metadata block#012GFS2: fsid=STSRHTS3525:80203.2:   bh = 159360 (magic number)#012GFS2: fsid=STSRHTS3525:80203.2:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 437
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: about to withdraw this file system
Sep 18 17:32:47 host-113 qarshd[4033]: Sending child 4038 signal 2
Sep 18 17:32:47 host-113 qarshd[4035]: Sending child 4040 signal 2
Sep 18 17:32:47 host-113 qarshd[4034]: Sending child 4039 signal 2
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: telling LM to unmount
Sep 18 17:32:47 host-113 kernel: GFS2: fsid=STSRHTS3525:80203.2: withdrawn
Sep 18 17:32:47 host-113 kernel: CPU: 0 PID: 3999 Comm: kworker/0:2H Not tainted 3.10.0-693.1.1.el7.x86_64 #1
Sep 18 17:32:47 host-113 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
Sep 18 17:32:47 host-113 kernel: Workqueue: glock_workqueue glock_work_func [gfs2]
Sep 18 17:32:47 host-113 kernel: ffff880020113000 000000006ad32300 ffff88002009fbf0 ffffffff816a3d91
Sep 18 17:32:47 host-113 kernel: ffff88002009fc68 ffffffffc03cb456 0000000000000030 ffff88002009fc98
Sep 18 17:32:47 host-113 kernel: ffff88002009fc20 000000006ad32300 ffff88002009fcb0 ffff88002009fc38
Sep 18 17:32:47 host-113 kernel: Call Trace:
Sep 18 17:32:47 host-113 kernel: [<ffffffff816a3d91>] dump_stack+0x19/0x1b
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03cb456>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03cb745>] gfs2_meta_check_ii+0x45/0x50 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03b2b59>] gfs2_meta_indirect_buffer+0xe9/0x150 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffff810bd504>] ? __wake_up+0x44/0x50
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aeb64>] gfs2_inode_refresh+0x34/0x90 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aec38>] inode_go_lock+0x78/0xf0 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03aae74>] do_promote+0x1b4/0x310 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03ac7f2>] finish_xmote+0x172/0x4c0 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffffc03ad192>] glock_work_func+0x122/0x150 [gfs2]
Sep 18 17:32:47 host-113 kernel: [<ffffffff810a881a>] process_one_work+0x17a/0x440
Sep 18 17:32:47 host-113 kernel: [<ffffffff810a94e6>] worker_thread+0x126/0x3c0
Sep 18 17:32:47 host-113 kernel: [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
Sep 18 17:32:47 host-113 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Sep 18 17:32:47 host-113 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Sep 18 17:32:47 host-113 kernel: [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
Sep 18 17:32:47 host-113 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Sep 18 17:33:06 host-113 crmd[3033]:  notice: High CPU load detected: 3.120000




Version-Release number of selected component (if applicable):
3.10.0-693.1.1.el7.x86_64

lvm2-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
lvm2-libs-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
lvm2-cluster-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-libs-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-event-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-event-libs-1.02.140-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 10:15:46 CDT 2017
cmirror-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017
sanlock-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
sanlock-lib-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
lvm2-lockd-2.02.171-8.el7    BUILT: Wed Jun 28 13:28:58 CDT 2017

Comment 2 David Teigland 2017-09-20 17:00:59 UTC
Two commits addressing mirror and pvmove issues in a shared VG:

https://sourceware.org/git/?p=lvm2.git;a=commit;h=518a8e8cfbb672c2bf5e3455f1fe7cd8d94eb5b0

https://sourceware.org/git/?p=lvm2.git;a=commit;h=f2ee0e7aca6020cf273f22dc5bca20bfa3c89051

LVs with 'mirror' type can now be activated with a shared lock (-asy) from multiple hosts, and lvm correctly uses cmirrord for them.  This only applies to shared VGs with lock_type dlm (sanlock is excluded since the combination of sanlock and corosync is not useful.)

pvmove in a shared VG is restricted to cases where an LV name is specified (pvmove -n LV).  A further restriction is that the node running pvmove must be able to exclusively activate the LV.  This means a shared VG does not support pvmove of an entire PV, and does not support pvmove of an LV active on multiple nodes, e.g. under gfs.

Comment 5 Corey Marthaler 2017-10-16 15:23:10 UTC
Verified that pvmoves on shared locked LVs (w/ or w/o a given -n lv name arg) is no longer allowed. LVs need to be exclusively locked (or converted to ex lock by being the only node w/ a shared lock) before a pvmove can take place.

   limitations of lockd VGs                                                                                                                                                                                  
       Things that do not yet work in lockd VGs:                                                                                                                                                             
       [...]                                                                                                                                                                                                 
       · pvmove of entire PVs, or under LVs activated with shared locks                                                                                                                                     
                                                                                                                                                                                                             
[root@host-026 ~]# lvs -a -o +devices pv_shuffle_A/stripe                                                                                                                                                    
  LV     VG           Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                                                                                           
  stripe pv_shuffle_A -wi-ao---- 2.00g                                                     /dev/sda3(512),/dev/sda2(0),/dev/sda1(0)                                                                          
                                                                                                                                                                                                             
[root@host-026 ~]# mount | grep stripe                                                                                                                                                                       
/dev/mapper/pv_shuffle_A-stripe on /mnt/pv_shuffle_A_stripe type gfs2 (rw,relatime,seclabel)

# Unnamed attempt:
[root@host-026 ~]# pvmove -b -v /dev/sda2 /dev/sdh1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  pvmove in a shared VG requires a named LV.

# Shared lock w/ lv name attempt:
[root@host-026 ~]# pvmove -b -v -n pv_shuffle_A/stripe /dev/sda2 /dev/sdh1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  LV locked by other host: pv_shuffle_A/stripe
  pvmove in a shared VG requires exclusive lock on named LV.

[root@host-026 ~]# lvchange -an pv_shuffle_A/stripe
[root@host-026 ~]# lvchange -aye pv_shuffle_A/stripe

[root@host-026 ~]# pvmove -b -v /dev/sda2 /dev/sdh1
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  pvmove in a shared VG requires a named LV.

[root@host-026 ~]#  pvmove -b -v -n pv_shuffle_A/stripe /dev/sda2 /dev/sdd2
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "pv_shuffle_A" metadata (seqno 27).
    Creating logical volume pvmove0
    Moving 171 extents of logical volume pv_shuffle_A/stripe
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_A/stripe.
    Setting up pvmove in on-disk volume group metadata.
    Creating pv_shuffle_A-pvmove0
    Loading pv_shuffle_A-pvmove0 table (253:6)
    Loading pv_shuffle_A-stripe table (253:3)
    Suspending pv_shuffle_A-stripe (253:3) with device flush
    activation/volume_list configuration setting not defined: Checking only host tags for pv_shuffle_A/pvmove0.
    Resuming pv_shuffle_A-pvmove0 (253:6)
    Loading pv_shuffle_A-pvmove0 table (253:6)
    Suppressed pv_shuffle_A-pvmove0 (253:6) identical table reload.
    Resuming pv_shuffle_A-stripe (253:3)
    Creating volume group backup "/etc/lvm/backup/pv_shuffle_A" (seqno 28).
    Checking progress before waiting every 15 seconds.

[root@host-026 ~]# lvs -a -o +devices pv_shuffle_A/stripe
  LV     VG           Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                   
  stripe pv_shuffle_A -wi-a----- 2.00g                                                     /dev/sda3(512),/dev/sdd2(257),/dev/sda1(0)


lvm2-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
lvm2-libs-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
lvm2-cluster-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-libs-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-event-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-event-libs-1.02.144-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 10:15:46 CDT 2017
cmirror-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017
sanlock-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
sanlock-lib-3.5.0-1.el7    BUILT: Wed Apr 26 09:37:30 CDT 2017
lvm2-lockd-2.02.175-2.el7    BUILT: Fri Oct 13 06:31:22 CDT 2017

Comment 8 errata-xmlrpc 2018-04-10 15:22:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0853