RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1595875 - split off --type mirror images are not being properly activated online
Summary: split off --type mirror images are not being properly activated online
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.6
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: rc
: ---
Assignee: Zdenek Kabelac
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
: 1597307 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-27 16:00 UTC by Corey Marthaler
Modified: 2021-09-03 12:48 UTC (History)
8 users (show)

Fixed In Version: lvm2-2.02.180-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 11:03:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
verbose lvconvert attempt (240.03 KB, text/plain)
2018-06-27 16:26 UTC, Corey Marthaler
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3193 0 None None None 2018-10-30 11:04:28 UTC

Description Corey Marthaler 2018-06-27 16:00:46 UTC
Description of problem:

raid1 split images properly show up as usable in /dev/vg/lv, mirror type split images do not, even though they're listed as "active".


### --type raid1                                                                                                                                                                                                    
                                                                                                                                                                                                                           
SCENARIO - [sequentially_split_off_all_raid1_pvs]                                                                                                                                                                          
Create a raid1 with many legs and then sequentially split off each one of the PVs                                                                                                                                                 
host-093: lvcreate --activate y --type raid1 -m 4 -n split_pvs_sequentially -L 300M split_image                                                                                                                                   
Waiting until all mirror|raid volumes become fully syncd...                                                                                                                                                                       
   0/1 mirror(s) are fully synced: ( 76.58% )                                                                                                                                                                                          
   1/1 mirror(s) are fully synced: ( 100.00% )                                                                                                                                                                                         
                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                           
[root@host-093 ~]# lvs -a -o +devices                                                                                                                                                                                                      
  LV                                VG          Attr       LSize    Log Cpy%Sync Devices                                                                                                                                                   
  split_pvs_sequentially            split_image rwi-a-r--- 300.00m      100.00   split_pvs_sequentially_rimage_0(0),split_pvs_sequentially_rimage_1(0),split_pvs_sequentially_rimage_2(0),split_pvs_sequentially_rimage_3(0),split_pvs_sequentially_rimage_4(0)                                                                                                                                                                                                                                       
  [split_pvs_sequentially_rimage_0] split_image iwi-aor--- 300.00m               /dev/sdc1(1)                                                                                                                                                  
  [split_pvs_sequentially_rimage_1] split_image iwi-aor--- 300.00m               /dev/sdb1(1)                                                                                                                                                    
  [split_pvs_sequentially_rimage_2] split_image iwi-aor--- 300.00m               /dev/sdd1(1)                                                                                                                                                    
  [split_pvs_sequentially_rimage_3] split_image iwi-aor--- 300.00m               /dev/sda1(1)                                                                                                                                                      
  [split_pvs_sequentially_rimage_4] split_image iwi-aor--- 300.00m               /dev/sde1(1)                                                                                                                                                      
  [split_pvs_sequentially_rmeta_0]  split_image ewi-aor---   4.00m               /dev/sdc1(0)                                                                                                                                                        
  [split_pvs_sequentially_rmeta_1]  split_image ewi-aor---   4.00m               /dev/sdb1(0)                                                                                                                                                          
  [split_pvs_sequentially_rmeta_2]  split_image ewi-aor---   4.00m               /dev/sdd1(0)                                                                                                                                                           
  [split_pvs_sequentially_rmeta_3]  split_image ewi-aor---   4.00m               /dev/sda1(0)
  [split_pvs_sequentially_rmeta_4]  split_image ewi-aor---   4.00m               /dev/sde1(0)

[root@host-093 ~]# lvconvert --yes --splitmirrors 1 --name new0 split_image/split_pvs_sequentially /dev/sda1
[root@host-093 ~]# ls /dev/split_image/new0 
/dev/split_image/new0
[root@host-093 ~]# dmsetup status | grep new0
split_image-new0: 0 614400 linear 




### --type mirror

SCENARIO - [sequentially_split_off_all_pvs]
Create a mirror with many legs and then sequentially split off each one of the PVs
host-093: lvcreate --type mirror -m 4 -n split_pvs_sequentially -L 300M split_image
Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 90.67% )
   1/1 mirror(s) are fully synced: ( 100.00% )
[root@host-093 ~]# lvs -a -o +devices
  LV                                VG          Attr       LSize    Log                           Cpy%Sync Devices
  split_pvs_sequentially            split_image mwi-a-m--- 300.00m  [split_pvs_sequentially_mlog] 100.00   split_pvs_sequentially_mimage_0(0),split_pvs_sequentially_mimage_1(0),split_pvs_sequentially_mimage_2(0),split_pvs_sequentially_mimage_3(0),split_pvs_sequentially_mimage_4(0)
  [split_pvs_sequentially_mimage_0] split_image iwi-aom--- 300.00m                                         /dev/sdc1(0)
  [split_pvs_sequentially_mimage_1] split_image iwi-aom--- 300.00m                                         /dev/sdb1(0)
  [split_pvs_sequentially_mimage_2] split_image iwi-aom--- 300.00m                                         /dev/sdd1(0)
  [split_pvs_sequentially_mimage_3] split_image iwi-aom--- 300.00m                                         /dev/sda1(0)
  [split_pvs_sequentially_mimage_4] split_image iwi-aom--- 300.00m                                         /dev/sde1(0)
  [split_pvs_sequentially_mlog]     split_image lwi-aom---   4.00m                                         /dev/sdh1(0)

[root@host-093 ~]# lvconvert --yes --splitmirrors 1 --name new0 split_image/split_pvs_sequentially /dev/sda1
  Logical volume split_image/split_pvs_sequentially converted.
[root@host-093 ~]# ls /dev/split_image/new0
ls: cannot access /dev/split_image/new0: No such file or directory
[root@host-093 ~]# dmsetup status | grep new0
[root@host-093 ~]# lvs -a -o +devices split_image/new0
  LV   VG          Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices     
  new0 split_image -wi-a----- 300.00m                                                     /dev/sda1(0)



Version-Release number of selected component (if applicable):
3.10.0-915.el7.x86_64

lvm2-2.02.179-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
lvm2-libs-2.02.179-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
lvm2-cluster-2.02.179-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
lvm2-lockd-2.02.179-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
lvm2-python-boom-0.9-3.el7    BUILT: Tue Jun 26 08:43:24 CDT 2018
cmirror-2.02.179-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
device-mapper-1.02.148-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
device-mapper-libs-1.02.148-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
device-mapper-event-1.02.148-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
device-mapper-event-libs-1.02.148-4.el7    BUILT: Tue Jun 26 08:35:45 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017

Comment 3 Corey Marthaler 2018-06-27 16:26:11 UTC
Created attachment 1455081 [details]
verbose lvconvert attempt

Comment 4 Corey Marthaler 2018-06-27 16:32:34 UTC
FWIW, reactivating the volume will properly activate it:


[root@host-093 ~]# ls /dev/split_image/new0
ls: cannot access /dev/split_image/new0: No such file or directory
[root@host-093 ~]# lvchange -an split_image/new0
[root@host-093 ~]# lvchange -ay split_image/new0
[root@host-093 ~]# lvs -a -o +devices split_image/new0
  LV   VG          Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices     
  new0 split_image -wi-a----- 300.00m                                                     /dev/sde1(0)
[root@host-093 ~]# ls /dev/split_image/new0
/dev/split_image/new0

Comment 6 David Teigland 2018-07-18 18:47:18 UTC
This is a regression from

commit 29b2cfba06ee849774025c50599edb1c7587b7d9
Author: Zdenek Kabelac <zkabelac>
Date:   Tue Mar 13 12:48:36 2018 +0100

    mirror: correct locking for mirror log initialization
    
    The code was not acking proper lock holding LVs when trying to
    initialize mirror log to predefined values.

https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=29b2cfba06ee849774025c50599edb1c7587b7d9

There is a lot going on in that commit, so it's not clear to me what part of it is at fault.

Comment 7 David Teigland 2018-07-18 18:50:14 UTC
*** Bug 1597307 has been marked as a duplicate of this bug. ***

Comment 8 David Teigland 2018-07-18 19:01:10 UTC
Probably this hunk at fault in _split_mirror_images

-       /*
-        * Recycle newly split LV so it is properly renamed.
-        *   Cluster requires the extra deactivate/activate calls.
-        */
-       if (vg_is_clustered(lv->vg) &&
-           (!deactivate_lv(cmd, new_lv) ||
-            !_activate_lv_like_model(lv, new_lv))) {
-               log_error("Failed to rename newly split LV in the kernel");
-               return 0;
-       }
-       if (!suspend_lv(cmd, new_lv) || !resume_lv(cmd, new_lv)) {
+       act = lv_is_active(lv_lock_holder(lv));
+
+       if (act && !_activate_lv_like_model(lv, new_lv)) {
                log_error("Failed to rename newly split LV in the kernel");
                return 0;
        }

Comment 9 Zdenek Kabelac 2018-07-26 18:13:25 UTC
Seems like the fix mentioned in comment 6 has fixed long outstanding bug where volume has been left active in dm table without taking explicit lock for it's activation.

If the wanted behavior is to have the spitted volume to be actually active - it would need to be explicitly activated - the question is how to do it.


Mirror can be cluster-wide activated (for lvm2 with cluster activation support) - but where it's splitted mirror image should be activated ?

If the mirror LV is active on node A - and split is executed on node B - where is it meant to be activated ?

IMHO the current behavior where splitted LV is left inactive is the only correct logic we can provide considering there is supported clustered activation and it's been before  'leaking' device being activated without holding proper lock on proper nodes.

But if there is some strong need to support some sort activation - we need to define where the splitted LV is supposed to appear active - easiest for implementation seems to be implementing  exclusive activation for the node executing split operation.

Comment 10 Zdenek Kabelac 2018-08-07 22:40:06 UTC
Should be resolved by this patch that fixes preload of splitted mirror leg  and adds deactivation for locally active device before reactivation.

https://www.redhat.com/archives/lvm-devel/2018-August/msg00014.html

Comment 12 Corey Marthaler 2018-08-23 16:04:47 UTC
Image splitting scenarios are once again work w/ the latest rpms. Marking verified.

3.10.0-937.el7.x86_64

lvm2-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
lvm2-libs-2.02.180-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-libs-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-event-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-event-libs-1.02.149-5.el7    BUILT: Tue Aug 21 11:29:37 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017



SCENARIO - [sequentially_split_off_all_pvs]
Create a mirror with many legs and then sequentially split off each one of the PVs
hayes-01: lvcreate --activate y --type mirror -m 4 -n split_pvs_sequentially -L 300M split_image
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

splitting off legs:
         /dev/sdj1 /dev/sdg1 /dev/sdk1 /dev/sdc1

Deactivating LV split_image/new0 on hayes-01... and removing
Deactivating LV split_image/new1 on hayes-01... and removing
Deactivating LV split_image/new2 on hayes-01... and removing
Deactivating LV split_image/new3 on hayes-01... and removing
Deactivating LV split_image/split_pvs_sequentially on hayes-01... and removing

Comment 14 errata-xmlrpc 2018-10-30 11:03:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3193


Note You need to log in before you can comment on or make changes to this bug.