Bug 1711427

Not reproducable by me.  Corey is gathering more info.

100% reproducible for me...  will copy comments from bug 1713716 and close that bug as a dup.

(copied from bug 1713716c7)

this scenario should never trigger dm-log-userspace to be used.  That is a cluster mirror component.  From what I can tell, you are activating the mirror exclusively initially.  Comment 3 seems to switch to non-exclusive activation.  This is fine, but I want to point out that there are probably two bugs here, 1) an undesired switch from exclusive to non-exclusive and 2) whatever is going wrong with the mirror conversion.

I'm not sure what is going wrong with the mirror after conversion... the primary is obviously misbehaving.  At that point, all bets are off.

(copied from bug 1713716c8)

couple bizarre things:

1) I'm pretty sure a linear -> mirror didn't have to be exclusively activated before... (unrelated to this bug, of course)
[root@bp-01 ~]# lvconvert -m1 vg/lv
Are you sure you want to convert linear LV vg/lv to raid1 with 2 images enhancing resilience? [y/n]: y
  vg/lv must be active exclusive locally to perform this operation.
This occurs because RAID1 (the default type) does not support cluster operation... This could be tough for a user to figure out.  However, it's not very important since cluster mirroring goes away in RHEL8.

2) WTF?  In REHL7.7?
[root@bp-01 ~]# lvconvert --type mirror -m1 vg/lv
  Shared cluster mirrors are not available.
2.1) Works just fine if switching to exclusive...
[root@bp-01 ~]# lvchange -an vg
[root@bp-01 ~]# lvchange -aey vg/lv
[root@bp-01 ~]# lvconvert --type mirror -m1 vg/lv
  Logical volume vg/lv being converted.
  vg/lv: Converted: 1.60%
  vg/lv: Converted: 100.00%
[root@bp-01 ~]# dmsetup table | grep vg-
vg-lv: 0 1024000 mirror disk 2 253:4 4096 2 253:5 0 253:6 0 1 handle_errors
vg-lv_mimage_1: 0 1024000 linear 8:33 2048
vg-lv_mimage_0: 0 1024000 linear 8:17 2048
vg-lv_mlog: 0 8192 linear 8:129 2048
... tables are correct, but the log entries are not right!
[104885.306295] device-mapper: dm-log-userspace: version 1.3.0 loaded
[104885.312750] device-mapper: dm-log-userspace: Unable to send log request [1] to userspace: -3
[104885.321288] device-mapper: dm-log-userspace: Userspace log server not found
[104885.328333] device-mapper: table: 253:3: mirror: Error creating mirror dirty log
[104885.335810] device-mapper: ioctl: error adding target to table
^^^ This should not be happening.

We should probably stop at problem #2 before moving on to a mirror-to-mirror upconvert... something is already pretty messed up.

*** Bug 1713716 has been marked as a duplicate of this bug. ***

Non-clustered scenario tested on el7, works fine w/o any primary errors like the 'Unable to read primary mirror during recovery':
- create linear + mkfs.xfs + mount + rsync
- wait for dirty rsync loadsync
- sync & lvconvert -y -m1 --type mirror ...
- remove files and rsync again
- sync & lvconvert -y -m4 --type mirror ...

(In reply to Jonathan Earl Brassow from comment #6)
> 
> 2) WTF?  In REHL7.7?
> [root@bp-01 ~]# lvconvert --type mirror -m1 vg/lv
>   Shared cluster mirrors are not available.

The error message goes away when 'cmirrord' is properly running.

Perhaps the message should be "cmirrord is not running" instead of "Shared cluster mirrors are not available."

My mistake though.

> 2.1) Works just fine if switching to exclusive...
> [root@bp-01 ~]# lvchange -an vg
> [root@bp-01 ~]# lvchange -aey vg/lv
> [root@bp-01 ~]# lvconvert --type mirror -m1 vg/lv
>   Logical volume vg/lv being converted.
>   vg/lv: Converted: 1.60%
>   vg/lv: Converted: 100.00%
> [root@bp-01 ~]# dmsetup table | grep vg-
> vg-lv: 0 1024000 mirror disk 2 253:4 4096 2 253:5 0 253:6 0 1 handle_errors
> vg-lv_mimage_1: 0 1024000 linear 8:33 2048
> vg-lv_mimage_0: 0 1024000 linear 8:17 2048
> vg-lv_mlog: 0 8192 linear 8:129 2048
> ... tables are correct, but the log entries are not right!
> [104885.306295] device-mapper: dm-log-userspace: version 1.3.0 loaded
> [104885.312750] device-mapper: dm-log-userspace: Unable to send log request
> [1] to userspace: -3
> [104885.321288] device-mapper: dm-log-userspace: Userspace log server not
> found
> [104885.328333] device-mapper: table: 253:3: mirror: Error creating mirror
> dirty log
> [104885.335810] device-mapper: ioctl: error adding target to table
> ^^^ This should not be happening.
> 
> We should probably stop at problem #2 before moving on to a mirror-to-mirror
> upconvert... something is already pretty messed up.

FWIW, the error messages go away when 'cmirrord' is running, but we shouldn't ever be using 'cmirrord' in the first place.

Up-converting from 2-way cluster mirror to 4-way cluster mirror "works", but you get these disturbing messages:
bp-01 login: [108819.675638] device-mapper: raid1: Unable to read primary mirror during recovery
[108819.683025] device-mapper: raid1: Primary mirror (253:7) failed while out-of-sync: Reads may fail.
[108819.688073] device-mapper: raid1: Mirror read failed.
[108819.688168] device-mapper: raid1: Mirror read failed.
[108819.688173] Buffer I/O error on dev dm-3, logical block 127984, async page read
[108822.734896] device-mapper: raid1: Mirror read failed.
[108822.740465] device-mapper: raid1: Mirror read failed.
[108822.745607] Buffer I/O error on dev dm-3, logical block 127984, async page read
[108838.035814] device-mapper: raid1: log presuspend failed
[108838.041494] device-mapper: raid1: log presuspend failed
[108838.046828] device-mapper: raid1: log postsuspend failed

There is good probability for corruption in there - especially around the front of the device.  My guess is that the mirror is being activated with a strange table ordering... a -vvvv trace would probably show the problematic table loads.

Looking into this commit as possible cause.  It seems to drop setting mirr_laopts.exclusive.  It is laopts.exclusive that determines whether a clustered log is used or not (see how _add_log() sets the clustered flag).

https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=a8921be641afe865c177e11b8859f4b937f76995

The reason that a clustered log is being used in _add_log() is because laopts.exclusive is not set.

In the commit referenced above, this appears to be the relevant change:

-               if (!vg_write_lock_held() && lv_is_mirror(lv)) {
-                       mirr_laopts.exclusive = lv_is_active_exclusive_locally(lv) ? 1 : 0;

After restoring these lines, along with debugging, the problem is rather in the fact that lv_is_active_exclusive_locally does not work.  I added this code back and tested:

+               if (!vg_write_lock_held() && lv_is_mirror(lv)) {
+                       /* mirr_laopts.exclusive = lv_is_active_exclusive_locally(lv) ? 1 : 0; */
+                       mirr_laopts.exclusive = 1;
+                       log_print("setting laopts.exclusive %d is_active_ex %d is_active_ex_local %d in monitor_dev_for_events",
+                                 mirr_laopts.exclusive,
+                                 lv_is_active_exclusive(lv),
+                                 lv_is_active_exclusive_locally(lv));
+               }

This forces the exclusive flag, and the problem is resolved, proving that the problem is the failure to set the flag.  Further, it shows that lv_is_active_exclusive_locally would not work for setting the exclusive flag:

#activate/activate.c:2057    setting laopts.exclusive 1 is_active_ex 0 is_active_ex_local 0 in monitor_dev_for_events

I don't know if lv_is_active_exclusive_locally has ever worked correctly in clvmd, or what the prospects might be for making it work.  It seems doubtful to me (I suspect it would have been fixed if it was possible.)  Unless someone else knows of a solution to this, we should disable mirror conversions of exclusively active LVs in a clustered VG.

Locally exclusive is defined as  'exclusively' locked LV which is active locally  - I'd assume this function always worked.

What is much hard is to activate volume locally exclusively - this can be prohibity by various tagging rules - but the basic rule is - every 'exclusive' activation is first tried locally.

If the requested activation was exclusive & locally - it needs to be checked once the exclusive lock is obtained whether device is present on local machine.

I need to check what has changed in that code.

pushed to stable-2.02
https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=8623e336513c6ac6fcb24aa7e2ef10b8a3a36c59

    lvconvert: disable linear to mirror of active LV in cluster VG
    
    Avoid bug 1711427 in which an exclusively active linear LV in a
    clustered VG, when upconverted to a mirror, will mistakenly use
    a userspace/cmirror log in _add_log() beause laopts.exclusive is
    not set.


Test result:

[root@null-02 ~]# lvcreate -n lv6 -aey -L64M cc
  Logical volume "lv6" created.

[root@null-02 ~]# dlm_tool lockdebug clvmd
Resource len 64  "oDBxuMigltJwjQtrHUMLSoz0XRbYMmDtgFMtb2HLjhxMQH6HXtCNtmQL692e70Y3"
Master           
Granted
00000002 EX                                    

[root@null-02 ~]# dmsetup table cc-lv6
0 131072 linear 8:16 655744

[root@null-02 ~]# lvconvert --type mirror -m1 cc/lv6
  Cannot convert active LV to mirror in clustered VG.

[root@null-02 ~]# lvchange -an cc/lv6

[root@null-02 ~]# lvconvert --type mirror -m1 cc/lv6
  Logical volume cc/lv6 being converted.
  Conversion starts after activation.

[root@null-02 ~]# lvchange -aey cc/lv6

[root@null-02 ~]# dmsetup table cc-lv6
0 131072 mirror disk 2 253:3 1024 2 253:4 0 253:5 0 1 handle_errors

Every failure I've seen in this bug and in bug 1713716 began with converting an exclusive linear LV to a mirror.  AFAICT, subsequent conversions of the same LV to include more legs would inherit problems from the original conversion (which is now prevented.)  Please let me know if I have missed any unique failures that would not be avoided by prevening the original convert.

(In reply to David Teigland from comment #19)
> Every failure I've seen in this bug and in bug 1713716 began with converting
> an exclusive linear LV to a mirror.  AFAICT, subsequent conversions of the
> same LV to include more legs would inherit problems from the original
> conversion (which is now prevented.)  Please let me know if I have missed
> any unique failures that would not be avoided by prevening the original
> convert.

Meaning you can reenable upconversion of exclusive linear LVs in a cluster again, i.e. revert patch as of c#16 ?

(In reply to Heinz Mauelshagen from comment #20)
> (In reply to David Teigland from comment #19)
> > Every failure I've seen in this bug and in bug 1713716 began with converting
> > an exclusive linear LV to a mirror.  AFAICT, subsequent conversions of the
> > same LV to include more legs would inherit problems from the original
> > conversion (which is now prevented.)  Please let me know if I have missed
> > any unique failures that would not be avoided by prevening the original
> > convert.
> 
> Meaning you can reenable upconversion of exclusive linear LVs in a cluster
> again, i.e. revert patch as of c#16 ?

Sorry, I meant c#17

How is this affecting dmevent upconverting mirror after lost leg?

(In reply to Heinz Mauelshagen from comment #21)
> (In reply to Heinz Mauelshagen from comment #20)
> > (In reply to David Teigland from comment #19)
> > > Every failure I've seen in this bug and in bug 1713716 began with converting
> > > an exclusive linear LV to a mirror.  AFAICT, subsequent conversions of the
> > > same LV to include more legs would inherit problems from the original
> > > conversion (which is now prevented.)  Please let me know if I have missed
> > > any unique failures that would not be avoided by prevening the original
> > > convert.
> > 
> > Meaning you can reenable upconversion of exclusive linear LVs in a cluster
> > again, i.e. revert patch as of c#16 ?
> 
> Sorry, I meant c#17

What I meant is that all of the problems in this bug began with the linear->mirror conversion.  That meant that they were all avoided with the patch to disallow the convert.

Now it seems we can revert the commit in comment 17, and allow the convert again, because another patch from another bug appears to fix it.

pushed to stable
https://sourceware.org/git/?p=lvm2.git;a=commit;h=f3a87a2c2e339328ea4d7448f43d5317806a6b24

Revert "lvconvert: disable linear to mirror of active LV in cluster VG"

This reverts commit 8623e336513c6ac6fcb24aa7e2ef10b8a3a36c59.

The problem this was avoiding now seems to be fixed by
commit d6bce036155ae973c869bdce3ca5f824f933f599
  mirror: fix monitoring change

(In reply to David Teigland from comment #23)
> (In reply to Heinz Mauelshagen from comment #21)
> > (In reply to Heinz Mauelshagen from comment #20)
> > > (In reply to David Teigland from comment #19)
> > > > Every failure I've seen in this bug and in bug 1713716 began with converting
> > > > an exclusive linear LV to a mirror.  AFAICT, subsequent conversions of the
> > > > same LV to include more legs would inherit problems from the original
> > > > conversion (which is now prevented.)  Please let me know if I have missed
> > > > any unique failures that would not be avoided by prevening the original
> > > > convert.
> > > 
> > > Meaning you can reenable upconversion of exclusive linear LVs in a cluster
> > > again, i.e. revert patch as of c#16 ?
> > 
> > Sorry, I meant c#17
> 
> What I meant is that all of the problems in this bug began with the
> linear->mirror conversion.  That meant that they were all avoided with the
> patch to disallow the convert.
> 
> Now it seems we can revert the commit in comment 17, and allow the convert
> again, because another patch from another bug appears to fix it.

For reference: we assume it to be the fix for bz#1719469 as commented in https://bugzilla.redhat.com/show_bug.cgi?id=1719469#c28

Using the stable branch, I've tested this and it seems to work.  Although, it does require the LV to be inactive when up-converting from linear to mirror:

[root@bp-01 ~]# lvs vg
  LV     VG Attr       LSize Pool Origin Data%  Meta%  Move Log           Cpy%Sync Convert
  lv     vg -wi-a----- 2.00g
  mirror vg mwi-a-m--- 2.00g                                [mirror_mlog] 36.33
[root@bp-01 ~]# lvconvert --type mirror -m 1 vg/lv
  Cannot convert active LV to mirror in clustered VG.
[root@bp-01 ~]# lvchange -an vg/lv
[root@bp-01 ~]# lvconvert --type mirror -m 1 vg/lv
  Logical volume vg/lv being converted.
  Conversion starts after activation.
[root@bp-01 ~]# lvchange -ay vg/lv
[root@bp-01 ~]# lvs vg
  LV     VG Attr       LSize Pool Origin Data%  Meta%  Move Log           Cpy%Sync Convert
  lv     vg mwi-a-m--- 2.00g                                [lv_mlog]     17.38
  mirror vg mwi-a-m--- 2.00g                                [mirror_mlog] 100.00
[root@bp-01 ~]# lvchange -an vg/lv
[root@bp-01 ~]# lvchange -aey vg/lv
[root@bp-01 ~]# lvs vg
  LV     VG Attr       LSize Pool Origin Data%  Meta%  Move Log           Cpy%Sync Convert
  lv     vg mwi-a-m--- 2.00g                                [lv_mlog]     100.00
  mirror vg mwi-a-m--- 2.00g                                [mirror_mlog] 100.00


While it should be fixed, I'm ok with needing to deactivate to perform the upconvert.  Need to ensure that it handles downconvert online though, otherwise, fault tolerance will not work.

Online down-convert works just fine, so fault tolerance is still fine.

[root@bp-01 ~]# off.sh sdc
Turning off sdc
[root@bp-01 ~]# dd if=/dev/zero of=/dev/vg/lv bs=4M count=10
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 53.8665 s, 779 kB/s
[root@bp-01 ~]# devices vg
  Couldn't find device with uuid mny1dl-pXGT-43gM-eNRU-WinD-gayV-1jiPo7.
  WARNING: Couldn't find all devices for LV vg/mirror_mimage_1 while checking used and assumed devices.
  LV                Attr       Cpy%Sync Devices
  lv                -wi-a-----          /dev/sdb1(0)

... and it maintains the exclusive nature of the mirror:
[root@bp-02 ~]# devices vg
  Couldn't find device with uuid mny1dl-pXGT-43gM-eNRU-WinD-gayV-1jiPo7.
  WARNING: Couldn't find all devices for LV vg/mirror_mimage_1 while checking used and assumed devices.
  LV                Attr       Cpy%Sync Devices
  lv                -wi-------          /dev/sdb1(0)

*** Bug 1719469 has been marked as a duplicate of this bug. ***

This scenario passes once again with the latest rpms. Marking verified.

3.10.0-1057.el7.x86_64

lvm2-2.02.185-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
lvm2-libs-2.02.185-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
lvm2-cluster-2.02.185-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
lvm2-lockd-2.02.185-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
lvm2-python-boom-0.9-18.el7    BUILT: Fri Jun 21 04:18:58 CDT 2019
cmirror-2.02.185-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
device-mapper-1.02.158-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
device-mapper-libs-1.02.158-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
device-mapper-event-1.02.158-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
device-mapper-event-libs-1.02.158-2.el7    BUILT: Fri Jun 21 04:18:48 CDT 2019
device-mapper-persistent-data-0.8.5-1.el7    BUILT: Mon Jun 10 03:58:20 CDT 2019



Scenario linear_conversion: Convert Linear volume
********* Take over hash info for this scenario *********
* from type:    linear
* to type:      mirror
* from legs:    0
* to legs:      2
* from region:  0
* to region:    0
* contiguous:   1
******************************************************


Creating original volume on harding-03...
harding-03: lvcreate -aye --type linear   -n takeover -L 2.75G centipede2
WARNING: xfs signature detected on /dev/centipede2/takeover at offset 0. Wipe it? [y/n]: [n]
  Aborted wiping of xfs.
  1 existing signature left on the device.

Current volume device structure:
  LV       Type   Attr       LSize    Cpy%Sync Devices               
  takeover linear -wi-a-----    2.75g          /dev/mapper/mpatha1(0)

Creating xfs on top of mirror(s) on harding-03...
warning: device is not properly aligned /dev/centipede2/takeover
Mounting mirrored xfs filesystems on harding-03...

Writing verification files (checkit) to mirror(s) on...
        ---- harding-03 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- harding-03 ----

TAKEOVER: lvconvert --force --yes  -m 1 --type mirror centipede2/takeover
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Current volume device structure:
  LV                  Type   Attr       LSize    Cpy%Sync Devices                                  
  takeover            mirror mwi-aom---    2.75g 100.00   takeover_mimage_0(0),takeover_mimage_1(0)
  [takeover_mimage_0] linear iwi-aom---    2.75g          /dev/mapper/mpatha1(0)                   
  [takeover_mimage_1] linear iwi-aom---    2.75g          /dev/mapper/mpathb1(0)                   
  [takeover_mlog]     linear lwi-aom---    4.00m          /dev/mapper/mpathh1(0)                   

Verifying files (checkit) on mirror(s) on...
        ---- harding-03 ----

RESHAPE: lvconvert --yes  -m 2 centipede2/takeover
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Current volume device structure:
  LV                  Type   Attr       LSize    Cpy%Sync Devices                                                       
  takeover            mirror mwi-aom---    2.75g 100.00   takeover_mimage_0(0),takeover_mimage_1(0),takeover_mimage_2(0)
  [takeover_mimage_0] linear iwi-aom---    2.75g          /dev/mapper/mpatha1(0)                                        
  [takeover_mimage_1] linear iwi-aom---    2.75g          /dev/mapper/mpathb1(0)                                        
  [takeover_mimage_2] linear iwi-aom---    2.75g          /dev/mapper/mpathc1(0)                                        
  [takeover_mlog]     linear lwi-aom---    4.00m          /dev/mapper/mpathh1(0)                                        


Verifying files (checkit) on mirror(s) on...
        ---- harding-03 ----

Unmounting xfs and removing mnt point on harding-03...

Deactivating and removing raid(s)
lvchange -an /dev/centipede2/takeover
lvremove -f /dev/centipede2/takeover

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2253