Bug 2076262

(In reply to Gobinda Das from comment #0)
> Always VG creation fails with below error:
> [root@tendrl25 ~]# vgcreate gluster_vg_sdb /dev/sdb 
>   Cannot use /dev/sdb: device is a multipath component
>   Command requires all devices to be found.
> 
> Eventhough  sdb has no multipath enabled

Looks like a bug in lvm.

David, is this a known issue?

I'm guessing might be caused by recent lvm improvements to identify multipath components.  lvm now also checks if a device's wwid is listed in /etc/multipath/wwids to see if it's a multipath component.  Could you check that for sdb, and if so, is there be a valid reason for the sdb wwid to be in wwids?  You can disable lvm's use of /etc/multipath/wwids by setting lvm.conf multipath_wwids_file="".  (I don't see an lvm version above so I'm not sure if you actually have this feature or not.)

I looked at the host.

# rpm -q lvm2
lvm2-2.03.14-3.el8.x86_64

/dev/sdb is a multipath device - mentioned in /etc/multipath/wwids.
I guess it was added when multipath detected the devices, before it
was blacklisted.

The multipath device is blacklisted:

# cat /etc/multipath/conf.d/blacklist.conf 
# BEGIN ANSIBLE MANAGED BLOCK
blacklist {
# BEGIN ANSIBLE MANAGED BLOCK sdb
wwid "3600508b1001c08ea930b4ffe4a7e0c33"
# END ANSIBLE MANAGED BLOCK sdb
}
# END ANSIBLE MANAGED BLOCK

So multipath ignores this device, and so should lvm.

The device was also added to system.devices file - probably when
the pv was created:

# cat /etc/lvm/devices/system.devices 
# LVM uses devices listed in this file.
# Created by LVM command vgcreate pid 309575 at Mon Apr 18 12:57:43 2022
VERSION=1.1.2
IDTYPE=sys_wwid IDNAME=naa.600508b1001c9bf88ce1781e2c53a362 DEVNAME=/dev/sda2 PVID=KkyROuDmAzXHu1CCiZ0OpKCjLxJtb6z9 PART=2
IDTYPE=sys_wwid IDNAME=naa.600508b1001c08ea930b4ffe4a7e0c33 DEVNAME=/dev/sdb PVID=DrUc4fKfgqbT3oI79DIAi5Pif6It7xnE

So I think we need to update lvm to consider also multiapth blacklists.

Until this is available in lvm, removing the device entry from /etc/multipath/wwids
before create the pv should fix this issue.

On the host, I did:

1. Remove the device entry from /etc/lvm/devices/system.devices
2. Remove the device entry from /etc/multiapth/wwids
3. Add the device back to devices file:

    lvmdevices --adddev /dev/sdb

This added:

    # cat /etc/lvm/devices/system.devices | grep /dev/sdb
    IDTYPE=sys_wwid IDNAME=naa.600508b1001c08ea930b4ffe4a7e0c33 DEVNAME=/dev/sdb PVID=DrUc4fKfgqbT3oI79DIAi5Pif6It7xnE

And with this the pv is visible to the host, and creating a vg works:

    # pvs /dev/sdb
      PV         VG             Fmt  Attr PSize    PFree   
      /dev/sdb   gluster_vg_sdb lvm2 a--  <372.58g <372.58g

So I think the gluster setup should do:

1. Get a device WWID (e.g. udevadm info /dev/sdX | grep ID_SERIAL=)
2. Add multiapth blacklist for the device
3. Remove the device entry from /etc/wwids

At this point creating the PV should work normally.

Ben - regarding comment 3:
- Is there a command to remove a device from /etc/multipath/wwids or
  the file should be modified directly?
- Do we need to reload multipathd after modifying /etc/multipath/wwids
or anything else?

(In reply to Nir Soffer from comment #3)
> The multipath device is blacklisted:
> 
> # cat /etc/multipath/conf.d/blacklist.conf 
> # BEGIN ANSIBLE MANAGED BLOCK
> blacklist {
> # BEGIN ANSIBLE MANAGED BLOCK sdb
> wwid "3600508b1001c08ea930b4ffe4a7e0c33"
> # END ANSIBLE MANAGED BLOCK sdb
> }
> # END ANSIBLE MANAGED BLOCK

Yes, I need to update lvm to consider the blacklist.

Note that when using the devices file, there's little additional benefit from using /etc/multipath/wwids in lvm, so you could consider disabling that feature (setting multipath_wwids_file=""), and avoiding this issue in that way.

(In reply to Nir Soffer from comment #4)
> Ben - regarding comment 3:
> - Is there a command to remove a device from /etc/multipath/wwids or
>   the file should be modified directly?

/etc/multipath/wwids should not need to be directly modified.

# multipath -w <device_or_wwid>

will remove a wwid from the wwids file.

# multipath -W

Will update the wwids file so that it only contains the current multipath devices.

> - Do we need to reload multipathd after modifying /etc/multipath/wwids
> or anything else?

Nope. Whenever multipathd needs to check the wwids file for something, it will re-read it, so no reloading is necessary.

fix https://sourceware.org/git/?p=lvm2.git;a=commit;h=494372b4eed0c8f6040e3357939eb7511ac25745

David, an estimate on when this fix will be available?

(In reply to Nir Soffer from comment #13)
> David, an estimate on when this fix will be available?

The next release which is 8.7.  Also the patch in comment 8 needs a small fix to handle wwid designator types 2/eui and 1/t10 (it only handles 3/naa).

*** Bug 2094535 has been marked as a duplicate of this bug. ***

To test:
pvcreate /dev/sdf
pvs should show the PV on sdf
multipath /dev/sdf
pvs should show the PV on the new mpath device
the wwid of sdf should be in /etc/multipath/wwids
add the wwid to the blacklist in multipath.conf
restart/reload multipath
pvs should show the PV on sdf
add the wwid to the blacklist exceptions in multipath.conf
restart/reload multipath
pvs should show the PV on the mpath device

This only supports wwid entries in the blacklist.
The lvm devices file needs to be disabled to test this.

(In reply to David Teigland from comment #18)
> add the wwid to the blacklist in multipath.conf

This is only one possible location, the blacklist can also be in any 
file in 

    /etc/multipath/conf.d/*.conf

I think the configuration files should be sorted by file name, and read
in order. I don't remember documentation about how same section in multiple
files are handled, but when I tried it you can have multiple blacklist 
sections and all of them are merged together with the blacklist in
/etc/mutipath.conf. 

In RHV, changes are never done in /etc/multiapth.conf - this file is managed
by vdsm and it is replaced on upgrades, removing the user changes.

In both case, you can add the wwids as is:

    blacklist {
        wwid "xxxyyy"
    }

or a regex matching the wwid

    blacklist {
        wwid "^xxxyyy$"
    }

There is also blacklist_exceptions that overrides items in blacklist.
So the code should make sue the blacklisted devices are does not have
exceptions.

(In reply to Nir Soffer from comment #20)
> (In reply to David Teigland from comment #18)
> > add the wwid to the blacklist in multipath.conf
> 
> This is only one possible location, the blacklist can also be in any 
> file in 
> 
>     /etc/multipath/conf.d/*.conf

yes, I'd forgotten that, lvm is reading blacklist and blacklist_exception sections in any of those files (all exceptions are processed last.)

> I think the configuration files should be sorted by file name, and read
> in order. I don't remember documentation about how same section in multiple
> files are handled, but when I tried it you can have multiple blacklist 
> sections and all of them are merged together with the blacklist in
> /etc/mutipath.conf. 

all the settings are merged, I'm not worrying about overriding rules at this point.

> In RHV, changes are never done in /etc/multiapth.conf - this file is managed
> by vdsm and it is replaced on upgrades, removing the user changes.
> 
> In both case, you can add the wwids as is:
> 
>     blacklist {
>         wwid "xxxyyy"
>     }
> 
> or a regex matching the wwid
> 
>     blacklist {
>         wwid "^xxxyyy$"
>     }

it's handling explicit wwids (with or without quotation), but not regex.  If people are in the habit of using ^$ around full wwids that would be simple enough to add immediately.  Otherwise, if people are really entering regexes with partial wwid strings, then we should create a new bug to add posix regex calls.

These two commits should be included in the 8.6.z backport:

filter-mpath: use multipath blacklist
https://sourceware.org/git/?p=lvm2.git;a=commit;h=494372b4eed0c8f6040e3357939eb7511ac25745

filter-mpath: handle other wwid types in blacklist
https://sourceware.org/git/?p=lvm2.git;a=commit;h=c302903dbab1d5fd05b344c654bed83c9ecb69f8

The test case in comment #18 doesn't appear to have the expected behavior, either with or without the fix.


# hayes-02 (pre fix)

[root@hayes-02 ~]# pvcreate /dev/sdf
  Physical volume "/dev/sdf" successfully created.
[root@hayes-02 ~]# pvs /dev/sdf
  PV         VG Fmt  Attr PSize  PFree 
  /dev/sdf      lvm2 ---  <1.82t <1.82t

[root@hayes-02 ~]# multipath /dev/sdf
create: mpatha (36d094660650d1e0022bd2a001f1ae16e) undef DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 0:2:5:0  sdf 8:80  undef ready running

[root@hayes-02 ~]#  pvs /dev/sdf
  Cannot use /dev/sdf: device is a multipath component

# pvs should show the PV on the new mpath device

[root@hayes-02 ~]# pvscan
  No matching physical volumes found


[root@hayes-02 ~]# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36d094660650d1e0022bd2a001f1ae16e/

[root@hayes-02 ~]# multipath -F
[root@hayes-02 ~]# multipath -v2
[root@hayes-02 ~]# multipath -l



[root@hayes-02 ~]# dmsetup ls
mpatha  (253:0)

[root@hayes-02 ~]# pvscan
  No matching physical volumes found

[root@hayes-02 ~]# cat /etc/lvm/devices/system.devices 
# LVM uses devices listed in this file.
# Created by LVM command pvremove pid 161051 at Tue Jul 19 08:07:24 2022
VERSION=1.1.44
IDTYPE=sys_wwid IDNAME=naa.6d094660650d1e0022bd2a001f1ae16e DEVNAME=/dev/sdf PVID=fuy8TmXGwy6XezuYxfoBgc2fTLi5jAZg

[root@hayes-02 ~]#  cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36d094660650d1e0022bd2a001f1ae16e/

# add to /etc/multipath.conf
blacklist {
        wwid "36d094660650d1e0022bd2a001f1ae16e"
}

[root@hayes-02 ~]# pvscan
  No matching physical volumes found




# hayes-01 (post fix)

[root@hayes-01 ~]# pvcreate /dev/sdf
  Physical volume "/dev/sdf" successfully created.
[root@hayes-01 ~]# pvs /dev/sdf
  PV         VG Fmt  Attr PSize  PFree 
  /dev/sdf      lvm2 ---  <1.82t <1.82t
[root@hayes-01 ~]# multipath /dev/sdf
create: mpatha (36d094660575ece002291bb4a313e4759) undef DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 0:2:5:0  sdf 8:80  undef ready running
[root@hayes-01 ~]# pvs /dev/sdf
  Cannot use /dev/sdf: device is a multipath component


# pvs should show the PV on the new mpath device

[root@hayes-01 ~]# pvscan
  No matching physical volumes found
[root@hayes-01 ~]# pvs /dev/mapper/mpatha
  Cannot use /dev/mapper/mpatha: device is not in devices file

[root@hayes-01 ~]# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36d094660575ece002291bb4a313e4759/

the devices file needs to be disabled for the test

Marking Verified:Tested with the latest rpms.

kernel-4.18.0-409.el8    BUILT: Tue Jul 12 00:16:54 CDT 2022
lvm2-2.03.14-5.el8    BUILT: Thu Jul 14 09:23:13 CDT 2022
lvm2-libs-2.03.14-5.el8    BUILT: Thu Jul 14 09:23:13 CDT 2022



[root@hayes-01 ~]# grep use_devicesfile /etc/lvm/lvm.conf 
        # Configuration option devices/use_devicesfile.
        # use_devicesfile = 0
        use_devicesfile = 0     # edited by QA Thu Jul 21 16:20:09 2022

[root@hayes-01 ~]# pvscan
  No matching physical volumes found

[root@hayes-01 ~]# pvcreate /dev/sdg
  Physical volume "/dev/sdg" successfully created.
[root@hayes-01 ~]# pvs
  PV         VG Fmt  Attr PSize  PFree 
  /dev/sdg      lvm2 ---  <1.82t <1.82t

[root@hayes-01 ~]# multipath /dev/sdg
create: mpathb (36d094660575ece002291bd67517f677a) undef DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 0:2:6:0  sdg 8:96  undef ready running

[root@hayes-01 ~]#  pvs
  PV                 VG Fmt  Attr PSize  PFree 
  /dev/mapper/mpathb    lvm2 ---  <1.82t <1.82t
[root@hayes-01 ~]# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36d094660575ece002291bd67517f677a/

/etc/multipath.conf
blacklist {
        wwid "36d094660575ece002291bd67517f677a"
}

[root@hayes-01 ~]#  multipath -F
[root@hayes-01 ~]#  multipath -v2
[root@hayes-01 ~]# multipath -l
[root@hayes-01 ~]# pvs
  PV         VG Fmt  Attr PSize  PFree 
  /dev/sdg      lvm2 ---  <1.82t <1.82t


/etc/multipath.conf
blacklist_exceptions {
        property "(SCSI_IDENT_|ID_WWN)"
        wwid "36d094660575ece002291bd67517f677a"
}
blacklist {
        wwid "36d094660575ece002291bd67517f677a"
}

[root@hayes-01 ~]# multipath -F
[root@hayes-01 ~]# multipath -v2
create: mpathb (36d094660575ece002291bd67517f677a) undef DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 0:2:6:0  sdg 8:96  undef ready running
[root@hayes-01 ~]# multipath -l
mpathb (36d094660575ece002291bd67517f677a) dm-0 DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 0:2:6:0  sdg 8:96  active undef running
[root@hayes-01 ~]# pvs
  PV                 VG Fmt  Attr PSize  PFree 
  /dev/mapper/mpathb    lvm2 ---  <1.82t <1.82t

Marking VERIFIED in the latest rpms.

kernel-4.18.0-414.el8    BUILT: Mon Aug  1 20:27:19 CDT 2022
lvm2-2.03.14-6.el8    BUILT: Fri Jul 29 05:40:53 CDT 2022
lvm2-libs-2.03.14-6.el8    BUILT: Fri Jul 29 05:40:53 CDT 2022


SCENARIO - lvm_usage_w_multipath_blacklist_and_devicesfile_disabled:  Test that lvm doesn't ignore multipath blacklist configuration with devicesfile disabled (bug 2076262)
adding entry to the devices file for /dev/sdf1
removing entry from the devices file for /dev/sdf1
Disabling devicesfile on hayes-01
Setting use_devicesfile to disable
Blanking out any potential partition header: dd if=/dev/zero of=/dev/sdf bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 4.23713 s, 124 MB/s
adding entry to the devices file for /dev/sdf
** use_devicesfile is NOT currently enabled **
creating PV on hayes-01 using device /dev/sdf
pvcreate --yes -ff   /dev/sdf
  Physical volume "/dev/sdf" successfully created.
WARNING: This scenario will end up repartitioning your device for use w/ mpath!
multipath /dev/sdf
create: mpathb (36d094660575ece002291bb4a313e4759) undef DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 0:2:5:0  sdf 8:80  undef ready running

Re-scanning lvm PVs (pvscan --cache)
  pvscan[69128] /dev/sdf excluded: device is a multipath component.
verify pvs multipath path /dev/mapper/mpathb
  PV                 VG Fmt  Attr PSize  PFree 
  /dev/mapper/mpathb    lvm2 ---  <1.82t <1.82t
verify pvs NON sd path /dev/sdf
  Cannot use /dev/sdf: device is a multipath component

grep 6d094660575ece002291bb4a313e4759 /etc/multipath/wwids
/36d094660575ece002291bb4a313e4759/
Added 6d094660575ece002291bb4a313e4759 to blacklist:
    wwid "6d094660575ece002291bb4a313e4759"
Copying out altered multipath.conf file to hayes-01
/tmp/multipath1721.conf -> hayes-01:/etc/multipath.conf
multipath -F
multipath -v2
multipath -l
verify pvs NON sd path /dev/sdf
  PV         VG Fmt  Attr PSize  PFree 
  /dev/sdf      lvm2 ---  <1.82t <1.82t

Added 6d094660575ece002291bb4a313e4759 to blacklist_exceptions:
    wwid "6d094660575ece002291bb4a313e4759"
Copying out altered multipath.conf file to hayes-01
/tmp/multipath373.conf -> hayes-01:/etc/multipath.conf
multipath -F
multipath -v2
create: mpathb (36d094660575ece002291bb4a313e4759) undef DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 0:2:5:0  sdf 8:80  undef ready running
multipath -l
mpathb (36d094660575ece002291bb4a313e4759) dm-0 DELL,PERC H330 Adp
size=1.8T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 0:2:5:0  sdf 8:80  active undef running
verify pvs multipath path /dev/mapper/mpathb
  PV                 VG Fmt  Attr PSize  PFree 
  /dev/mapper/mpathb    lvm2 ---  <1.82t <1.82t
verify pvs NON sd path /dev/sdf
  Cannot use /dev/sdf: device is a multipath component

multipath -f mpathb
multipath -W
successfully reset wwids
Using gpt label
Disk size: 1907199 MB
Num partitions: 1
Adding new primary partition: (40, 3905945559) = 1907199 MB
Committing changes
Removing wwid entry from /etc/multipath.conf:
    wwid "6d094660575ece002291bb4a313e4759" # edited by QA Tue Aug  9 14:11:59 2022
Removing wwid entry from /etc/multipath.conf:
    wwid "6d094660575ece002291bb4a313e4759" # edited by QA Tue Aug  9 14:11:52 2022
Copying out altered multipath.conf file to hayes-01
/tmp/multipath343.conf -> hayes-01:/etc/multipath.conf
Enabling devicesfile on hayes-01
Setting use_devicesfile to enable

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7792

*** Bug 2087534 has been marked as a duplicate of this bug. ***