Bug 2109351

Summary:	multipath_component_detection = 0 in lvm.conf does not have any effect
Product:	Red Hat Enterprise Linux 9	Reporter:	Gopinath Marappan <marappan_gopinath>
Component:	lvm2	Assignee:	David Teigland <teigland>
lvm2 sub component:	Configuration files	QA Contact:	cluster-qe <cluster-qe>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	unspecified	CC:	agk, Ajaykumar.Rajappa, cmarthal, heinzm, jbrassow, madhu.tarikere, marappan_gopinath, mcsontos, msnitzer, prajnoha, teigland, wayne.berthiaume, yi.he, zkabelac
Version:	9.0	Keywords:	Triaged
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	lvm2-2.03.16-3.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2111136 2111137 (view as bug list)		Environment:
Last Closed:	2022-11-15 11:20:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2111136, 2111137

Description Gopinath Marappan 2022-07-21 03:25:44 UTC

Description of problem:

Disabling multipath component detection in lvm.conf does not have any effect. It always ignores PVs on native devices irrespective of whether multipath component detection is enabled or not.

We have a specific requirement for lvm to NOT ignore PVs on multiple native devices when dm-multipath is NOT enabled.

This is needed to support Dell EMC PowerPath multipathing software with RHEL9 on BFS (Boot from SAN) hosts. When a BFS host is booted up, the root LV needs to be seen from one of the native devices to the boot lun. Then after the kernel loads, PowerPath will start and do multipathing.

Because of the multipath_component_detection issue, lvm does not see the PVs on the native devices on bot (as it ignores them) and hence the host does not boot up when there are multiple paths to the boot lun.

Version-Release number of selected component (if applicable):

lvm2-libs-2.03.14-4.el9.x86_64
lvm2-2.03.14-4.el9.x86_64

How reproducible:

Every time.

Steps to Reproduce:

1. Disable both the use of devices file and multipath component detection in /etc/lvm/lvm.conf by setting the following.

use_devicesfile = 0
multipath_component_detection = 0

2. Create a PV on one path of a native device to an array LUN.

[root@ncpp227046 ~]# pvcreate /dev/sdci
Physical volume "/dev/sdci" successfully created.

3. Run pvscan to see the just created PV.

[root@ncpp227046 ~]# pvscan
PV /dev/sdh3 VG rhel_ncpp227046 lvm2 [53.41 GiB / 0 free]
Total: 1 [53.41 GiB] / in use: 1 [53.41 GiB] / in no VG: 0 [0 ]

The PV shown above is a different PV on the local hard disk. It does not show the PV on the just created device.

Actual results:

pvscan does not show the PV on a native device. Here, /dev/sdci, /dev/sddc and /dev/sdbk are all paths to the same array LUN.

As seen below from the snippet of 'pvscan -vvv' output, lvm ignores all the native devices as it sees duplicate PVIDs on them. It does this because it believes there will be a dm-multipath device also available - but in our case there will not be any dm-multipath device as dm-multipath is not enabled (and cannot be enabled also).

Resolving duplicate devices
Checking for multipath components for duplicate PVID fNvc7okgiFdvF1ociAUMMGwhic5rfHxL
Same wwids for duplicate PVs /dev/sdci /dev/sddc
Same wwids for duplicate PVs /dev/sdci /dev/sdbk
Ignoring multipath component /dev/sdci with PVID fNvc7okgiFdvF1ociAUMMGwhic5rfHxL (dropping info)
Ignoring multipath component /dev/sddc with PVID fNvc7okgiFdvF1ociAUMMGwhic5rfHxL (dropping duplicate)
Ignoring multipath component /dev/sdbk with PVID fNvc7okgiFdvF1ociAUMMGwhic5rfHxL (dropping duplicate)

Expected results:

pvscan should have shown /dev/sdci and duplicate PVs on /dev/sddc and /dev/sdbk. Essentially we require a similar behavior as in older LVM versions (such as in RHEL 8).

Additional info:

We cannot use devices file also in our scenario as there is a need to support SRDF (remote replication) scenarios where the wwids of the devices can be different depending on whether the paths are coming from the local or remote array.

Comment 1 David Teigland 2022-07-25 16:24:42 UTC

I will look into applying the multipath_component_detection setting to the newly expanded detection features also.

Comment 2 Gopinath Marappan 2022-07-25 16:30:42 UTC

Thanks David!

Btw, I have a patch that fixes this issue.  But I couldn't raise a pull request at https://github.com/lvmteam/lvm2 as I dont have access.

If you can add me as a contributor, I can do a pull request.

I have also filed an issue for this in the github repo - https://github.com/lvmteam/lvm2/issues/86 .

Please advise.  Thanks.

Comment 3 David Teigland 2022-07-25 19:26:04 UTC

fix in main branch:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=99ce09ae778c2cc4aa2611e425bba5287b8b9513

# rm /run/lvm/hints 
# pvs
  PV         VG           Fmt  Attr PSize    PFree
  /dev/sda2  rhel_null-04 lvm2 a--  <464.76g    0 
# rm /run/lvm/hints 
# pvs --config devices/multipath_component_detection=0
  WARNING: Not using device /dev/sdj for PV CqOFo8-ozdc-4yiu-SFjd-jcH2-qFqs-V2KneR.
  WARNING: PV CqOFo8-ozdc-4yiu-SFjd-jcH2-qFqs-V2KneR prefers device /dev/sdi because device was seen first.
  PV         VG           Fmt  Attr PSize    PFree  
  /dev/sda2  rhel_null-04 lvm2 a--  <464.76g      0 
  /dev/sdi                lvm2 ---   100.00m 100.00m

Comment 4 David Teigland 2022-07-25 19:29:31 UTC

(In reply to Gopinath Marappan from comment #2)
> Thanks David!
> 
> Btw, I have a patch that fixes this issue.  But I couldn't raise a pull
> request at https://github.com/lvmteam/lvm2 as I dont have access.
> 
> If you can add me as a contributor, I can do a pull request.

Sorry, I'm afraid we largely ignore the github repo and primarily use sourceware.

> I have also filed an issue for this in the github repo -
> https://github.com/lvmteam/lvm2/issues/86 .

OK, we also need a bz for the issue in order to get it released in RHEL.

Comment 5 Gopinath Marappan 2022-07-26 03:25:48 UTC

Thanks for the quick fix David.

Will it be possible to backport the fix to RHEL9.0 and RHEL9.1?

Comment 7 Corey Marthaler 2022-08-04 22:15:26 UTC

Marking Verified in the latest rpms.

kernel-5.14.0-138.el9    BUILT: Sun Jul 31 06:20:38 AM CDT 2022
lvm2-2.03.16-3.el9    BUILT: Mon Aug  1 04:42:35 AM CDT 2022
lvm2-libs-2.03.16-3.el9    BUILT: Mon Aug  1 04:42:35 AM CDT 2022




[root@hayes-03 ~]# /usr/tests/sts-rhel8.7/lvm2/bin/qe_lvmconf2 -Pe
Setting multipath_component_detection to enable

[root@hayes-03 ~]#  grep multipath_component_detection /etc/lvm/lvm.conf 
        # Configuration option devices/multipath_component_detection.
        # multipath_component_detection = 1
        multipath_component_detection = 1     # edited by QA Thu Aug  4 16:18:10 2022
[root@hayes-03 ~]# grep multipath_wwids_file /etc/lvm/lvm.conf 
        # Configuration option devices/multipath_wwids_file.
        # multipath_wwids_file = "/etc/multipath/wwids"

[root@hayes-03 ~]# multipath /dev/sdb
97959.552904 | mpathb: addmap [0 936640512 multipath 0 0 1 1 service-time 0 1 1 8:16 1]
create: mpathb (36d09466083d8e100233c17f9212bf63e) undef DELL,PERC H330 Adp
size=447G features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 0:2:3:0  sdb 8:16  undef ready running
[root@hayes-03 ~]# pvcreate /dev/mapper/mpathb1
  Physical volume "/dev/mapper/mpathb1" successfully created.
[root@hayes-03 ~]# vgcreate test /dev/mapper/mpathb1
  Volume group "test" successfully created
[root@hayes-03 ~]# lvcreate -L 10M test
  Rounding up size to full physical extent 12.00 MiB
  Logical volume "lvol0" created.

[root@hayes-03 ~]# pvscan
  PV /dev/mapper/mpathb1   VG test            lvm2 [446.62 GiB / <446.61 GiB free]
  Total: 1 [446.62 GiB] / in use: 1 [446.62 GiB] / in no VG: 0 [0   ]



[root@hayes-03 ~]# /usr/tests/sts-rhel8.7/lvm2/bin/qe_lvmconf2 -Pd
Setting multipath_component_detection to disable

[root@hayes-03 ~]# grep multipath_component_detection /etc/lvm/lvm.conf
        # Configuration option devices/multipath_component_detection.
        # multipath_component_detection = 1
        multipath_component_detection = 0     # edited by QA Thu Aug  4 16:22:25 2022
[root@hayes-03 ~]# grep multipath_wwids_file /etc/lvm/lvm.conf 
        # Configuration option devices/multipath_wwids_file.
        # multipath_wwids_file = "/etc/multipath/wwids"
        multipath_wwids_file = ""
[root@hayes-03 ~]# pvscan
  PV /dev/mapper/mpathb1   VG test            lvm2 [446.62 GiB / <446.61 GiB free]
  Total: 1 [446.62 GiB] / in use: 1 [446.62 GiB] / in no VG: 0 [0   ]
[root@hayes-03 ~]# pvs
  PV                  VG   Fmt  Attr PSize   PFree   
  /dev/mapper/mpathb1 test lvm2 a--  446.62g <446.61g
[root@hayes-03 ~]# echo $?
0
[root@hayes-03 ~]# vgchange -ay test
  1 logical volume(s) in volume group "test" now active
[root@hayes-03 ~]# echo $?
0
[root@hayes-03 ~]# lvs test
  LV    VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lvol0 test -wi-a----- 12.00m                                                    
[root@hayes-03 ~]# echo $?
0

Comment 11 errata-xmlrpc 2022-11-15 11:20:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8358