2028835 – Duplicate UUID found for a multipath zFCP device

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2028835 - Duplicate UUID found for a multipath zFCP device

Summary: Duplicate UUID found for a multipath zFCP device

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	device-mapper-multipath
Sub Component:
Version:	9.0
Hardware:	s390x
OS:	All
Priority:	high
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Ben Marzinski
QA Contact:	Lin Li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1942219
TreeView+	depends on / blocked

Reported:	2021-12-03 14:08 UTC by Jan Stodola
Modified:	2022-05-17 16:24 UTC (History)
CC List:	8 users (show)
Fixed In Version:	device-mapper-multipath-0.8.7-4.el9
Doc Type:	Bug Fix
Doc Text:	Cause: Path devices added immediately after a multipath device is created for the first time may not be claimed by multipath, allowing other programs to use them directly Consequence: Anaconda may access multiple path devices directly, causing warning messages or failures during installation. Fix: Multipathd will now correctly claim path devices added immediately after a multipath device is created for the first time. Result: Anaconda will no longer try to access multipath path devices directly, avoiding installation warning messages and failures due to duplicate UUIDs.
Clone Of:
Environment:
Last Closed:	2022-05-17 15:56:11 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-104706	0	None	None	None	2021-12-03 14:09:49 UTC
Red Hat Product Errata	RHBA-2022:3971	0	None	None	None	2022-05-17 15:56:30 UTC

Description Jan Stodola 2021-12-03 14:08:31 UTC

Description of problem:
When a zFCP multipath LUN is added on s390x, the installer complains that duplicate UUID was found:
"Duplicate UUID 'b5ab8432-01' found for devices: 'sdd1' and 'sdc1'."

The zFCP device has NPIV mode enabled and has one LUN with 4 paths:
[anaconda root@rtt2 ~]# lsluns 
Scanning for LUNs on adapter 0.0.a002
        at port 0x500507680d751cc1:
                0x0000000000000000
        at port 0x500507680d751cc2:
                0x0000000000000000
        at port 0x500507680d7d1cc1:
                0x0000000000000000
        at port 0x500507680d7d1cc2:
                0x0000000000000000
[anaconda root@rtt2 ~]#

This issue looks like bug 1862032 reported against RHEL-8.

Version-Release number of selected component (if applicable):
RHEL-9.0.0-20211201.1
anaconda-34.25.0.20-1.el9
python3-blivet-3.4.0-7.el9

How reproducible:
Always

Steps to Reproduce:
1. Start a graphical (vnc) installation on an s390x system with a zFCP adapter with NPIV mode enabled and having multiple paths to a single LUN.
2. Proceed to the Installation destination spoke to add advanced storage devices (zFCP, iSCSI,..).
3. Click the "Add zFCP LUN..." button, enter the zFCP device number, WWPN and LUN.
4. Click the "Start Discovery" button.
5. Anaconda says that "Device discovery succeeded.". Click "OK".

Actual results:
Anaconda says that "Duplicate UUID 'b5ab8432-01' found for devices: 'sdd1' and 'sdc1'."

Expected results:
No duplicate devices are detected and the multipath device is properly assembled.

Additional info:

* Before adding the zFCP device in step 3:
[anaconda root@rtt2 ~]# lsblk 
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0  547M  1 loop 
loop1         7:1    0    3G  1 loop 
|-live-rw   253:0    0    3G  0 dm   /
`-live-base 253:1    0    3G  1 dm   
loop2         7:2    0   32G  0 loop 
`-live-rw   253:0    0    3G  0 dm   /
zram0       252:0    0  3.9G  0 disk [SWAP]
[anaconda root@rtt2 ~]#

* After clicking the "Start Discovery" button in step 4, the following messages are printed by "udevadm monitor":
KERNEL[635.071181] add      /devices/css0/0.0.0010 (css)
KERNEL[635.071205] add      /devices/css0/0.0.0010/0.0.a002 (ccw)
UDEV  [635.072422] add      /devices/css0/0.0.0010 (css)
KERNEL[635.099556] bind     /devices/css0/0.0.0010/0.0.a002 (ccw)
KERNEL[635.099571] add      /bus/ccw/drivers/zfcp (drivers)
KERNEL[635.099577] add      /module/zfcp (module)
UDEV  [635.100541] add      /devices/css0/0.0.0010/0.0.a002 (ccw)
UDEV  [635.100555] bind     /devices/css0/0.0.0010/0.0.a002 (ccw)
UDEV  [635.106541] add      /bus/ccw/drivers/zfcp (drivers)
UDEV  [635.107538] add      /module/zfcp (module)
KERNEL[636.201882] add      /devices/virtual/workqueue/scsi_tmf_0 (workqueue)
KERNEL[636.204608] add      /devices/css0/0.0.0010/0.0.a002/host0 (scsi)
UDEV  [636.204623] add      /devices/css0/0.0.0010/0.0.a002/host0 (scsi)
KERNEL[636.204631] add      /devices/css0/0.0.0010/0.0.a002/host0/scsi_host/host0 (scsi_host)
UDEV  [636.204639] add      /devices/virtual/workqueue/scsi_tmf_0 (workqueue)
KERNEL[636.204646] add      /devices/virtual/workqueue/scsi_wq_0 (workqueue)
UDEV  [636.204654] add      /devices/css0/0.0.0010/0.0.a002/host0/scsi_host/host0 (scsi_host)
KERNEL[636.204665] add      /devices/css0/0.0.0010/0.0.a002/host0/bsg/fc_host0 (bsg)
KERNEL[636.204672] add      /devices/css0/0.0.0010/0.0.a002/host0/fc_host/host0 (fc_host)
UDEV  [636.205736] add      /devices/virtual/workqueue/scsi_wq_0 (workqueue)
UDEV  [636.205747] add      /devices/css0/0.0.0010/0.0.a002/host0/fc_host/host0 (fc_host)
UDEV  [636.206118] add      /devices/css0/0.0.0010/0.0.a002/host0/bsg/fc_host0 (bsg)
KERNEL[636.213709] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/fc_remote_ports/rport-0:0-0 (fc_remote_ports)
KERNEL[636.214068] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/bsg/rport-0:0-0 (bsg)
KERNEL[636.214078] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/fc_remote_ports/rport-0:0-1 (fc_remote_ports)
KERNEL[636.214294] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/bsg/rport-0:0-1 (bsg)
KERNEL[636.214303] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/fc_remote_ports/rport-0:0-2 (fc_remote_ports)
KERNEL[636.214548] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/bsg/rport-0:0-2 (bsg)
KERNEL[636.214555] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/fc_remote_ports/rport-0:0-3 (fc_remote_ports)
KERNEL[636.214588] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/bsg/rport-0:0-3 (bsg)
UDEV  [636.216255] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/fc_remote_ports/rport-0:0-0 (fc_remote_ports)
UDEV  [636.216705] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/fc_remote_ports/rport-0:0-1 (fc_remote_ports)
KERNEL[636.217174] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0 (scsi)
KERNEL[636.217182] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/fc_transport/target0:0:0 (fc_transport)
KERNEL[636.217192] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0 (scsi)
KERNEL[636.217198] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/scsi_device/0:0:0:0 (scsi_device)
KERNEL[636.217234] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/bsg/0:0:0:0 (bsg)
UDEV  [636.217591] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/bsg/rport-0:0-0 (bsg)
KERNEL[636.218218] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1 (scsi)
KERNEL[636.218227] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/fc_transport/target0:0:1 (fc_transport)
KERNEL[636.218235] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0 (scsi)
KERNEL[636.218241] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/scsi_device/0:0:1:0 (scsi_device)
KERNEL[636.218275] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/bsg/0:0:1:0 (bsg)
UDEV  [636.235899] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/bsg/rport-0:0-3 (bsg)
UDEV  [636.235913] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/bsg/rport-0:0-1 (bsg)
UDEV  [636.235922] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/fc_remote_ports/rport-0:0-2 (fc_remote_ports)
UDEV  [636.235930] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/bsg/rport-0:0-2 (bsg)
UDEV  [636.235938] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/fc_remote_ports/rport-0:0-3 (fc_remote_ports)
KERNEL[636.236467] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2 (scsi)
KERNEL[636.236474] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/fc_transport/target0:0:2 (fc_transport)
KERNEL[636.236482] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0 (scsi)
KERNEL[636.236487] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/scsi_device/0:0:2:0 (scsi_device)
KERNEL[636.236516] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/bsg/0:0:2:0 (bsg)
KERNEL[636.237310] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3 (scsi)
KERNEL[636.237319] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/fc_transport/target0:0:3 (fc_transport)
KERNEL[636.237326] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0 (scsi)
KERNEL[636.237332] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/scsi_device/0:0:3:0 (scsi_device)
KERNEL[636.237367] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/bsg/0:0:3:0 (bsg)
KERNEL[636.245802] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0 (scsi)
KERNEL[636.245815] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0 (scsi)
KERNEL[636.248978] add      /class/scsi_generic (class)
UDEV  [636.249256] add      /class/scsi_generic (class)
KERNEL[636.250220] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/scsi_generic/sg0 (scsi_generic)
KERNEL[636.250246] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/scsi_generic/sg1 (scsi_generic)
KERNEL[636.250281] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/scsi_generic/sg2 (scsi_generic)
KERNEL[636.250307] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/scsi_generic/sg3 (scsi_generic)
KERNEL[636.250313] add      /module/sg (module)
UDEV  [636.250499] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0 (scsi)
UDEV  [636.252370] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2 (scsi)
UDEV  [636.253996] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/fc_transport/target0:0:0 (fc_transport)
UDEV  [636.255205] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1 (scsi)
UDEV  [636.255512] add      /module/sg (module)
UDEV  [636.255624] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/fc_transport/target0:0:2 (fc_transport)
KERNEL[636.258753] add      /class/scsi_disk (class)
KERNEL[636.258780] add      /bus/scsi/drivers/sd (drivers)
KERNEL[636.258786] add      /module/sd_mod (module)
KERNEL[636.258806] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0 (scsi_disk)
KERNEL[636.258847] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/scsi_disk/0:0:2:0 (scsi_disk)
KERNEL[636.258852] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/scsi_disk/0:0:1:0 (scsi_disk)
KERNEL[636.258941] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0 (scsi)
UDEV  [636.259173] add      /module/sd_mod (module)
UDEV  [636.259252] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/fc_transport/target0:0:1 (fc_transport)
UDEV  [636.259308] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0 (scsi)
UDEV  [636.260779] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/scsi_device/0:0:2:0 (scsi_device)
UDEV  [636.261002] add      /bus/scsi/drivers/sd (drivers)
UDEV  [636.261075] add      /class/scsi_disk (class)
KERNEL[636.262429] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/scsi_disk/0:0:3:0 (scsi_disk)
KERNEL[636.262544] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0 (scsi)
UDEV  [636.262773] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/bsg/0:0:2:0 (bsg)
UDEV  [636.262832] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/scsi_generic/sg2 (scsi_generic)
UDEV  [636.263819] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/scsi_disk/0:0:2:0 (scsi_disk)
UDEV  [636.264160] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3 (scsi)
UDEV  [636.264341] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0 (scsi)
UDEV  [636.264678] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0 (scsi)
UDEV  [636.265145] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/scsi_device/0:0:3:0 (scsi_device)
UDEV  [636.265509] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/scsi_disk/0:0:3:0 (scsi_disk)
UDEV  [636.265569] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/fc_transport/target0:0:3 (fc_transport)
UDEV  [636.266224] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/bsg/0:0:3:0 (bsg)
UDEV  [636.266469] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/scsi_generic/sg3 (scsi_generic)
UDEV  [636.266914] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0 (scsi)
KERNEL[636.301629] add      /devices/virtual/bdi/8:0 (bdi)
KERNEL[636.301698] add      /devices/virtual/bdi/8:16 (bdi)
UDEV  [636.302087] add      /devices/virtual/bdi/8:0 (bdi)
UDEV  [636.302378] add      /devices/virtual/bdi/8:16 (bdi)
KERNEL[636.303289] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb (block)
KERNEL[636.303306] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb1 (block)
KERNEL[636.303317] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb2 (block)
KERNEL[636.303913] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda (block)
KERNEL[636.303928] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
KERNEL[636.303940] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda2 (block)
KERNEL[636.304567] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0 (scsi)
KERNEL[636.304594] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0 (scsi)
KERNEL[636.364692] add      /devices/virtual/bdi/8:48 (bdi)
UDEV  [636.364708] add      /devices/virtual/bdi/8:48 (bdi)
KERNEL[636.364737] add      /devices/virtual/bdi/8:32 (bdi)
UDEV  [636.365100] add      /devices/virtual/bdi/8:32 (bdi)
KERNEL[636.366734] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd (block)
KERNEL[636.366750] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
KERNEL[636.366762] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
KERNEL[636.367357] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0 (scsi)
KERNEL[636.376777] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc (block)
KERNEL[636.376799] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
KERNEL[636.376815] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
KERNEL[636.376988] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0 (scsi)
UDEV  [636.377852] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0 (scsi)
UDEV  [636.378841] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/bsg/0:0:0:0 (bsg)
UDEV  [636.378960] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0 (scsi)
UDEV  [636.379712] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/bsg/0:0:1:0 (bsg)
UDEV  [636.379987] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/scsi_device/0:0:1:0 (scsi_device)
UDEV  [636.380370] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/scsi_device/0:0:0:0 (scsi_device)
KERNEL[636.380623] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb (block)
UDEV  [636.380798] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0 (scsi)
KERNEL[636.380815] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda (block)
UDEV  [636.381675] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/scsi_generic/sg1 (scsi_generic)
UDEV  [636.382364] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/scsi_disk/0:0:1:0 (scsi_disk)
UDEV  [636.382630] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0 (scsi)
UDEV  [636.383267] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/scsi_generic/sg0 (scsi_generic)
UDEV  [636.383729] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0 (scsi_disk)
UDEV  [636.468228] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb (block)
UDEV  [636.473476] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda (block)
UDEV  [636.487101] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc (block)
UDEV  [636.494933] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd (block)
UDEV  [636.533220] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb1 (block)
UDEV  [636.535414] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
UDEV  [636.545016] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
UDEV  [636.546033] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
UDEV  [636.547615] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb2 (block)
UDEV  [636.549252] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [636.549278] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0 (scsi)
UDEV  [636.550734] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0 (scsi)
UDEV  [636.551082] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda2 (block)
UDEV  [636.552728] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0 (scsi)
UDEV  [636.576570] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
UDEV  [636.578187] bind     /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0 (scsi)
UDEV  [636.588247] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb (block)
UDEV  [636.591162] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda (block)
KERNEL[637.594275] add      /devices/virtual/bdi/253:2 (bdi)
KERNEL[637.594374] add      /devices/virtual/block/dm-2 (block)
UDEV  [637.595128] add      /devices/virtual/bdi/253:2 (bdi)
UDEV  [637.595353] add      /devices/virtual/block/dm-2 (block)
KERNEL[637.605750] add      /module/dm_service_time (module)
UDEV  [637.606106] add      /module/dm_service_time (module)
KERNEL[637.727152] change   /devices/virtual/block/dm-2 (block)
KERNEL[637.727177] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb (block)
KERNEL[637.727190] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb1 (block)
KERNEL[637.727201] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb2 (block)
KERNEL[637.727332] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda (block)
KERNEL[637.727397] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
KERNEL[637.727410] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda2 (block)
KERNEL[637.749064] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb1 (block)
KERNEL[637.749157] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb2 (block)
UDEV  [637.749593] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb (block)
UDEV  [637.753688] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb2 (block)
UDEV  [637.755408] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb2 (block)
UDEV  [637.755801] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb1 (block)
UDEV  [637.757664] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-1/target0:0:1/0:0:1:0/block/sdb/sdb1 (block)
KERNEL[637.774267] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
KERNEL[637.774377] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda2 (block)
UDEV  [637.774949] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda (block)
UDEV  [637.777744] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [637.779485] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [637.780859] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda2 (block)
UDEV  [637.782868] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sda/sda2 (block)
KERNEL[637.822046] add      /devices/virtual/bdi/253:3 (bdi)
KERNEL[637.822066] add      /devices/virtual/block/dm-3 (block)
KERNEL[637.822079] change   /devices/virtual/block/dm-3 (block)
KERNEL[637.822086] add      /devices/virtual/bdi/253:4 (bdi)
KERNEL[637.822095] add      /devices/virtual/block/dm-4 (block)
KERNEL[637.822107] change   /devices/virtual/block/dm-4 (block)
UDEV  [637.822740] add      /devices/virtual/bdi/253:4 (bdi)
UDEV  [637.823240] add      /devices/virtual/block/dm-3 (block)
UDEV  [637.823601] add      /devices/virtual/bdi/253:3 (bdi)
UDEV  [637.823820] add      /devices/virtual/block/dm-4 (block)
UDEV  [637.824983] change   /devices/virtual/block/dm-2 (block)
UDEV  [637.874889] change   /devices/virtual/block/dm-4 (block)
UDEV  [637.877975] change   /devices/virtual/block/dm-3 (block)
KERNEL[639.011617] change   /devices/virtual/block/dm-2 (block)
UDEV  [639.047775] change   /devices/virtual/block/dm-2 (block)

[anaconda root@rtt2 ~]# lsblk 
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0  547M  1 loop  
loop1         7:1    0    3G  1 loop  
|-live-rw   253:0    0    3G  0 dm    /
`-live-base 253:1    0    3G  1 dm    
loop2         7:2    0   32G  0 loop  
`-live-rw   253:0    0    3G  0 dm    /
sda           8:0    0   50G  0 disk  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
sdb           8:16   0   50G  0 disk  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
sdc           8:32   0   50G  0 disk  
|-sdc1        8:33   0    1G  0 part  
|-sdc2        8:34   0   15G  0 part  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
sdd           8:48   0   50G  0 disk  
|-sdd1        8:49   0    1G  0 part  
|-sdd2        8:50   0   15G  0 part  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
zram0       252:0    0  3.9G  0 disk  [SWAP]
[anaconda root@rtt2 ~]#

Note that partitions sdc1, sdc2, sdd1 and sdd2 are still printed by lsblk, which doesn't look correct for a multipath device.

* After clicking "OK" after successful device discovery in step 5, the following udev events are printed by "udevadm monitor":
KERNEL[742.167012] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
KERNEL[742.167509] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
UDEV  [742.175130] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
UDEV  [742.178164] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
KERNEL[742.180241] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc (block)
KERNEL[742.180258] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
KERNEL[742.180273] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
KERNEL[742.205146] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
KERNEL[742.205284] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
UDEV  [742.205904] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc (block)
UDEV  [742.209531] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
UDEV  [742.211751] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
UDEV  [742.213756] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc1 (block)
UDEV  [742.214699] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-2/target0:0:2/0:0:2:0/block/sdc/sdc2 (block)
KERNEL[742.425519] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
KERNEL[742.426568] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
UDEV  [742.431121] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
UDEV  [742.432411] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
KERNEL[742.435000] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
KERNEL[742.435050] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
KERNEL[742.435145] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd (block)
KERNEL[742.435204] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
KERNEL[742.435242] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
UDEV  [742.438427] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
UDEV  [742.439103] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
KERNEL[742.459670] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
KERNEL[742.459778] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
UDEV  [742.460392] change   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd (block)
UDEV  [742.506814] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
UDEV  [742.515900] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd2 (block)
UDEV  [742.517461] add      /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)
UDEV  [742.518303] remove   /devices/css0/0.0.0010/0.0.a002/host0/rport-0:0-3/target0:0:3/0:0:3:0/block/sdd/sdd1 (block)

[anaconda root@rtt2 ~]# lsblk 
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0  547M  1 loop  
loop1         7:1    0    3G  1 loop  
|-live-rw   253:0    0    3G  0 dm    /
`-live-base 253:1    0    3G  1 dm    
loop2         7:2    0   32G  0 loop  
`-live-rw   253:0    0    3G  0 dm    /
sda           8:0    0   50G  0 disk  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
sdb           8:16   0   50G  0 disk  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
sdc           8:32   0   50G  0 disk  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
sdd           8:48   0   50G  0 disk  
`-mpatha    253:2    0   50G  0 mpath 
  |-mpatha1 253:3    0    1G  0 part  
  `-mpatha2 253:4    0   15G  0 part  
zram0       252:0    0  3.9G  0 disk  [SWAP]
[anaconda root@rtt2 ~]#

[anaconda root@rtt2 ~]# multipath -ll
mpatha (3600507638080a09b0800000000000051) dm-2 IBM,2145
size=50G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 0:0:1:0 sdb 8:16 active ready running
| `- 0:0:3:0 sdd 8:48 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 0:0:0:0 sda 8:0  active ready running
  `- 0:0:2:0 sdc 8:32 active ready running
[anaconda root@rtt2 ~]#

[anaconda root@rtt2 ~]# blkid
/dev/loop1: LABEL="Anaconda" UUID="ba97ba3b-44d7-4ac6-bfa8-c4205f589801" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/live-base: LABEL="Anaconda" UUID="ba97ba3b-44d7-4ac6-bfa8-c4205f589801" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdd: PTUUID="b5ab8432" PTTYPE="dos"
/dev/sdb: PTUUID="b5ab8432" PTTYPE="dos"
/dev/mapper/mpatha2: UUID="a3d3c609-a22c-4743-a27e-8b1fc1993751" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="b5ab8432-02"
/dev/mapper/mpatha: PTUUID="b5ab8432" PTTYPE="dos"
/dev/loop0: TYPE="squashfs"
/dev/mapper/live-rw: LABEL="Anaconda" UUID="ba97ba3b-44d7-4ac6-bfa8-c4205f589801" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdc: PTUUID="b5ab8432" PTTYPE="dos"
/dev/sda: PTUUID="b5ab8432" PTTYPE="dos"
/dev/zram0: LABEL="zram0" UUID="9b0185c0-525a-4d5b-9c4a-ff78d9f7eb6d" TYPE="swap"
/dev/mapper/mpatha1: UUID="b28b3774-1cee-4e4b-bc42-a02bab749a40" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="b5ab8432-01"
[anaconda root@rtt2 ~]#

Comment 7 Jan Stodola 2021-12-04 17:06:07 UTC

I can reliably reproduce this problem also on a freshly installed system with the following steps:

[root@rtt2 ~]# yum install device-mapper-multipath
...
[root@rtt2 ~]# rpm -q kernel device-mapper-multipath systemd-udev
kernel-5.14.0-24.el9.s390x
device-mapper-multipath-0.8.7-3.el9.s390x
systemd-udev-249-9.el9.s390x
[root@rtt2 ~]# mpathconf --enable
[root@rtt2 ~]# systemctl enable --now multipathd
[root@rtt2 ~]# lsblk 
NAME     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
dasda     94:0    0  6.9G  0 disk 
└─dasda1  94:1    0  6.9G  0 part /
dasdb     94:4    0  6.9G  0 disk 
├─dasdb1  94:5    0    1G  0 part /boot
└─dasdb2  94:6    0  1.4G  0 part [SWAP]
[root@rtt2 ~]# cio_ignore -r a002
[root@rtt2 ~]# echo 1 > /sys/bus/ccw/drivers/zfcp/0.0.a002/online
[root@rtt2 ~]# lsblk 
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   50G  0 disk  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
sdb           8:16   0   50G  0 disk  
├─sdb1        8:17   0    1G  0 part  
├─sdb2        8:18   0   15G  0 part  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
sdc           8:32   0   50G  0 disk  
├─sdc1        8:33   0    1G  0 part  
├─sdc2        8:34   0   15G  0 part  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
sdd           8:48   0   50G  0 disk  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
dasda        94:0    0  6.9G  0 disk  
└─dasda1     94:1    0  6.9G  0 part  /
dasdb        94:4    0  6.9G  0 disk  
├─dasdb1     94:5    0    1G  0 part  /boot
└─dasdb2     94:6    0  1.4G  0 part  [SWAP]
[root@rtt2 ~]#

On the other hand, I was not able to reproduce the problem after reboot of the system, the mpatha and sd* devices look correct:
[root@rtt2 ~]# lsblk 
NAME     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
dasda     94:0    0  6.9G  0 disk 
└─dasda1  94:1    0  6.9G  0 part /
dasdb     94:4    0  6.9G  0 disk 
├─dasdb1  94:5    0    1G  0 part /boot
└─dasdb2  94:6    0  1.4G  0 part [SWAP]
[root@rtt2 ~]# cio_ignore -r a002
[root@rtt2 ~]# echo 1 > /sys/bus/ccw/drivers/zfcp/0.0.a002/online
[root@rtt2 ~]# lsblk 
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   50G  0 disk  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
sdb           8:16   0   50G  0 disk  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
sdc           8:32   0   50G  0 disk  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
sdd           8:48   0   50G  0 disk  
└─mpatha    253:0    0   50G  0 mpath 
  ├─mpatha1 253:1    0    1G  0 part  
  └─mpatha2 253:2    0   15G  0 part  
dasda        94:0    0  6.9G  0 disk  
└─dasda1     94:1    0  6.9G  0 part  /
dasdb        94:4    0  6.9G  0 disk  
├─dasdb1     94:5    0    1G  0 part  /boot
└─dasdb2     94:6    0  1.4G  0 part  [SWAP]
[root@rtt2 ~]#

Reassigning to device-mapper-multipath for further review.

Comment 8 Ben Marzinski 2021-12-06 17:20:05 UTC

If multipath is configured with "find_multipaths yes", the first time it sees a scsi device, there is no way for it to know that the scsi device should be part of a multpath device.  Once the second device has appeared, and multipathd has created a multipath device out of the two path devices, it will remember the WWID, and going forward, it will claim the scsi device as a multipath path device as soon as it sees it. So sdc1 and sdd1 will always get created on installation without extra work (like adding the WWID of the device to the kernel command line with the argument: mpath.wwid=<WWID>). However multipathd should have triggered a change udev event on the scsi devices when it created the multipath device, and the
68-del-part-nodes.rules udev rule should have removed those partitions, assuming that they weren't in use.

Jan, is there a s390x machine I can get on to try and see why those partitions aren't getting removed?

Comment 10 Ben Marzinski 2021-12-06 19:47:14 UTC

Ah. There is a race.  With find_multipaths enabled, multipathd needs to see two paths to know to create a multipath device the first time (before it records the WWID). The two paths
that multipathd knows about when it creates the multipath device do get their partitions removed. Notice that in the Comment #7 sda and sdd do have their partition devices removed.  However the other two paths have already make it through udev processing before the multipath device is created. Because the multipath device already exists when they (sdb and sdc, in Comment #7) are added, multipathd assumes that they were properly claimed when they were processed by udev, so it doesn't trigger a change event on them. But since they already made it through udev processing before the multipath device was created, they were not claimed. Because of this, these paths still retain their partitions, and have

DM_MULTIPATH_DEVICE_PATH=0

set, and don't have

SYSTEMD_READY=0

set in the udev database, meaning that as far as udev is concerned, they are not claimed by multipath. The installer error message only lists two partitions that have duplicate IDs. Assuming that this is like the installed machine, where there are four paths, then it seems likely that if multipathd correctly deletes these partitions, this error won't occur. I can fix that.

Comment 11 Ben Marzinski 2022-01-18 18:21:49 UTC

The easiest way to check my fix for this is simply to

1. delete /etc/multipath/wwids
2. start multipathd
3. Create 4 scsi_debug path devices
# modprobe scsi_debug vpd_use_hostno=0 add_host=4 dev_size_mb=100
4. For each multipath path device, run
# udevadm info -q property -n <devname> | grep DM_MULTIPATH_DEVICE_PATH

If, for instance, you had this multipath device:

mpathd (333333330000007d0) dm-6 Linux,scsi_debug
size=100M features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| `- 9:0:0:0  sde 8:64 active undef running
|-+- policy='service-time 0' prio=0 status=enabled
| `- 10:0:0:0 sdf 8:80 active undef running
|-+- policy='service-time 0' prio=0 status=enabled
| `- 8:0:0:0  sdd 8:48 active undef running
`-+- policy='service-time 0' prio=0 status=enabled
  `- 11:0:0:0 sdg 8:96 active undef running

You could run:
# udevadm info -q property -n sde | grep DM_MULTIPATH_DEVICE_PATH
# udevadm info -q property -n sdf | grep DM_MULTIPATH_DEVICE_PATH
# udevadm info -q property -n sdd | grep DM_MULTIPATH_DEVICE_PATH
# udevadm info -q property -n sdg | grep DM_MULTIPATH_DEVICE_PATH

Without this fix, you are are likely to see two paths that print:
DM_MULTIPATH_DEVICE_PATH=1

and two that return:
DM_MULTIPATH_DEVICE_PATH=0

Although this is a race, and there is a chance is won't always occur.

With this fix, all paths should always return:
DM_MULTIPATH_DEVICE_PATH=1

You can also try the reproducer steps from comment #7 to verify this fixes the original problem.

Comment 14 Lin Li 2022-01-27 04:08:55 UTC

Reproduced on device-mapper-multipath-0.8.7-3.el9
1,[root@storageqe-05 ~]# rpm -qa | grep multipath
device-mapper-multipath-libs-0.8.7-3.el9.x86_64
device-mapper-multipath-0.8.7-3.el9.x86_64

2,[root@storageqe-05 ~]# rm -rf /etc/multipath/wwids
[root@storageqe-05 ~]# echo $?
0

3,[root@storageqe-05 ~]# service multipathd restart
Redirecting to /bin/systemctl restart multipathd.service

4,[root@storageqe-05 ~]# modprobe scsi_debug vpd_use_hostno=0 add_host=4 dev_size_mb=100

5,[root@storageqe-05 ~]# multipath -ll mpatha
mpatha (333333330000007d0) dm-8 Linux,scsi_debug
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:0 sdw 65:96  active ready running
|-+- policy='service-time 0' prio=10 status=enabled
| `- 7:0:0:0 sdx 65:112 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 8:0:0:0 sdy 65:128 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 5:0:0:0 sdv 65:80  active ready running
 
6,[root@storageqe-05 ~]# udevadm info -q property -n sdw | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=0
[root@storageqe-05 ~]# udevadm info -q property -n sdx | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=0
[root@storageqe-05 ~]# udevadm info -q property -n sdy | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=1
[root@storageqe-05 ~]# udevadm info -q property -n sdv | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=1


Verified on device-mapper-multipath-0.8.7-4.el9
1,[root@storageqe-05 ~]# rpm -qa | grep multipath
device-mapper-multipath-libs-0.8.7-4.el9.x86_64
device-mapper-multipath-0.8.7-4.el9.x86_64

2,[root@storageqe-05 ~]# rm -rf /etc/multipath/wwids
[root@storageqe-05 ~]# echo $?
0

3,[root@storageqe-05 ~]# service multipathd restart
Redirecting to /bin/systemctl restart multipathd.service

4,[root@storageqe-05 ~]# modprobe scsi_debug vpd_use_hostno=0 add_host=4 dev_size_mb=100

5,[root@storageqe-05 ~]# multipath -ll mpatha
mpatha (333333330000007d0) dm-8 Linux,scsi_debug
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:0 sdw 65:96  active ready running
|-+- policy='service-time 0' prio=10 status=enabled
| `- 7:0:0:0 sdx 65:112 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 8:0:0:0 sdy 65:128 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 5:0:0:0 sdv 65:80  active ready running

6,[root@storageqe-05 ~]# udevadm info -q property -n sdw | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=1
[root@storageqe-05 ~]# udevadm info -q property -n sdx | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=1
[root@storageqe-05 ~]# udevadm info -q property -n sdy | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=1
[root@storageqe-05 ~]# udevadm info -q property -n sdv | grep DM_MULTIPATH_DEVICE_PATH
DM_MULTIPATH_DEVICE_PATH=1



Test result:With this fix, all paths always return:
DM_MULTIPATH_DEVICE_PATH=1

Comment 16 errata-xmlrpc 2022-05-17 15:56:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: device-mapper-multipath), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:3971

Note You need to log in before you can comment on or make changes to this bug.