RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2031810 - Symlinks to NVMe drives are missing in /dev/disk/by-path
Summary: Symlinks to NVMe drives are missing in /dev/disk/by-path
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: systemd
Version: 9.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: David Tardon
QA Contact: Frantisek Sumsal
URL:
Whiteboard:
: 2124964 2133681 (view as bug list)
Depends On: 2138081
Blocks: 2099868 2124964 2217131
TreeView+ depends on / blocked
 
Reported: 2021-12-13 13:27 UTC by Lukas Musil
Modified: 2023-06-24 06:39 UTC (History)
15 users (show)

Fixed In Version: systemd-252-3.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2217131 (view as bug list)
Environment:
Last Closed: 2023-05-09 08:21:58 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Output from udevadm info (3.27 KB, text/plain)
2021-12-13 13:27 UTC, Lukas Musil 		no flags Details
patched systemd-udev (1.47 MB, application/octet-stream)
2022-10-24 14:21 UTC, IBM Bug Proxy 		no flags Details
patched systemd (4.30 MB, application/octet-stream)
2022-10-24 14:21 UTC, IBM Bug Proxy 		no flags Details
patch for systemd/udev (1.92 KB, application/octet-stream)
2022-10-24 14:21 UTC, IBM Bug Proxy 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github redhat-plumbers systemd-rhel9 pull 114 0 None closed (#2031810) Support by-path devlink for multipath NVMe block devices 2022-12-08 14:15:16 UTC
Github systemd systemd pull 24766 0 None Merged udev: support by-path devlink for multipath nvme block devices 2022-10-10 07:30:54 UTC
Github systemd systemd pull 24946 0 None Merged udev: drop assertion which is always false 2022-10-11 06:47:47 UTC
Github systemd systemd pull 24954 0 None Merged udev: add one more assertion 2022-10-11 06:47:21 UTC
Github systemd systemd pull 24959 0 None Merged test: add coverage for the nvme-subsystem 2022-10-11 11:42:28 UTC
Red Hat Issue Tracker RHELPLAN-105532 0 None None None 2021-12-13 13:29:46 UTC
Red Hat Product Errata RHBA-2023:2531 0 None None None 2023-05-09 08:22:25 UTC

Description Lukas Musil 2021-12-13 13:27:43 UTC 
Created attachment 1846089 [details]
Output from udevadm info

Created attachment 1846089 [details]
Output from udevadm info

Description of problem:
Symlinks to NVMe drives are missing in /dev/disk/by-path, but this can be vendor-specific. For example, we found this problem on AMD Epyc machines based on Supermicro mainboards with Samsung NVMe drives. On RHEL8, all symlinks were created; on RHEL9, only created by-path symlinks for Micron NVMe SSD (other vendors untested yet). 

Version-Release number of selected component (if applicable):
RHEL-9.0.0-20211210.d.3, but other versions are affected too.

RHEL9 results:
[root@hulk ~]# ls -l /dev/disk/by-path/ | grep -i nvme
[root@hulk ~]# udevadm info --query=property --name=/dev/nvme0n1 |grep -i ID_MODEL
ID_MODEL=SAMSUNG MZWLL800HEHP-00003

[root@frost ~]# ls -l /dev/disk/by-path/ | grep -i nvme
[root@frost ~]# udevadm info --query=property --name=/dev/nvme0n1 |grep -i ID_MODEL
ID_MODEL=SAMSUNG MZWLJ1T9HBJR-00007

And there is the result from the machine with Micron SSD NVMe
[root@korg ~]# ls -l /dev/disk/by-path/ | grep -i nvme 
lrwxrwxrwx. 1 root root 13 Dec 10  2021 pci-0000:03:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx. 1 root root 13 Dec 10  2021 pci-0000:04:00.0-nvme-1 -> ../../nvme1n1
[root@korg ~]# udevadm info --query=property --name=/dev/nvme0n1 |grep ID_MODEL
ID_MODEL=Micron_7300_MTFDHBA480TDF
[root@korg ~]# 

RHEL8 results:
[root@hulk ~]# ls -l /dev/disk/by-path | grep -i nvme
lrwxrwxrwx. 1 root root 13 Dec 10  2021 pci-0000:21:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx. 1 root root 13 Dec 10  2021 pci-0000:22:00.0-nvme-1 -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Dec 10  2021 pci-0000:61:00.0-nvme-1 -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 Dec 10  2021 pci-0000:62:00.0-nvme-1 -> ../../nvme3n1
[root@hulk ~]# udevadm info --query=property --name=/dev/nvme0n1 |grep ID_MODEL
ID_MODEL=SAMSUNG MZWLL800HEHP-00003

[root@frost ~]# ls -l /dev/disk/by-path | grep -i nvme
lrwxrwxrwx. 1 root root 13 Dec 10 23:03 pci-0000:c1:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx. 1 root root 13 Dec 10 23:03 pci-0000:c2:00.0-nvme-1 -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Dec 10 23:03 pci-0000:c3:00.0-nvme-1 -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 Dec 10 23:03 pci-0000:c4:00.0-nvme-1 -> ../../nvme3n1
[root@frost ~]# udevadm info --query=property --name=/dev/nvme0n1 |grep ID_MODEL
ID_MODEL=SAMSUNG MZWLJ1T9HBJR-00007

Additional info:
Hulk is a machine with AMD EPYC 7251 on Supermicro H11SSL-NC with Samsung PM1725a
Frost is a machine with AMD EPYC 7232P on Supermicro H12SSW-NTR
 with Samsung PM1733

I found there is a "virtual" in DEVPATH for Samsung NVMe SSD on RHEL9; is this correct?
[root@hulk ~]# udevadm info --query=property --name=/dev/nvme0n1
DEVPATH=/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1

complete udevadm info output in attachment
Comment 18 Volodymyr Rudniev 2022-07-26 12:37:27 UTC 
Hello. 

We have the same problem with Dell Ent NVMe AGN MU U.2 1.6TB (Samsung PM1733). 
Our setup has:
- 4x NVMe Dell Express Flash NVMe P4610 3.2TB SFF (Intel P4610 3.2TB)
- 7x NVMe Dell Ent NVMe AGN MU U.2 1.6TB (Samsung PM1733)
- SATA 480 SSD

# nvme list
Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          S61ENA0R100937       Dell Ent NVMe AGN MU U.2 1.6TB           1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
/dev/nvme1n1          S61ENA0R100933       Dell Ent NVMe AGN MU U.2 1.6TB           1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
/dev/nvme10n1         PHLN040100CD3P2BGN   Dell Express Flash NVMe P4610 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25
/dev/nvme2n1          S61ENA0R100766       Dell Ent NVMe AGN MU U.2 1.6TB           1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
/dev/nvme3n1          S61ENA0R100935       Dell Ent NVMe AGN MU U.2 1.6TB           1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
/dev/nvme4n1          S61ENA0R100938       Dell Ent NVMe AGN MU U.2 1.6TB           1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
/dev/nvme5n1          S61ENE0NB02363       Dell Ent NVMe AGN MU U.2 1.6TB           1           1.60  TB /   1.60  TB    512   B +  0 B   2.0.2
/dev/nvme6n1          S61ENA0R100931       Dell Ent NVMe AGN MU U.2 1.6TB           1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
/dev/nvme7n1          PHLN040100X13P2BGN   Dell Express Flash NVMe P4610 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25
/dev/nvme8n1          PHLN0401019B3P2BGN   Dell Express Flash NVMe P4610 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25
/dev/nvme9n1          PHLN0401011L3P2BGN   Dell Express Flash NVMe P4610 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25


/dev/disk/by-path/ doesn't contain symlinks for "7x NVMe Dell Ent NVMe AGN MU U.2 1.6TB":

# ls -al /dev/disk/by-path/
total 0
drwxr-xr-x 2 root root 200 Jul 26 13:01 .
drwxr-xr-x 7 root root 140 Jul 26 13:01 ..
lrwxrwxrwx 1 root root   9 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0 -> ../../sda
lrwxrwxrwx 1 root root  10 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root  10 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root  10 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root  13 Jul 26 13:01 pci-0000:da:00.0-nvme-1 -> ../../nvme7n1
lrwxrwxrwx 1 root root  13 Jul 26 13:01 pci-0000:db:00.0-nvme-1 -> ../../nvme8n1
lrwxrwxrwx 1 root root  13 Jul 26 13:01 pci-0000:dc:00.0-nvme-1 -> ../../nvme9n1
lrwxrwxrwx 1 root root  14 Jul 26 13:01 pci-0000:dd:00.0-nvme-1 -> ../../nvme10n1


As I can see, the problem repeats in all distros which use kernel v4.19 and newer.

I noticed that a kernel creates DEVPATH `/sys/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0/nvme0c0n1` for the problem drives, which should creates in multipath mode, according to kernel sources (https://github.com/torvalds/linux/blob/v4.19/drivers/nvme/host/multipath.c#L41). So I disabled NVMe multipath with "nvme-core.multipath=N" kernel cmdline parameter and it looks like a workaround:

# ls -al /dev/disk/by-path/
total 0
drwxr-xr-x 2 root root 340 Jul 26 10:59 .
drwxr-xr-x 7 root root 140 Jul 26 10:59 ..
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:3d:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:3f:00.0-nvme-1 -> ../../nvme1n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:40:00.0-nvme-1 -> ../../nvme2n1
lrwxrwxrwx 1 root root   9 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0 -> ../../sda
lrwxrwxrwx 1 root root  10 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root  10 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root  10 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b1:00.0-nvme-1 -> ../../nvme3n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b2:00.0-nvme-1 -> ../../nvme4n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b3:00.0-nvme-1 -> ../../nvme5n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b4:00.0-nvme-1 -> ../../nvme6n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:da:00.0-nvme-1 -> ../../nvme7n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:db:00.0-nvme-1 -> ../../nvme8n1
lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:dc:00.0-nvme-1 -> ../../nvme9n1
lrwxrwxrwx 1 root root  14 Jul 26 10:59 pci-0000:dd:00.0-nvme-1 -> ../../nvme10n1


Additional info:
# multipath -e -ll
Segmentation fault (core dumped)

# nvme list-subsys
nvme-subsys0 - NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100937
\
 +- nvme0 pcie 0000:3d:00.0 live
nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100933
\
 +- nvme1 pcie 0000:3f:00.0 live
nvme-subsys10 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN040100CD3P2BGN  Dell Express Flash NVMe P4610 3.2TB SFF
\
 +- nvme10 pcie 0000:dd:00.0 live
nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100766
\
 +- nvme2 pcie 0000:40:00.0 live
nvme-subsys3 - NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100935
\
 +- nvme3 pcie 0000:b1:00.0 live
nvme-subsys4 - NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100938
\
 +- nvme4 pcie 0000:b2:00.0 live
nvme-subsys5 - NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENE0NB02363
\
 +- nvme5 pcie 0000:b3:00.0 live
nvme-subsys6 - NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100931
\
 +- nvme6 pcie 0000:b4:00.0 live
nvme-subsys7 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN040100X13P2BGN  Dell Express Flash NVMe P4610 3.2TB SFF
\
 +- nvme7 pcie 0000:da:00.0 live
nvme-subsys8 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN0401019B3P2BGN  Dell Express Flash NVMe P4610 3.2TB SFF
\
 +- nvme8 pcie 0000:db:00.0 live
nvme-subsys9 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN0401011L3P2BGN  Dell Express Flash NVMe P4610 3.2TB SFF
\
 +- nvme9 pcie 0000:dc:00.0 live

# nvme list-ctrl /dev/nvme0
num of ctrls present: 2
[   0]:0x41
[   1]:0x42


# udevadm info --query=all --name=/dev/nvme0n1
P: /devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
N: nvme0n1
L: 0
S: disk/by-id/nvme-Dell_Ent_NVMe_AGN_MU_U.2_1.6TB_S61ENA0R100937
S: disk/by-id/nvme-eui.36314530521009370025384100000004
E: DEVPATH=/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
E: DEVNAME=/dev/nvme0n1
E: DEVTYPE=disk
E: DISKSEQ=10
E: MAJOR=259
E: MINOR=1
E: SUBSYSTEM=block
E: USEC_INITIALIZED=4344756
E: MPATH_SBIN_PATH=/sbin
E: DM_MULTIPATH_DEVICE_PATH=0
E: ID_SERIAL_SHORT=S61ENA0R100937
E: ID_WWN=eui.36314530521009370025384100000004
E: ID_MODEL=Dell Ent NVMe AGN MU U.2 1.6TB
E: ID_REVISION=2.4.0
E: ID_SERIAL=Dell_Ent_NVMe_AGN_MU_U.2_1.6TB_S61ENA0R100937
E: DEVLINKS=/dev/disk/by-id/nvme-Dell_Ent_NVMe_AGN_MU_U.2_1.6TB_S61ENA0R100937 /dev/disk/by-id/nvme-eui.36314530521009370025384100000004
E: TAGS=:systemd:
E: CURRENT_TAGS=:systems:
Comment 19 gopal krishna tiwari 2022-07-27 06:43:18 UTC 
(In reply to Volodymyr Rudniev from comment #18)
> Hello. 
> 
> We have the same problem with Dell Ent NVMe AGN MU U.2 1.6TB (Samsung
> PM1733). 
> Our setup has:
> - 4x NVMe Dell Express Flash NVMe P4610 3.2TB SFF (Intel P4610 3.2TB)
> - 7x NVMe Dell Ent NVMe AGN MU U.2 1.6TB (Samsung PM1733)
> - SATA 480 SSD
> 
> # nvme list
> Node                  SN                   Model                            
> Namespace Usage                      Format           FW Rev
> --------------------- --------------------
> ---------------------------------------- ---------
> -------------------------- ---------------- --------
> /dev/nvme0n1          S61ENA0R100937       Dell Ent NVMe AGN MU U.2 1.6TB   
> 1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
> /dev/nvme1n1          S61ENA0R100933       Dell Ent NVMe AGN MU U.2 1.6TB   
> 1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
> /dev/nvme10n1         PHLN040100CD3P2BGN   Dell Express Flash NVMe P4610
> 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25
> /dev/nvme2n1          S61ENA0R100766       Dell Ent NVMe AGN MU U.2 1.6TB   
> 1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
> /dev/nvme3n1          S61ENA0R100935       Dell Ent NVMe AGN MU U.2 1.6TB   
> 1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
> /dev/nvme4n1          S61ENA0R100938       Dell Ent NVMe AGN MU U.2 1.6TB   
> 1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
> /dev/nvme5n1          S61ENE0NB02363       Dell Ent NVMe AGN MU U.2 1.6TB   
> 1           1.60  TB /   1.60  TB    512   B +  0 B   2.0.2
> /dev/nvme6n1          S61ENA0R100931       Dell Ent NVMe AGN MU U.2 1.6TB   
> 1           1.60  TB /   1.60  TB    512   B +  0 B   2.4.0
> /dev/nvme7n1          PHLN040100X13P2BGN   Dell Express Flash NVMe P4610
> 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25
> /dev/nvme8n1          PHLN0401019B3P2BGN   Dell Express Flash NVMe P4610
> 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25
> /dev/nvme9n1          PHLN0401011L3P2BGN   Dell Express Flash NVMe P4610
> 3.2TB SFF  1           3.20  TB /   3.20  TB    512   B +  0 B   VDV1DP25
> 
> 
> /dev/disk/by-path/ doesn't contain symlinks for "7x NVMe Dell Ent NVMe AGN
> MU U.2 1.6TB":
> 
> # ls -al /dev/disk/by-path/
> total 0
> drwxr-xr-x 2 root root 200 Jul 26 13:01 .
> drwxr-xr-x 7 root root 140 Jul 26 13:01 ..
> lrwxrwxrwx 1 root root   9 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0 ->
> ../../sda
> lrwxrwxrwx 1 root root  10 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0-part1
> -> ../../sda1
> lrwxrwxrwx 1 root root  10 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0-part2
> -> ../../sda2
> lrwxrwxrwx 1 root root  10 Jul 26 13:01 pci-0000:86:00.0-scsi-0:2:0:0-part3
> -> ../../sda3
> lrwxrwxrwx 1 root root  13 Jul 26 13:01 pci-0000:da:00.0-nvme-1 ->
> ../../nvme7n1
> lrwxrwxrwx 1 root root  13 Jul 26 13:01 pci-0000:db:00.0-nvme-1 ->
> ../../nvme8n1
> lrwxrwxrwx 1 root root  13 Jul 26 13:01 pci-0000:dc:00.0-nvme-1 ->
> ../../nvme9n1
> lrwxrwxrwx 1 root root  14 Jul 26 13:01 pci-0000:dd:00.0-nvme-1 ->
> ../../nvme10n1
> 
> 
> As I can see, the problem repeats in all distros which use kernel v4.19 and
> newer.
> 
> I noticed that a kernel creates DEVPATH
> `/sys/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0/nvme0c0n1` for the
> problem drives, which should creates in multipath mode, according to kernel
> sources
> (https://github.com/torvalds/linux/blob/v4.19/drivers/nvme/host/multipath.
> c#L41). So I disabled NVMe multipath with "nvme-core.multipath=N" kernel
> cmdline parameter and it looks like a workaround:
> 
> # ls -al /dev/disk/by-path/
> total 0
> drwxr-xr-x 2 root root 340 Jul 26 10:59 .
> drwxr-xr-x 7 root root 140 Jul 26 10:59 ..
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:3d:00.0-nvme-1 ->
> ../../nvme0n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:3f:00.0-nvme-1 ->
> ../../nvme1n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:40:00.0-nvme-1 ->
> ../../nvme2n1
> lrwxrwxrwx 1 root root   9 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0 ->
> ../../sda
> lrwxrwxrwx 1 root root  10 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0-part1
> -> ../../sda1
> lrwxrwxrwx 1 root root  10 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0-part2
> -> ../../sda2
> lrwxrwxrwx 1 root root  10 Jul 26 10:59 pci-0000:86:00.0-scsi-0:2:0:0-part3
> -> ../../sda3
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b1:00.0-nvme-1 ->
> ../../nvme3n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b2:00.0-nvme-1 ->
> ../../nvme4n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b3:00.0-nvme-1 ->
> ../../nvme5n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:b4:00.0-nvme-1 ->
> ../../nvme6n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:da:00.0-nvme-1 ->
> ../../nvme7n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:db:00.0-nvme-1 ->
> ../../nvme8n1
> lrwxrwxrwx 1 root root  13 Jul 26 10:59 pci-0000:dc:00.0-nvme-1 ->
> ../../nvme9n1
> lrwxrwxrwx 1 root root  14 Jul 26 10:59 pci-0000:dd:00.0-nvme-1 ->
> ../../nvme10n1
> 
> 
> Additional info:
> # multipath -e -ll
> Segmentation fault (core dumped)
> 
> # nvme list-subsys
> nvme-subsys0 -
> NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100937
> \
>  +- nvme0 pcie 0000:3d:00.0 live
> nvme-subsys1 -
> NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100933
> \
>  +- nvme1 pcie 0000:3f:00.0 live
> nvme-subsys10 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN040100CD3P2BGN 
> Dell Express Flash NVMe P4610 3.2TB SFF
> \
>  +- nvme10 pcie 0000:dd:00.0 live
> nvme-subsys2 -
> NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100766
> \
>  +- nvme2 pcie 0000:40:00.0 live
> nvme-subsys3 -
> NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100935
> \
>  +- nvme3 pcie 0000:b1:00.0 live
> nvme-subsys4 -
> NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100938
> \
>  +- nvme4 pcie 0000:b2:00.0 live
> nvme-subsys5 -
> NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENE0NB02363
> \
>  +- nvme5 pcie 0000:b3:00.0 live
> nvme-subsys6 -
> NQN=nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S61ENA0R100931
> \
>  +- nvme6 pcie 0000:b4:00.0 live
> nvme-subsys7 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN040100X13P2BGN 
> Dell Express Flash NVMe P4610 3.2TB SFF
> \
>  +- nvme7 pcie 0000:da:00.0 live
> nvme-subsys8 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN0401019B3P2BGN 
> Dell Express Flash NVMe P4610 3.2TB SFF
> \
>  +- nvme8 pcie 0000:db:00.0 live
> nvme-subsys9 - NQN=nqn.2014.08.org.nvmexpress:80861028PHLN0401011L3P2BGN 
> Dell Express Flash NVMe P4610 3.2TB SFF
> \
>  +- nvme9 pcie 0000:dc:00.0 live
> 
> # nvme list-ctrl /dev/nvme0
> num of ctrls present: 2
> [   0]:0x41
> [   1]:0x42
> 
> 
> # udevadm info --query=all --name=/dev/nvme0n1
> P: /devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
> N: nvme0n1
> L: 0
> S: disk/by-id/nvme-Dell_Ent_NVMe_AGN_MU_U.2_1.6TB_S61ENA0R100937
> S: disk/by-id/nvme-eui.36314530521009370025384100000004
> E: DEVPATH=/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
> E: DEVNAME=/dev/nvme0n1
> E: DEVTYPE=disk
> E: DISKSEQ=10
> E: MAJOR=259
> E: MINOR=1
> E: SUBSYSTEM=block
> E: USEC_INITIALIZED=4344756
> E: MPATH_SBIN_PATH=/sbin
> E: DM_MULTIPATH_DEVICE_PATH=0
> E: ID_SERIAL_SHORT=S61ENA0R100937
> E: ID_WWN=eui.36314530521009370025384100000004
> E: ID_MODEL=Dell Ent NVMe AGN MU U.2 1.6TB
> E: ID_REVISION=2.4.0
> E: ID_SERIAL=Dell_Ent_NVMe_AGN_MU_U.2_1.6TB_S61ENA0R100937
> E:
> DEVLINKS=/dev/disk/by-id/nvme-Dell_Ent_NVMe_AGN_MU_U.2_1.6TB_S61ENA0R100937
> /dev/disk/by-id/nvme-eui.36314530521009370025384100000004
> E: TAGS=:systemd:
> E: CURRENT_TAGS=:systems:

Wondering if this is how it is designed. Here is the explanation from upstream 

~~~
If you're using nvme native multipathing, and your namespace reports
that it is multipath capable (ID_NS.NMIC), then the driver will set up
the virtual device for the visible block device.

If your namespace isn't multipath capable, you will only get the
physical device. That's just for pci, though; fabrics targets always
link to a virtual nvme subsystem device.

In the case you have a multipath namespace, the driver will also create
"hidden" block devices for each controller path that it found. As an
exampe, if you have multipath nvme /sys/block/nvme0n1, there should be a
/sys/block/nvme0c0n1, which should link to a physical device in sysfs
for pci.

~~~

Thanks
Gopal
Comment 20 Yu Watanabe 2022-09-20 17:38:18 UTC 
Hi, could anyone test the upstream PR?
https://github.com/systemd/systemd/pull/24766

I merely check RHBZ. Please feedback on the PR page, instead of here.

Thank you.
Comment 21 Yu Watanabe 2022-10-10 05:24:05 UTC 
Sorry, the upstream PR has a bug. Follow-up is here https://github.com/systemd/systemd/pull/24946
Comment 22 David Tardon 2022-10-14 06:25:43 UTC 
*** Bug 2124964 has been marked as a duplicate of this bug. ***
Comment 23 IBM Bug Proxy 2022-10-14 06:36:10 UTC 
------- Comment From wenxiong.com 2022-09-08 11:55 EDT-------
I didn't submit/upstream the patch yet. Can systemd team in Redhat review the patch or do the pull request for systemd github?

This is general issue in rhel87 and rhel91.

Thanks,
Wendy

------- Comment From wenxiong.com 2022-09-19 13:31 EDT-------
Pull Request to github/systemd.

https://github.com/systemd/systemd/pull/24748

------- Comment From wenxiong.com 2022-09-22 19:56 EDT-------
>>Possible replacement: https://github.com/systemd/systemd/pull/24766

[root@ltcrain119-lp3 by-path]# pwd
/dev/disk/by-path
[root@ltcrain119-lp3 by-path]# ls -l|more
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx. 1 root root 14 Sep 22 18:45 pci-0581:10:00.0-nvme-10 -> ../../nvme0n10
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-2 -> ../../nvme0n2
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-3 -> ../../nvme0n3
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-4 -> ../../nvme0n4
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-5 -> ../../nvme0n5
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-6 -> ../../nvme0n6
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-7 -> ../../nvme0n7
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-8 -> ../../nvme0n8
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0581:10:00.0-nvme-9 -> ../../nvme0n9
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0582:18:00.0-nvme-1 -> ../../nvme1n1
lrwxrwxrwx. 1 root root 14 Sep 22 18:45 pci-0582:18:00.0-nvme-10 -> ../../nvme1n10
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0582:18:00.0-nvme-2 -> ../../nvme1n2
lrwxrwxrwx. 1 root root 13 Sep 22 18:45 pci-0582:18:00.0-nvme-3 -> ../../nvme1n3

It works! Thanks!

------- Comment From wenxiong.com 2022-10-03 11:16 EDT-------
>>Possible replacement: https://github.com/systemd/systemd/pull/24766

Please pull the patch into rhel91.

Thanks,
Wendy

------- Comment From wenxiong.com 2022-10-10 14:49 EDT-------
Tested https://github.com/systemd/systemd/pull/24766 + 24946 on powerpc, works fine.

[root@ltcrain119-lp3 by-path]# pwd
/dev/disk/by-path

lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx. 1 root root 14 Oct 10 13:44 pci-0581:10:00.0-nvme-10 -> ../../nvme0n
10
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-2 -> ../../nvme0n2
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-3 -> ../../nvme0n3
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-4 -> ../../nvme0n4
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-5 -> ../../nvme0n5
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-6 -> ../../nvme0n6
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-7 -> ../../nvme0n7
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-8 -> ../../nvme0n8
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0581:10:00.0-nvme-9 -> ../../nvme0n9
lrwxrwxrwx. 1 root root 13 Oct 10 13:44 pci-0582:18:00.0-nvme-1 -> ../../nvme1n1
lrwxrwxrwx. 1 root root 14 Oct 10 13:44 pci-0582:18:00.0-nvme-10 -> ../../nvme1n

------- Comment From wenxiong.com 2022-10-13 14:07 EDT-------
Can you include these PR in rhel91?

https://github.com/systemd/systemd/pull/24946

Thanks,
Wendy
Comment 25 IBM Bug Proxy 2022-10-24 14:21:48 UTC 
------- Comment From wenxiong.com 2022-10-24 10:10 EDT-------
>>I found there is a "virtual" in DEVPATH for Samsung NVMe SSD on RHEL9; is this correct?

yes. Because nvme_core.multipath=y is set in rhel9 by default.

The output of udevadm looks good.
Comment 26 IBM Bug Proxy 2022-10-24 14:21:51 UTC 
Created attachment 1919991 [details]
patched systemd-udev
Comment 27 IBM Bug Proxy 2022-10-24 14:21:54 UTC 
Created attachment 1919992 [details]
patched systemd
Comment 28 IBM Bug Proxy 2022-10-24 14:21:56 UTC 
Created attachment 1919994 [details]
patch for systemd/udev
Comment 29 David Tardon 2022-10-25 11:59:57 UTC 
*** Bug 2133681 has been marked as a duplicate of this bug. ***
Comment 31 IBM Bug Proxy 2023-01-10 18:10:59 UTC 
------- Comment From wenxiong.com 2023-01-10 13:09 EDT-------
[root@ltcd41-lp11 by-path]# ls -l
total 0
lrwxrwxrwx. 1 root root 13 Jan  9 17:33 nvme-236 -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 Jan  9 17:33 nvme-237 -> ../../nvme2n2
lrwxrwxrwx. 1 root root 13 Jan  9 17:33 nvme-619 -> ../../nvme2n3
lrwxrwxrwx. 1 root root 13 Jan  9 17:33 nvme-620 -> ../../nvme2n4
lrwxrwxrwx. 1 root root 15 Jan  9 17:33 nvme-620-part1 -> ../../nvme2n4p1
lrwxrwxrwx. 1 root root 15 Jan  9 17:33 nvme-620-part2 -> ../../nvme2n4p2
lrwxrwxrwx. 1 root root 15 Jan  9 17:33 nvme-620-part3 -> ../../nvme2n4p3
lrwxrwxrwx. 1 root root 15 Jan  9 17:33 nvme-620-part4 -> ../../nvme2n4p4
lrwxrwxrwx. 1 root root 15 Jan  9 17:33 nvme-620-part5 -> ../../nvme2n4p5
lrwxrwxrwx. 1 root root 15 Jan  9 17:33 nvme-620-part6 -> ../../nvme2n4p6
[root@ltcd41-lp11 by-path]# uname -r
5.14.0-226.el9.ppc64le
[root@ltcd41-lp11 by-path]# pwd
/dev/disk/by-path
Comment 32 Jan Macku 2023-01-24 12:50:51 UTC 
This issue should be fixed via rebase to systemd v252.
Comment 38 errata-xmlrpc 2023-05-09 08:21:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (systemd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2531
Note You need to log in before you can comment on or make changes to this bug.