RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1809053 - [Azure][RHEL7.6]Inconsistent creation of symlinks in /dev/disk/by-path in Azure VMs
Summary: [Azure][RHEL7.6]Inconsistent creation of symlinks in /dev/disk/by-path in Azu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.6
Hardware: Unspecified
OS: Linux
medium
low
Target Milestone: rc
: ---
Assignee: David Tardon
QA Contact: xuli
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-02 10:45 UTC by Jose Castillo
Modified: 2023-12-15 17:25 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-29 20:32:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github systemd-rhel rhel-7 pull 93 0 None closed (#1809053) v3: Properly parsing SCSI Hyperv devices (#8509) 2020-09-11 07:58:33 UTC
Github systemd systemd pull 8509 0 None closed v3: Properly parsing SCSI Hyperv devices 2020-09-11 07:58:35 UTC
Red Hat Product Errata RHSA-2020:4007 0 None None None 2020-09-29 20:33:15 UTC

Description Jose Castillo 2020-03-02 10:45:33 UTC 
Description of problem:
In Azure VMS we are seeing that the paths inside the directory /dev/disk/by-path are created inconsistently. In the example below you can see that all devices both on scsi0 and scsi1 are mapped to '-scsi-0:0:0:0-' . Because of this the sdc on scsi1 devices  /dev/sdc disk devices and /dev/sdc1 partition devices are hiding/overwriting the /dev/sda and /dev/sda1.

	$ find /dev/disk/azure -ls
	 14229    0 drwxr-xr-x   3 root     root          140 Feb 17 15:53 /dev/disk/azure
	 23455    0 lrwxrwxrwx   1 root     root           10 Feb 19 18:34 /dev/disk/azure/resource-part1 -> ../../sdb1
	 15470    0 drwxr-xr-x   2 root     root           80 Feb 17 15:52 /dev/disk/azure/scsi1
	 15489    0 lrwxrwxrwx   1 root     root           13 Feb 19 18:34 /dev/disk/azure/scsi1/lun0-part1 -> ../../../sdc1
	 15471    0 lrwxrwxrwx   1 root     root           12 Feb 19 18:34 /dev/disk/azure/scsi1/lun0 -> ../../../sdc
	 14245    0 lrwxrwxrwx   1 root     root           10 Feb 19 18:34 /dev/disk/azure/root-part1 -> ../../sda1
	 14932    0 lrwxrwxrwx   1 root     root            9 Feb 19 18:34 /dev/disk/azure/resource -> ../../sdb
	 14230    0 lrwxrwxrwx   1 root     root            9 Feb 19 18:34 /dev/disk/azure/root -> ../../sda

	$ find /dev/disk/by-path -ls
	 10544    0 drwxr-xr-x   2 root     root          120 Feb 19 18:34 /dev/disk/by-path
	4732282    0 lrwxrwxrwx   1 root     root           10 Feb 19 18:34 /dev/disk/by-path/acpi-VMBUS:01-scsi-0:0:0:0-part1 -> ../../sdc1
	4732960    0 lrwxrwxrwx   1 root     root            9 Feb 19 18:34 /dev/disk/by-path/acpi-VMBUS:01-scsi-0:0:0:0 -> ../../sdc
	 23464    0 lrwxrwxrwx   1 root     root           10 Feb 19 18:34 /dev/disk/by-path/acpi-VMBUS:01-scsi-0:0:1:0-part1 -> ../../sdb1
	 10545    0 lrwxrwxrwx   1 root     root            9 Feb 19 18:34 /dev/disk/by-path/acpi-VMBUS:01-scsi-0:0:1:0 -> ../../sdb

	$ cat sos_commands/block/lsblk
	NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
	sda      8:0    0  32G  0 disk 
	`-sda1   8:1    0  32G  0 part /
	sdb      8:16   0  32G  0 disk 
	`-sdb1   8:17   0  32G  0 part /mnt/resource
	sdc      8:32   0  10G  0 disk 
	`-sdc1   8:33   0  10G  0 part /lvuser1

	$ cat sos_commands/scsi/lsscsi 
	[2:0:0:0]    disk    Msft     Virtual Disk     1.0   /dev/sda 
	[3:0:1:0]    disk    Msft     Virtual Disk     1.0   /dev/sdb 
	[5:0:0:0]    disk    Msft     Virtual Disk     1.0   /dev/sdc 

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 7
Reproduced so far in as early as systemd-219-62.el7.x86_64 and
as late as systemd-219-62.el7_6.11.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Add multiple virtual SCSI devices in an Azure VM.
2. Simply check the content of /dev/disk/by-path


Actual results:
Not all devices appear in /dev/disk/by-path

Expected results:
All devices' symlinks should appear inside /dev/disk/by-path
Comment 2 Jose Castillo 2020-03-03 08:22:21 UTC 
I'm manipulating the systemd sources, specifically the functions:

sd_device_get_syspath()
udev_device_get_syspath()
builtin_path_id()
handle_scsi_hyperv()

And reproducing with scsi_debug, because I don't have an AZURE vm and I haven't been able to find a procedure to get one. But if you have one available where I can test, I can progress a bit more with my debug code.
Comment 3 xuli 2020-03-03 08:38:17 UTC 
Test on Hyper-V 2019 host and Azure with RHEL 7.8 kernel 3.10.0-1127.el7.x86_64 by me and Yuxin, can reproduce this issue.

Test steps:
1. Add 3 new scsi disks, 2 disks are located under the same controller with system disk, and the 3rd disk added to the other SCSI controller.
2. After vm boot up, check below information.
 
Step 1: # ll /dev/sd*
brw-rw----. 1 root disk 8,  0 Mar  3 15:19 /dev/sda
brw-rw----. 1 root disk 8,  1 Mar  3 15:19 /dev/sda1
brw-rw----. 1 root disk 8,  2 Mar  3 15:19 /dev/sda2
brw-rw----. 1 root disk 8,  3 Mar  3 15:19 /dev/sda3
brw-rw----. 1 root disk 8, 16 Mar  3 15:19 /dev/sdb
brw-rw----. 1 root disk 8, 17 Mar  3 15:19 /dev/sdb1
brw-rw----. 1 root disk 8, 32 Mar  3 15:19 /dev/sdc
brw-rw----. 1 root disk 8, 48 Mar  3 15:19 /dev/sdd

# ls -alh /dev/disk/by-path                 (There is no ../../sdd path)

lrwxrwxrwx. 1 root root   9 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:1 -> ../../sdb
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:1-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:2 -> ../../sdc

Step 2: Use "fdisk /dev/sdd" to show /dev/sdd information

Step 3: #ls -alh /dev/disk/by-path           (There is ../../sdd path, but without ../../sda after fdisk )

lrwxrwxrwx. 1 root root   9 Mar  3 15:34 acpi-VMBUS:00-scsi-0:0:0:0 -> ../../sdd
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:1 -> ../../sdb
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:1-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 acpi-VMBUS:00-scsi-0:0:0:2 -> ../../sdc

Other information:
# rpm -qa | grep -i udev
python-gudev-147.2-7.el7.x86_64
system-config-printer-udev-1.4.1-23.el7.x86_64
libgudev1-219-73.el7.1.x86_64
python-pyudev-0.15-9.el7.noarch
# rpm -qa | grep -i systemd
systemd-python-219-73.el7.1.x86_64
systemd-libs-219-73.el7.1.x86_64
systemd-sysv-219-73.el7.1.x86_64
systemd-219-73.el7.1.x86_64

# ls -alh /dev/disk/by-id

lrwxrwxrwx. 1 root root  10 Mar  3 15:19 dm-name-rhel_bootp--73--199--15-root -> ../../dm-0
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 dm-name-rhel_bootp--73--199--15-swap -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 dm-uuid-LVM-KAfrJASN3DD7gca3I0DuApYNE2Fqf9ScPv1DpKJoEBb6hnIld3w3FnuVc9MxXD0i -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 dm-uuid-LVM-KAfrJASN3DD7gca3I0DuApYNE2Fqf9ScsMVpuPuq9CbMDqREkfQXKxWZrpdfEzfq -> ../../dm-0
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 lvm-pv-uuid-Jf3Sid-JxxL-wPT6-gtaE-11nZ-N87i-LJnXQQ -> ../../sda3
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 scsi-360022480877f26dc71cc5975143d2769 -> ../../sdb
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 scsi-360022480877f26dc71cc5975143d2769-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 scsi-360022480b448dc2d2dc9ded2a07b28b6 -> ../../sdc
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 scsi-360022480e6d94fd911b8026ca7e1290b -> ../../sda
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 scsi-360022480e6d94fd911b8026ca7e1290b-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 scsi-360022480e6d94fd911b8026ca7e1290b-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 scsi-360022480e6d94fd911b8026ca7e1290b-part3 -> ../../sda3
lrwxrwxrwx. 1 root root   9 Mar  3 15:34 scsi-360022480f57c4ce94a48ac7068d52a0f -> ../../sdd
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 wwn-0x60022480877f26dc71cc5975143d2769 -> ../../sdb
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 wwn-0x60022480877f26dc71cc5975143d2769-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 wwn-0x60022480b448dc2d2dc9ded2a07b28b6 -> ../../sdc
lrwxrwxrwx. 1 root root   9 Mar  3 15:19 wwn-0x60022480e6d94fd911b8026ca7e1290b -> ../../sda
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 wwn-0x60022480e6d94fd911b8026ca7e1290b-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 wwn-0x60022480e6d94fd911b8026ca7e1290b-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Mar  3 15:19 wwn-0x60022480e6d94fd911b8026ca7e1290b-part3 -> ../../sda3
lrwxrwxrwx. 1 root root   9 Mar  3 15:34 wwn-0x60022480f57c4ce94a48ac7068d52a0f -> ../../sdd


Actual results:

Inconsistent creation of symlinks in /dev/disk/by-path for disks.

We also tested RHEL 8 kernel 4.18.0-184.el8.x86_64 and systemd-udev-239-27.el8.x86_64 on both Azure and Hyper-V, /dev/disk/by-path could display normally.

[root@bootp-73-199-166 ~]# ls /dev/disk/by-path -alh

lrwxrwxrwx. 1 root root   9 Mar  3 16:13 acpi-VMBUS:00-vmbus-3234cbc19d98492eb4628c35a7d4785e-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  10 Mar  3 16:13 acpi-VMBUS:00-vmbus-3234cbc19d98492eb4628c35a7d4785e-lun-0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Mar  3 16:13 acpi-VMBUS:00-vmbus-3234cbc19d98492eb4628c35a7d4785e-lun-0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Mar  3 16:13 acpi-VMBUS:00-vmbus-3234cbc19d98492eb4628c35a7d4785e-lun-0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root   9 Mar  3 16:13 acpi-VMBUS:00-vmbus-3234cbc19d98492eb4628c35a7d4785e-lun-1 -> ../../sdb
lrwxrwxrwx. 1 root root  10 Mar  3 16:13 acpi-VMBUS:00-vmbus-3234cbc19d98492eb4628c35a7d4785e-lun-1-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root   9 Mar  3 16:13 acpi-VMBUS:00-vmbus-3234cbc19d98492eb4628c35a7d4785e-lun-2 -> ../../sdc
lrwxrwxrwx. 1 root root   9 Mar  3 16:13 acpi-VMBUS:00-vmbus-e6f6aa9e972b43c08087db6d9c0b0428-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root   9 Mar  3 16:13 acpi-VMBUS:00-vmbus-e6f6aa9e972b43c08087db6d9c0b0428-lun-1 -> ../../sde

There is documention methioned that "The Path attribute is unreliable, and Red Hat does not recommend using it." 
@ https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/managing_storage_devices/index#assembly_overview-of-persistent-naming-attributes_managing-storage-devices.

In summary, this issue reproduces on RHEL 7.8 but not RHEL 8.2. May I ask should we fix it in RHEL 7.9 or only suggest users to use /dev/disk/by-id, or uuid but not by-path?

Thank you.
Comment 4 xuli 2020-03-03 09:02:23 UTC 
(In reply to Jose Castillo from comment #2)
> I'm manipulating the systemd sources, specifically the functions:
> 
> sd_device_get_syspath()
> udev_device_get_syspath()
> builtin_path_id()
> handle_scsi_hyperv()
> 
> And reproducing with scsi_debug, because I don't have an AZURE vm and I
> haven't been able to find a procedure to get one. But if you have one
> available where I can test, I can progress a bit more with my debug code.

Hi Jose,

Have sent an email to you about Hyper-V RHEL 7 VM IP for your debugging.

If anything more needed, feel free to let us know.
Comment 11 Lukáš Nykrýn 2020-03-17 14:35:09 UTC 
fix merged to github master branch -> https://github.com/systemd-rhel/rhel-7/pull/93
Comment 30 errata-xmlrpc 2020-09-29 20:32:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: systemd security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4007
Note You need to log in before you can comment on or make changes to this bug.