Bug 1697319 - [RFE] Use udev bindings for block device enumeration instead of symlinks in /dev/disk
Summary: [RFE] Use udev bindings for block device enumeration instead of symlinks in /...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-os-brick
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Cinder Bugs List
QA Contact: Evelina Shames
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-08 11:01 UTC by Tomáš Bžatek
Modified: 2024-04-04 16:14 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-04-04 16:14:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-2845 0 None None None 2022-12-08 22:43:33 UTC

Description Tomáš Bžatek 2019-04-08 11:01:34 UTC
(Note: this is a spin-off from bug 1684302 and we're still collecting information)

Version-Release number of selected component (if applicable):
os_brick-1.13.1-py3.6.egg (TBD, extracted from the logs)

Description of problem:
Custom udev rules (such as those shipped in sg3_utils) that affects presence of /dev/disk symlinks break OpenStack functionality. Although there may be certain guarantees of stability of block device symlinks within the /dev/disk structure, any custom rule either placed by a customer or shipped by some other package may affect their presence or naming. Other than /dev/disk/by-uuid where naming is relatively predictable and stable, symlinks e.g. under /dev/disk/by-path are not fully stable and are basically of free-form naming.

This has happened when e.g. we tried to package extra sg3_utils that (for some reason) removed /dev/disk/by-path/pci-%s-fc-%s-lun-%s symlinks that are required by os-brick. Similar change might have happened e.g. with udev update in case the symlink is enhanced by extra attribute to make it truly unique.

Looking at the upstream os-brick git master, I can see several issues:
https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/fibre_channel.py#L278
> host_device = "/dev/disk/by-path/pci-%s-fc-%s-lun-%s" % (
>                 pci_num,
>                 target_wwn,
>                 self._linuxscsi.process_lun_id(lun))

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/fibre_channel.py#L72
>     def get_search_path(self):
>         """Where do we look for FC based volumes."""
>         return '/dev/disk/by-path'

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/fibre_channel.py#L352
>      was_multipath = '/pci-' not in path_used

(there are more occurrences of accessing /dev for device enumeration in other source files)


Expected results:
os-brick should use proper udev bindings (e.g. pyudev) and match block devices by attributes. In case any attributes are missing, os-brick should ship their own set of udev rules so that it's guaranteed that any external change wouldn't affect availability of required information.

Comment 3 Miguel Murillo 2019-05-07 20:34:21 UTC
Please, do we have any estimation to fix this issue?

Best Regards,
Miguel

Comment 6 Pablo Caruana 2019-06-12 15:24:51 UTC
Hello Miguel,  Apologies if any delayed response as I was still  reading https://bugzilla.redhat.com/show_bug.cgi?id=1684302 and the Red Hat portal ticket 02337296 to have some background as probably some puzzle pieces still missing.


As ticket started with hpe-storage/dory as Kubernetes Flexvolume Driver for Docker and some rhel entries but probably missing some of the logs in debug for OS-Brick to compare exactly where the code or errors iterations?

Do you think you can provide those additional debug logs so we are comparing this scenario? my impression that from https://bugzilla.redhat.com/show_bug.cgi?id=1684302#c37  was not complete full answered and still some assumptions between  https://bugzilla.redhat.com/show_bug.cgi?id=1684302#c33 to https://bugzilla.redhat.com/show_bug.cgi?id=1684302#c52 without much other log details.

As it's mentioned, current os-brick code. when FC/FCOE environment os-brick currently expects symbolic link to be formed for any of new device under /dev/disk/by-path/ directory with format in the form of “pci-<H:C:T.L>-fc-0x<targetportwwn>-lun-<id>”  
The articles mentioned there was https://www.suse.com/support/kb/doc/?id=7022659 where using some fc-wwpn-id.rules defined rules for  SLES but no mentioning any Suse cloud considerations (their OSP product)

If this symbolic link is not available under specified directory, all vendors which uses os-brick code will face this issue as still on master tree, there is space to improvement but my impression from that  02337296 was probably missing some conversation how openshift over openstack and rhel iteration works (not surprises as most people are aware of their own product/code size). Habing said that, this method was still refered on all RHEL versions is the use or examination of links in /dev//by-id as mentioned on https://access.redhat.com/articles/17054 

You are correct than current os-brick code similar to the Fibre channel arch variations https://opendev.org/openstack/os-brick/src/branch/master/os_brick/initiator/connectors/fibre_channel.py still relies on  dev/disk/by-path/ that are just symbolic names (udev pro and limitations are well documented at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/persistent_naming)

Having all logs also we will help us for understanding if any additional considerations. Othwerwise, we are missing some relevant details if related to some functions like  _get_host_devices or something more complex like incorporating iterations from libudev, still this need some evaluation as  udev/kernel/blkid or any other source that can trigger any device correlation changes.

Have a great day.
Regards,
Pablo.

Comment 11 Tomáš Bžatek 2019-07-03 12:51:46 UTC
FYI, I've confirmed the structure and the presence of /dev/disk symlinks is considered a stable public ABI that shouldn't change in within a RHEL release. Not sure if it's written somewhere officially, this is a strong guideline though.

That means we'll try our best not to break presence of FC /dev/disk/by-path symlinks and this should guarantee os-brick functionality at the current level. We'll have to find some other way to fix the bug 1684302. We can decouple this bugreport from the customer case now.

However this bugreport still stands on its own, it's not an issue anymore, but a RFE rather. Using block device symlinks is still fragile and the code should use udev directly via bindings.

Comment 12 Leslie Satenstein 2023-05-02 16:00:54 UTC
I have been testing Fedora  39 rawhide, and herein is my bug report for my issues with /dev/disk/by-label and /dev/disk/by-partlabel.

I do apologize for it being wordy, summary, information omitted from the two outputs when the disklabel is less than 5 characters.

We start with my /etc/fstab...  


# /etc/fstab
# Created by anaconda on Mon May  1 18:51:01 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
#<file system>                            <mount>      <type> <options>          <dmp pass>  <xref>           <label/partlabel>
UUID=e1df91e8-0d21-4aab-9b83-ee6399f67c0b /            btrfs  defaults                   0 0 #/dev/sdb3       F39Root
UUID=0c666a93-5874-48e5-a3fb-def185a28228 /Backup      ext4   defaults                   1 2 #/dev/sdc1       Backup
UUID=9fd6c4b2-ba5f-4b9e-b10c-a0b6bb994167 /BackupOnE   btrfs  subvolid=5                 0 0 #/dev/sde1       BackupOnE
UUID=c389f15b-ff90-4900-8ca8-7cb6402213c4 /Development btrfs  subvolid=5                 0 0 #/dev/nvme0n1p10 Development
UUID=97604949-f682-4baa-9fc7-4172c3bc6faa /Leslie      ext4   defaults                   1 2 #/dev/nvme0n1p7  Leslie
UUID=23fbf855-3261-402b-8f7a-02837bd5e8c8 /LinuxStuff  btrfs  subvolid=5                 0 0 #/dev/nvme0n1p11 LinuxStuff
UUID=084cf7e5-2cef-406c-bad3-32035ad2245f /Music       btrfs  subvolid=5                 0 0 #/dev/sde2       Music
UUID=58ec912a-47d1-4760-baad-b9c0e6d79586 /backup      btrfs  subvolid=5                 0 0 #/dev/sdd5       backup
UUID=7a17c394-9f05-4edf-a316-d75930bc40b2 /boot        ext4   defaults                   1 2 #/dev/sdb2       F39Boot
UUID=16FE-1640                            /boot/efi    vfat   umask=0077,shortname=winnt 0 2 #/dev/sdb1       F39EFI
UUID=5cb61360-b7b0-4a27-b7fc-efc23508b780 /home2       btrfs  defaults,compress=zstd:2   0 0 #/dev/sdb4       
UUID=44bab0fb-5630-4a6d-9039-280ecb75295f /home        btrfs  defaults,compress=zstd:2   0 0 #/dev/sdb5       F
UUID=5843dd3b-0cf3-4192-a6db-9b94224c8c63 /share       btrfs  subvolid=5                 0 0 #/dev/nvme0n1p1  Share
UUID=901498fc-cbd7-4748-ae75-ccbce4e57ee1 /temp        ext4   defaults                   1 2 #/dev/nvme0n1p5  temp


As the ultimate information from blkid, the label for /home2 is shown.
It is not showing up with /dev/disk/by-label
As the ultimate information about /dev/disk/by-partlabel, the name field is empty.


#############################################################
########### SECOND VERSION with PARTUUID    #################
#############################################################
#
# /etc/fstab
# Created by anaconda on Mon May  1 18:51:01 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
#<file system>                                <mount>      <type> <options>          <dmp pass>  <xref>             <label/partlabel>
UUID=e1df91e8-0d21-4aab-9b83-ee6399f67c0b     /            btrfs  defaults                   0 0 #/dev/sdb3         F39Root
PARTUUID=8fac984c-8dce-46b9-a9d7-f42abc72116c /Backup      ext4   defaults                   1 2 #/dev/sdc1         Backup
PARTUUID=8e79b486-4e5f-48cf-b630-6efee60f0268 /BackupOnE   btrfs  subvolid=5                 0 0 #/dev/sde1         BackupOnE
PARTUUID=8e9168c3-d018-4a9f-b9e4-80b7bca3c6e9 /Development btrfs  subvolid=5                 0 0 #/dev/nvme0n1p10   nvme0Development
PARTUUID=3c90f0ad-2168-4c15-8167-91872fcc56ca /Leslie      ext4   defaults                   1 2 #/dev/nvme0n1p7    nvme0Leslie
PARTUUID=a8b25778-4563-4eae-80fe-e02bb66786cc /LinuxStuff  btrfs  subvolid=5                 0 0 #/dev/nvme0n1p11   nvme0LinuxStuff
PARTUUID=f04f0296-bb00-4114-967e-fc17fb49a8b8 /Music       btrfs  subvolid=5                 0 0 #/dev/sde2         Music
PARTUUID=435f1bc1-8c08-48ce-8420-a8c5c0e255a1 /backup      btrfs  subvolid=5                 0 0 #/dev/sdd5         sddbackup
PARTUUID=a12f0cd6-0094-4c02-b33d-6261eb8601e7 /boot        ext4   defaults                   1 2 #/dev/sdb2         F39Boot
UUID=16FE-1640                                /boot/efi    vfat   umask=0077,shortname=winnt 0 2 #/dev/sdb1         F39EFI
PARTUUID=c7e402da-2f18-4834-b1d6-d026b4fa3105 /home2       btrfs  defaults,compress=zstd:2   0 0 #/dev/sdb4         F39home
PARTUUID=5ab621d8-ec3e-4ade-bb09-d4a15e1fce4a /home        btrfs  defaults,compress=zstd:2   0 0 #/dev/sdb5         
PARTUUID=2cff995a-c733-4e34-9ea5-c9597217ab68 /share       btrfs  subvolid=5                 0 0 #/dev/nvme0n1p1    nvme0Share
PARTUUID=80f224c5-26ef-4c9c-b2c8-42bbf31ebd49 /temp        ext4   defaults                   1 2 #/dev/nvme0n1p5    nvme0temp

Note: /home is missing the PARTUUID-label (immediate previous listing)

What follows is the output from blkid /dev/sdb*

The PARTLABEL and LABEL are present for /home and /home2
/dev/sdb: PTUUID="f7d48706-0978-4a5c-a7e2-c9bfb31adf9c" PTTYPE="gpt"
/dev/sdb1: SEC_TYPE="msdos" LABEL_FATBOOT="F39EFI" LABEL="F39EFI" UUID="16FE-1640" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="0f5dd947-3297-409b-8c07-6d578a7b5af4"
/dev/sdb2: LABEL="F39Boot" UUID="7a17c394-9f05-4edf-a316-d75930bc40b2" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="F39Boot" PARTUUID="a12f0cd6-0094-4c02-b33d-6261eb8601e7"
/dev/sdb3: LABEL="F39Root" UUID="e1df91e8-0d21-4aab-9b83-ee6399f67c0b" UUID_SUB="0e4b7368-be77-488a-8afd-1935161f6739" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="F39Root" PARTUUID="2fabd78c-5fa9-4913-8750-6d542f445a72"
/dev/sdb4: LABEL="F39home" UUID="5cb61360-b7b0-4a27-b7fc-efc23508b780" UUID_SUB="a9228664-bedc-493c-b160-b8ad34a4aa1c" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="F39home" PARTUUID="c7e402da-2f18-4834-b1d6-d026b4fa3105"
/dev/sdb5: LABEL="F" UUID="44bab0fb-5630-4a6d-9039-280ecb75295f" UUID_SUB="d77cd175-40d1-41a0-b23f-ab06132de292" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="F39home" PARTUUID="5ab621d8-ec3e-4ade-bb09-d4a15e1fce4a"


By the way, as an aside, the utility to produce the fstab in either format is available on request (I wrote it and I make it freely available as per gpl3)


Note You need to log in before you can comment on or make changes to this bug.