(Note: this is a spin-off from bug 1684302 and we're still collecting information) Version-Release number of selected component (if applicable): os_brick-1.13.1-py3.6.egg (TBD, extracted from the logs) Description of problem: Custom udev rules (such as those shipped in sg3_utils) that affects presence of /dev/disk symlinks break OpenStack functionality. Although there may be certain guarantees of stability of block device symlinks within the /dev/disk structure, any custom rule either placed by a customer or shipped by some other package may affect their presence or naming. Other than /dev/disk/by-uuid where naming is relatively predictable and stable, symlinks e.g. under /dev/disk/by-path are not fully stable and are basically of free-form naming. This has happened when e.g. we tried to package extra sg3_utils that (for some reason) removed /dev/disk/by-path/pci-%s-fc-%s-lun-%s symlinks that are required by os-brick. Similar change might have happened e.g. with udev update in case the symlink is enhanced by extra attribute to make it truly unique. Looking at the upstream os-brick git master, I can see several issues: https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/fibre_channel.py#L278 > host_device = "/dev/disk/by-path/pci-%s-fc-%s-lun-%s" % ( > pci_num, > target_wwn, > self._linuxscsi.process_lun_id(lun)) https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/fibre_channel.py#L72 > def get_search_path(self): > """Where do we look for FC based volumes.""" > return '/dev/disk/by-path' https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/fibre_channel.py#L352 > was_multipath = '/pci-' not in path_used (there are more occurrences of accessing /dev for device enumeration in other source files) Expected results: os-brick should use proper udev bindings (e.g. pyudev) and match block devices by attributes. In case any attributes are missing, os-brick should ship their own set of udev rules so that it's guaranteed that any external change wouldn't affect availability of required information.
Please, do we have any estimation to fix this issue? Best Regards, Miguel
Hello Miguel, Apologies if any delayed response as I was still reading https://bugzilla.redhat.com/show_bug.cgi?id=1684302 and the Red Hat portal ticket 02337296 to have some background as probably some puzzle pieces still missing. As ticket started with hpe-storage/dory as Kubernetes Flexvolume Driver for Docker and some rhel entries but probably missing some of the logs in debug for OS-Brick to compare exactly where the code or errors iterations? Do you think you can provide those additional debug logs so we are comparing this scenario? my impression that from https://bugzilla.redhat.com/show_bug.cgi?id=1684302#c37 was not complete full answered and still some assumptions between https://bugzilla.redhat.com/show_bug.cgi?id=1684302#c33 to https://bugzilla.redhat.com/show_bug.cgi?id=1684302#c52 without much other log details. As it's mentioned, current os-brick code. when FC/FCOE environment os-brick currently expects symbolic link to be formed for any of new device under /dev/disk/by-path/ directory with format in the form of “pci-<H:C:T.L>-fc-0x<targetportwwn>-lun-<id>” The articles mentioned there was https://www.suse.com/support/kb/doc/?id=7022659 where using some fc-wwpn-id.rules defined rules for SLES but no mentioning any Suse cloud considerations (their OSP product) If this symbolic link is not available under specified directory, all vendors which uses os-brick code will face this issue as still on master tree, there is space to improvement but my impression from that 02337296 was probably missing some conversation how openshift over openstack and rhel iteration works (not surprises as most people are aware of their own product/code size). Habing said that, this method was still refered on all RHEL versions is the use or examination of links in /dev//by-id as mentioned on https://access.redhat.com/articles/17054 You are correct than current os-brick code similar to the Fibre channel arch variations https://opendev.org/openstack/os-brick/src/branch/master/os_brick/initiator/connectors/fibre_channel.py still relies on dev/disk/by-path/ that are just symbolic names (udev pro and limitations are well documented at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/persistent_naming) Having all logs also we will help us for understanding if any additional considerations. Othwerwise, we are missing some relevant details if related to some functions like _get_host_devices or something more complex like incorporating iterations from libudev, still this need some evaluation as udev/kernel/blkid or any other source that can trigger any device correlation changes. Have a great day. Regards, Pablo.
FYI, I've confirmed the structure and the presence of /dev/disk symlinks is considered a stable public ABI that shouldn't change in within a RHEL release. Not sure if it's written somewhere officially, this is a strong guideline though. That means we'll try our best not to break presence of FC /dev/disk/by-path symlinks and this should guarantee os-brick functionality at the current level. We'll have to find some other way to fix the bug 1684302. We can decouple this bugreport from the customer case now. However this bugreport still stands on its own, it's not an issue anymore, but a RFE rather. Using block device symlinks is still fragile and the code should use udev directly via bindings.
I have been testing Fedora 39 rawhide, and herein is my bug report for my issues with /dev/disk/by-label and /dev/disk/by-partlabel. I do apologize for it being wordy, summary, information omitted from the two outputs when the disklabel is less than 5 characters. We start with my /etc/fstab... # /etc/fstab # Created by anaconda on Mon May 1 18:51:01 2023 # # Accessible filesystems, by reference, are maintained under '/dev/disk/'. # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info. # # After editing this file, run 'systemctl daemon-reload' to update systemd # units generated from this file. # #<file system> <mount> <type> <options> <dmp pass> <xref> <label/partlabel> UUID=e1df91e8-0d21-4aab-9b83-ee6399f67c0b / btrfs defaults 0 0 #/dev/sdb3 F39Root UUID=0c666a93-5874-48e5-a3fb-def185a28228 /Backup ext4 defaults 1 2 #/dev/sdc1 Backup UUID=9fd6c4b2-ba5f-4b9e-b10c-a0b6bb994167 /BackupOnE btrfs subvolid=5 0 0 #/dev/sde1 BackupOnE UUID=c389f15b-ff90-4900-8ca8-7cb6402213c4 /Development btrfs subvolid=5 0 0 #/dev/nvme0n1p10 Development UUID=97604949-f682-4baa-9fc7-4172c3bc6faa /Leslie ext4 defaults 1 2 #/dev/nvme0n1p7 Leslie UUID=23fbf855-3261-402b-8f7a-02837bd5e8c8 /LinuxStuff btrfs subvolid=5 0 0 #/dev/nvme0n1p11 LinuxStuff UUID=084cf7e5-2cef-406c-bad3-32035ad2245f /Music btrfs subvolid=5 0 0 #/dev/sde2 Music UUID=58ec912a-47d1-4760-baad-b9c0e6d79586 /backup btrfs subvolid=5 0 0 #/dev/sdd5 backup UUID=7a17c394-9f05-4edf-a316-d75930bc40b2 /boot ext4 defaults 1 2 #/dev/sdb2 F39Boot UUID=16FE-1640 /boot/efi vfat umask=0077,shortname=winnt 0 2 #/dev/sdb1 F39EFI UUID=5cb61360-b7b0-4a27-b7fc-efc23508b780 /home2 btrfs defaults,compress=zstd:2 0 0 #/dev/sdb4 UUID=44bab0fb-5630-4a6d-9039-280ecb75295f /home btrfs defaults,compress=zstd:2 0 0 #/dev/sdb5 F UUID=5843dd3b-0cf3-4192-a6db-9b94224c8c63 /share btrfs subvolid=5 0 0 #/dev/nvme0n1p1 Share UUID=901498fc-cbd7-4748-ae75-ccbce4e57ee1 /temp ext4 defaults 1 2 #/dev/nvme0n1p5 temp As the ultimate information from blkid, the label for /home2 is shown. It is not showing up with /dev/disk/by-label As the ultimate information about /dev/disk/by-partlabel, the name field is empty. ############################################################# ########### SECOND VERSION with PARTUUID ################# ############################################################# # # /etc/fstab # Created by anaconda on Mon May 1 18:51:01 2023 # # Accessible filesystems, by reference, are maintained under '/dev/disk/'. # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info. # # After editing this file, run 'systemctl daemon-reload' to update systemd # units generated from this file. # #<file system> <mount> <type> <options> <dmp pass> <xref> <label/partlabel> UUID=e1df91e8-0d21-4aab-9b83-ee6399f67c0b / btrfs defaults 0 0 #/dev/sdb3 F39Root PARTUUID=8fac984c-8dce-46b9-a9d7-f42abc72116c /Backup ext4 defaults 1 2 #/dev/sdc1 Backup PARTUUID=8e79b486-4e5f-48cf-b630-6efee60f0268 /BackupOnE btrfs subvolid=5 0 0 #/dev/sde1 BackupOnE PARTUUID=8e9168c3-d018-4a9f-b9e4-80b7bca3c6e9 /Development btrfs subvolid=5 0 0 #/dev/nvme0n1p10 nvme0Development PARTUUID=3c90f0ad-2168-4c15-8167-91872fcc56ca /Leslie ext4 defaults 1 2 #/dev/nvme0n1p7 nvme0Leslie PARTUUID=a8b25778-4563-4eae-80fe-e02bb66786cc /LinuxStuff btrfs subvolid=5 0 0 #/dev/nvme0n1p11 nvme0LinuxStuff PARTUUID=f04f0296-bb00-4114-967e-fc17fb49a8b8 /Music btrfs subvolid=5 0 0 #/dev/sde2 Music PARTUUID=435f1bc1-8c08-48ce-8420-a8c5c0e255a1 /backup btrfs subvolid=5 0 0 #/dev/sdd5 sddbackup PARTUUID=a12f0cd6-0094-4c02-b33d-6261eb8601e7 /boot ext4 defaults 1 2 #/dev/sdb2 F39Boot UUID=16FE-1640 /boot/efi vfat umask=0077,shortname=winnt 0 2 #/dev/sdb1 F39EFI PARTUUID=c7e402da-2f18-4834-b1d6-d026b4fa3105 /home2 btrfs defaults,compress=zstd:2 0 0 #/dev/sdb4 F39home PARTUUID=5ab621d8-ec3e-4ade-bb09-d4a15e1fce4a /home btrfs defaults,compress=zstd:2 0 0 #/dev/sdb5 PARTUUID=2cff995a-c733-4e34-9ea5-c9597217ab68 /share btrfs subvolid=5 0 0 #/dev/nvme0n1p1 nvme0Share PARTUUID=80f224c5-26ef-4c9c-b2c8-42bbf31ebd49 /temp ext4 defaults 1 2 #/dev/nvme0n1p5 nvme0temp Note: /home is missing the PARTUUID-label (immediate previous listing) What follows is the output from blkid /dev/sdb* The PARTLABEL and LABEL are present for /home and /home2 /dev/sdb: PTUUID="f7d48706-0978-4a5c-a7e2-c9bfb31adf9c" PTTYPE="gpt" /dev/sdb1: SEC_TYPE="msdos" LABEL_FATBOOT="F39EFI" LABEL="F39EFI" UUID="16FE-1640" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="0f5dd947-3297-409b-8c07-6d578a7b5af4" /dev/sdb2: LABEL="F39Boot" UUID="7a17c394-9f05-4edf-a316-d75930bc40b2" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="F39Boot" PARTUUID="a12f0cd6-0094-4c02-b33d-6261eb8601e7" /dev/sdb3: LABEL="F39Root" UUID="e1df91e8-0d21-4aab-9b83-ee6399f67c0b" UUID_SUB="0e4b7368-be77-488a-8afd-1935161f6739" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="F39Root" PARTUUID="2fabd78c-5fa9-4913-8750-6d542f445a72" /dev/sdb4: LABEL="F39home" UUID="5cb61360-b7b0-4a27-b7fc-efc23508b780" UUID_SUB="a9228664-bedc-493c-b160-b8ad34a4aa1c" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="F39home" PARTUUID="c7e402da-2f18-4834-b1d6-d026b4fa3105" /dev/sdb5: LABEL="F" UUID="44bab0fb-5630-4a6d-9039-280ecb75295f" UUID_SUB="d77cd175-40d1-41a0-b23f-ab06132de292" BLOCK_SIZE="4096" TYPE="btrfs" PARTLABEL="F39home" PARTUUID="5ab621d8-ec3e-4ade-bb09-d4a15e1fce4a" By the way, as an aside, the utility to produce the fstab in either format is available on request (I wrote it and I make it freely available as per gpl3)