== Comment: #0 - Stefan Haberland <stefan.haberland.com> - 2024-11-20 06:08:26 == F41 installer fails to install on s390 using FCP devices (SCSI). Looks like the WWID is not passed to the installer correctly. Please see attached pictures of a working F40 and a not working F41 installer. I also noticed differences in 'udevadm test' output. On F41 at least DM_WWN and DM_SERIAL are missing. 'udevadm test' from not working F41 installer: --- Properties: DEVPATH=/devices/virtual/block/dm-0 DEVNAME=/dev/dm-0 DEVTYPE=disk DISKSEQ=20 MAJOR=253 MINOR=0 SUBSYSTEM=block ACTION=add TAGS=:systemd: DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 DM_UDEV_PRIMARY_SOURCE_FLAG=1 DM_UDEV_RULES_VSN=3 DM_UDEV_DISABLE_OTHER_RULES_FLAG= DM_ACTIVATION=1 DM_NAME=mpatha DM_UUID=mpath-36005076307ffd72c000000000000184b .DM_SUSPENDED=0 DEVLINKS=/dev/mapper/mpatha /dev/disk/by-id/dm-name-mpatha /dev/disk/by-id/dm-uuid-mpath-36005076307ffd72c000000000000184b MPATH_DEVICE_READY=1 CURRENT_TAGS=:systemd: USEC_INITIALIZED=17093220 ID_PROCESSING=1 Tags: systemd Device node symlinks: (priority=50) /dev/mapper/mpatha /dev/disk/by-id/dm-name-mpatha /dev/disk/by-id/dm-uuid-mpath-36005076307ffd72c000000000000184b Inotify watch: enabled Device node group: disk (gid=6) Queued commands: RUN{program} : /sbin/kpartx -un /dev/dm-0 --- 'udevadm test' from working F40 installer: --- DEVPATH=/devices/virtual/block/dm-0 DEVNAME=/dev/dm-0 DEVTYPE=disk DISKSEQ=20 MAJOR=253 MINOR=0 SUBSYSTEM=block ACTION=add TAGS=:systemd: DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 DM_UDEV_PRIMARY_SOURCE_FLAG=1 DM_UDEV_RULES_VSN=2 DM_ACTIVATION=1 DM_NAME=mpatha DM_UUID=mpath-36005076307ffd72c000000000000184b DM_SUSPENDED=0 DEVLINKS=/dev/disk/by-id/scsi-36005076307ffd72c000000000000184b /dev/disk/by-id/dm-uuid-mpath-36005076307ffd72c000000000000184b /dev/disk/by-id/dm-name-mpatha /dev/mapper/mpatha /dev/disk/by-id/wwn-0x6005076307ffd72c000000000000184b MPATH_DEVICE_READY=1 .MPATH_DEVICE_READY_OLD=1 MPATH_SBIN_PATH=/sbin DM_TYPE=scsi DM_WWN=0x6005076307ffd72c000000000000184b DM_SERIAL=36005076307ffd72c000000000000184b CURRENT_TAGS=:systemd: SYSTEMD_READY=1 USEC_INITIALIZED=14947811 run: '/sbin/kpartx -un /dev/dm-0' --- == Comment: #3 - Stefan Haberland <stefan.haberland.com> - 2024-11-20 07:55:07 == additional information, udevadm info does not show that much of a difference: working F40: --- anaconda root@a46lp76 ~]# udevadm info /dev/mapper/mpatha P: /devices/virtual/block/dm-0 M: dm-0 R: 0 U: block T: disk D: b 253:0 N: dm-0 L: 50 S: disk/by-id/dm-name-mpatha S: disk/by-id/wwn-0x6005076307ffd72c00000000000019e1 S: disk/by-id/dm-uuid-mpath-36005076307ffd72c00000000000019e1 S: mapper/mpatha S: disk/by-id/scsi-36005076307ffd72c00000000000019e1 Q: 12 E: DEVPATH=/devices/virtual/block/dm-0 E: DEVNAME=/dev/dm-0 E: DEVTYPE=disk E: DISKSEQ=12 E: MAJOR=253 E: MINOR=0 E: SUBSYSTEM=block E: USEC_INITIALIZED=34641962 E: DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 E: DM_UDEV_PRIMARY_SOURCE_FLAG=1 E: DM_SUBSYSTEM_UDEV_FLAG0=1 E: DM_ACTIVATION=0 E: DM_NAME=mpatha E: DM_UUID=mpath-36005076307ffd72c00000000000019e1 E: DM_SUSPENDED=0 E: DM_UDEV_RULES_VSN=2 E: MPATH_DEVICE_READY=1 E: MPATH_SBIN_PATH=/sbin E: MPATH_UNCHANGED=1 E: DM_TYPE=scsi E: DM_WWN=0x6005076307ffd72c00000000000019e1 E: DM_SERIAL=36005076307ffd72c00000000000019e1 E: ID_PART_TABLE_UUID=3a9d5433-6c8b-4739-b725-65265833d7bd E: ID_PART_TABLE_TYPE=gpt E: NVME_HOST_IFACE=none E: SYSTEMD_READY=1 E: DEVLINKS=/dev/disk/by-id/dm-name-mpatha /dev/disk/by-id/wwn-0x6005076307ffd72c00000000000019e1 /dev/disk/by-id/dm-uuid-mpath-36005076307ffd72c00000000000019e1 /dev/mapper/mpatha /dev/disk/by-id/scsi-36005076307ffd72c00000000000019e1 E: TAGS=:systemd: E: CURRENT_TAGS=:systemd: --- not working F41: --- [anaconda root@a46lp76 ~]# udevadm info /dev/mapper/mpathc P: /devices/virtual/block/dm-2 M: dm-2 R: 2 U: block T: disk D: b 253:2 N: dm-2 L: 50 S: disk/by-id/scsi-36005076307ffd72c00000000000019e1 S: mapper/mpathc S: disk/by-id/dm-name-mpathc S: disk/by-id/dm-uuid-mpath-36005076307ffd72c00000000000019e1 S: disk/by-id/wwn-0x6005076307ffd72c00000000000019e1 Q: 14 E: DEVPATH=/devices/virtual/block/dm-2 E: DEVNAME=/dev/dm-2 E: DEVTYPE=disk E: DISKSEQ=14 E: MAJOR=253 E: MINOR=2 E: SUBSYSTEM=block E: USEC_INITIALIZED=40388916 E: DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 E: DM_UDEV_PRIMARY_SOURCE_FLAG=1 E: DM_UDEV_RULES_VSN=3 E: DM_NAME=mpathc E: DM_UUID=mpath-36005076307ffd72c00000000000019e1 E: MPATH_DEVICE_READY=1 E: DM_TYPE=scsi E: DM_WWN=0x6005076307ffd72c00000000000019e1 E: DM_SERIAL=36005076307ffd72c00000000000019e1 E: ID_PART_TABLE_UUID=3a9d5433-6c8b-4739-b725-65265833d7bd E: ID_PART_TABLE_TYPE=gpt E: NVME_HOST_IFACE=none E: DEVLINKS=/dev/disk/by-id/scsi-36005076307ffd72c00000000000019e1 /dev/mapper/mpathc /dev/disk/by-id/dm-name-mpathc /dev/disk/by-id/dm-uuid-mpath-36005076307ffd72c00000000000019e1 /dev/disk/by-id/wwn-0x6005076307ffd72c00000000000019e1 E: TAGS=:systemd: E: CURRENT_TAGS=:systemd: ---
Created attachment 2058933 [details] destination window of working F40 installer
Created attachment 2058934 [details] destination window of not working F41 installer
Created attachment 2062184 [details] Temporary workaround patch for kickstart installation issue
Created attachment 2062185 [details] Screenshot of incorrect installation destination UI columns
Created attachment 2062186 [details] Logs of working Fedora 41 interactive installation
Created attachment 2062187 [details] Logs of failed Fedora 41 kickstart installation
------- Comment From Peter.Oberparleiter.com 2024-12-12 10:00 EDT------- Additional analysis points to an erroneous upstream Anaconda commit - please route to Red Hat Anaconda developers. The following Anaconda commit [1] causes kickstart-based installations on multipath devices to fail with "No usable disks" error (seen on fedora 41 with Anaconda 41.35 + pykickstart v3.58 + Blivet 3.11.0): commit 248995131f955fd8b55519534e7b158678016e76 Author: Vojtech Trefny <vtrefny> Date: Wed Jun 26 14:58:09 2024 +0200 Adjust custom partitioning and storage spokes to the device ID API This should cover all places where name was used as device identifier in the storage, advanced storage and custom spokes. With this commit, Anaconda internally switches from using device names to device IDs for managing list of devices. This change does not cover device lists generated by kickstart directives such as 'ignoredisk' and 'part' though which are generated by method pyanaconda/core/storage.py:device_matches(). In a kickstart based installation, these device-name lists are matched against device-ID lists in method InstallerStorage:select_disks(), resulting in an empty list of usable devices and the subsequent installation error. See attachment 'logs-kickstart.tar.gz' for kickstart file and logs. An interactive installation via the Anaconda UI works, although the installation destination UI also shows incorrect row data (see attachment 'fedora41-install-dest-column-bug.png' for screenshot, 'logs-interactive.tar.gz' for logs). For testing purpose, the attached patch (attachment 'devid-workaround.patch') modifies the disk selection logic in InstallerStorage to also accept device names. Using this patch, an automated kickstart installation of Fedora 41 succeeds, although the correct change likely includes a more involved modification of the device_matches() method. [1] https://github.com/rhinstaller/anaconda/commit/248995131f955fd8b55519534e7b158678016e76
IBM, do I read it correctly that (possibly) all multipath devices are affected and this is not specific to zFCP?
------- Comment From Peter.Oberparleiter.com 2024-12-12 10:50 EDT------- (In reply to comment #15) > IBM, do I read it correctly that (possibly) all multipath devices are > affected and this is not specific to zFCP? While I have only tested with kickstart + zFCP devices + multipath, from my rudimentary understanding of the code, the underlying issue is not specific to zFCP devices, nor multipath. I would speculate that it could also be triggered by any other combination of kickstart device specification + device types where the device ID differs from the previously used device name. FYI - the device ID is used for the following device types: - LVM-<name> for LVM VGs and LVs - BTRFS-<uuid>-<name> for btrfs volumes and subvolumes - STRATIS-<name> for Stratis devices - LUKS-<name> for LUKS mapped devices - MDRAID-<name> for MD arrays - DM-<name> for generic DM devices See also the blivet commit that initially introduced the concept of device IDs [1]: 4c6743851dcf ("Add "device ID" that could be used as a unique device identifier"). [1] https://github.com/storaged-project/blivet/commit/4c6743851dcf1e399a785482ac78ffa3df40cc7f
Created attachment 2062299 [details] Temporary workaround patch for kickstart installation issue v2 ------- Comment on attachment From Peter.Oberparleiter.com 2024-12-13 09:00 EDT------- It turns out that our original workaround patch did not address all places where device ID and device names where incorrectly mixed in Anaconda. For our internal testing purpose, the only effective workaround is to completely disable the us of device-IDs for DM-devices in blivet (see attached patch).
> For our internal testing purpose, the only effective workaround is to completely disable the us of device-IDs for DM-devices in blivet (see attached patch). I went similar way, changing that only for multipath devices: https://github.com/storaged-project/blivet/pull/1327 I was able to test it locally in a VM with a fake multipath device but it would be nice to test the fix on real hardware. Updates image for Fedora 41: https://vtrefny.fedorapeople.org/img/rhbz2327619.img
------- Comment From Peter.Oberparleiter.com 2024-12-16 07:09 EDT------- (In reply to comment #18) > I went similar way, changing that only for multipath devices: > https://github.com/storaged-project/blivet/pull/1327 I was able to test it > locally in a VM with a fake multipath device but it would be nice to test > the fix on real hardware. Updates image for Fedora 41: > https://vtrefny.fedorapeople.org/img/rhbz2327619.img Installation was successful using the specified updates image with a kickstart mutipath installation. Thank you for the fix! Are you also planning to address the UI display issue shown in the attached screenshot? I'm assuming that the following hunk from Anaconda commit 248995131f95 caused the mix-up of storage columns in the UI: --- a/pyanaconda/ui/gui/spokes/advanced_storage.py +++ b/pyanaconda/ui/gui/spokes/advanced_storage.py @@ -56,7 +56,7 @@ DiskStoreRow = namedtuple("DiskStoreRow", [ "visible", "selected", "mutable", - "name", "type", "model", "capacity", + "name", "device_id", "type", "model", "capacity", "vendor", "interconnect", "serial", "wwid", "paths", "port", "target", "lun", "ccw", "wwpn", "namespace", "mode", When I modify the code to add the device_id at the very end of the list, data such as vendor or capacity is found in the correct column again, but the WWID remains empty. It appears that another commit [1] broke the WWID data source: d40cb44ee92a ("Add a NVMe-FC tab to the Advanced Storage screen") - wwid=device_data.attrs.get("path-id", "") or device_data.attrs.get("wwn", ""), + wwid=attrs.get("path-id", ""), If I change that back to: - wwid=attrs.get("path-id", "") or attrs.get("wwn", ""), the WWID column is shown correctly. [1] https://github.com/rhinstaller/anaconda/commit/d40cb44ee92a
Thank you for testing the fix. I opened a separate bug https://bugzilla.redhat.com/show_bug.cgi?id=2332568 to track the UI issue and Ill be working on fixing that as well.