Created attachment 1699845 [details] rdsosreport I have a VM with a multipath SCSI device (a raw image referenced by 2 SCSI disks, with the same serial number), libvirt allowed that without complains and I was able to successfully install from Fedora-Rawhide-20200703.n.0 compose onto the detected mpatha device. But after rebooting to the newly installed system I get to the emergency shell, because the rootfs device isn't found. steps I did to continue with the boot: # multipathd -k - add path sda failed - add path sda ok - add path sdb failed - add path sdb ok - show maps mpatha ... # activated the LVs on /dev/mapper/mpatha3 # exit (from shell) then it finished the boot Version-Release number of selected component (if applicable): device-mapper-multipath-0.8.4-1.fc33.ppc64le Additional info: I'm using Fedora 30 for ppc64le as my workstation, but I believe there is nothing ppc64le specific, so it should be reproducable on x86 as well.
Created attachment 1699871 [details] output of "journalctl -b" I believe it shows my manual attempts to activate the multipath device.
Thanks for catching this! There's two problems with /etc/multipath.conf here (assuming your non-booting system looks like mine). One is that there are two find_multipaths settings. The find_multipaths smart line should definitely not be there. I believe that anaconda was using that when it initially installed the system, but it should not set that in the multipath.conf file that it writes out. The other problem is that multipath has started including blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } in the default multipath.conf. This ends up blacklisting devices that don't have these udev database variables. This was supposed to go along with the inclusion of /lib/udev/rules.d/61-scsi-sg3_id.rules. This rules file now exists in RHEL8, and sets SCSI_IDENT_*. Unfortunately, it doesn't exist in fedora, currently. To get your system working. Remove the above lines from /etc/multipath.conf and remake your initramfs. I will back out the default inclusion of that blacklist_exceptions line from the default multipath.conf file, and update the multipath packages. Then I'll ask around to see if/when sg_utils plans on including the additional rules files in fedora, and verify that anaconda is what's adding find_multipaths smart to the config.
Thanks, Ben, for looking. Adding Tomas to CC as the sg3_utils maintainer. 2 observations: - /lib/udev/rules.d/61-scsi-sg3_id.rules exists in Fedora, it is part of sg3_utils rpm, but it isn't installed by default (https://src.fedoraproject.org/rpms/sg3_utils/blob/master/f/sg3_utils.spec#_94), only sg3_utils-libs is installed. We may need to add an explicit dependency to device-mapper-multipath. Dunno if there is a better solution, maybe some of the rules should be part of the -libs subpackage (if they don't require the utilities from the main sg3_utils package). - still trying to find what adds "find_multipaths smart", seems the string "find_multipaths" isn't used anywhere in the installer (anaconda, blivet), and only used as --find_multipaths parameter to "mpathconf" in libblockdev (./src/plugins/mpath.c:532). But will continue to look further.
"find_multipaths smart" could be caused by this change: https://github.com/weldr/lorax/pull/835 And anaconda code copies /etc/multipath.conf to the installed system: https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/storage/devicetree/fsset.py#L656
(In reply to Jan Stodola from comment #4) > "find_multipaths smart" could be caused by this change: > https://github.com/weldr/lorax/pull/835 yes, it's it. defaults { find_multipaths smart user_friendly_names yes } blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } is /etc/multipath.conf in the installer image. It then combines with mpathconf --find_multipaths y --user_friendly_names y --with_multipathd y (from https://github.com/storaged-project/libblockdev/blob/master/src/plugins/mpath.c#L531) that the mpath plugin of libblockdev runs and the result is defaults { find_multipaths yes find_multipaths smart user_friendly_names yes } blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } blacklist { } on the installed system > And anaconda code copies /etc/multipath.conf to the installed system: > https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/ > storage/devicetree/fsset.py#L656 ack
(In reply to Jan Stodola from comment #4) > "find_multipaths smart" could be caused by this change: > https://github.com/weldr/lorax/pull/835 > > And anaconda code copies /etc/multipath.conf to the installed system: > https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/ > storage/devicetree/fsset.py#L656 Cc: dlehman
(In reply to Dan Horák from comment #3) > Thanks, Ben, for looking. Adding Tomas to CC as the sg3_utils maintainer. I'm not really sg3_utils maintainer! For the record, > > 2 observations: > - /lib/udev/rules.d/61-scsi-sg3_id.rules exists in Fedora, it is part of > sg3_utils rpm, but it isn't installed by default > (https://src.fedoraproject.org/rpms/sg3_utils/blob/master/f/sg3_utils. > spec#_94), only sg3_utils-libs is installed. We may need to add an explicit > dependency to device-mapper-multipath. Dunno if there is a better solution, > maybe some of the rules should be part of the -libs subpackage (if they > don't require the utilities from the main sg3_utils package). > > - still trying to find what adds "find_multipaths smart", seems the string > "find_multipaths" isn't used anywhere in the installer (anaconda, blivet), > and only used as --find_multipaths parameter to "mpathconf" in libblockdev > (./src/plugins/mpath.c:532). But will continue to look further.
For the record, the sg3_utils udev rules have been added only recently, in sg3_utils-1.44-1.fc32. I'm okay with adding a sg3-utils dependency in device-mapper-multipath, still would be nice to move some of the rules to 60-persistent-storage.rules directly.
Actually, since dracut never pulls in the new sg3_utils rules, adding a dependency won't currently make this work any better. I think the issue is simply that I was too hasty will adding the udev property requirements to the default config. As long as find_multipaths is turned on in the default config, there's no real benefit, since multipath already won't set up on a device unless there are multiple paths to it. The property requirements should mean that multipathd gives up on considering the path sooner, but in actual fact, they don't.
mpathconf in this package won't set blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } by default. If you want it (which is only really important if you turn off find_multipaths), you can run mpathconf with "--property-blacklist y" This doesn't fix the anaconda issue. Multipath should still work with the duplicate find_multipaths line. It will use find_multipaths smart which means that on every boot, if there are non-multipathed devices that are needed to boot, there will be a small delay while multipath waits to see if a second path appears.
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle. Changing version to 33.
FTR - seems also real HW is also affected by this bug, seen on a Power8 system with an IPR disk controller
This message is a reminder that Fedora 33 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '33'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 33 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
I'm reopening this bug, since the problem still exists on F39/Rawhide, tested on an x86_64 VM. Removing: blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } from /etc/multipath.conf and re-creating initramfs can be used as a workaround. Tested with device-mapper-multipath-0.9.5-1.fc39.x86_64.
(In reply to Jan Stodola from comment #17) > I'm reopening this bug, since the problem still exists on F39/Rawhide, > tested on an x86_64 VM. > > Removing: > > blacklist_exceptions { > property "(SCSI_IDENT_|ID_WWN)" > } > > from /etc/multipath.conf and re-creating initramfs can be used as a > workaround. > > Tested with device-mapper-multipath-0.9.5-1.fc39.x86_64. mpathconf no longer includes this in the default config it generates. However since the multipath config file is not installed by the rpm, updating the package doesn't change it. So if your multipath.conf file was created with an earlier version that did set the property blacklist exceptions, then that config section will still be there when you upgrade the package. To verify this, you can remove /etc/multipath.conf (or just rename it to /etc/multipath.conf.bak, if you need to save your existing config) and use mpathconf to generate a default config by running # mpathconf --enable This should generate an /etc/multipath.conf file without that section. Is this happening on a newly installed rawhide system? If so, then in seems that anaconda must be specifically adding that line to the config it generates, and the bug needs to move over to the installer.
Thanks for the hint, Ben. The following block in /etc/multipath.conf on the installation media: blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } is written by lorax when creating the installation media: https://github.com/weldr/lorax/blob/fa70fea84382049f628c1af1318d897873988547/share/templates.d/99-generic/runtime-postinstall.tmpl#L106 This block was added because of bug 1685363. 2) After the installation is complete, anaconda copies /etc/multipath.conf to the installed system: https://github.com/rhinstaller/anaconda/blob/84229e7be394dff4cdc3c2e6352633058c03fdee/pyanaconda/modules/storage/devicetree/fsset.py#L665 With lorax's commit https://github.com/weldr/lorax/pull/632/commits/b1407900ac4b19d58951e1283070aadaa9b4729e reverted, the installation starts fine and the installed system boots successfully, tested both multipath and single-path VMs. Ben, can you please confirm that not writing that "blacklist_exceptions" block by lorax is a safe solution?
(In reply to Jan Stodola from comment #19) > Thanks for the hint, Ben. > Ben, can you please confirm that not writing that "blacklist_exceptions" > block by lorax is a safe solution? Yes. As long as multipath.conf uses either: find_multipaths yes or find_multipaths smart everything should be fine. That priority blacklist_exceptions block was supposed to weed out devices that couldn't be multipathed. The two find_multipaths settings above will not multipath devices if multiple paths do not exist. If multiple paths exist, then obviously the devices are multipathable. I know that while anaconda is installing the system, it makes use of "find_multipaths smart". The way that the smart method works, systemd will wait briefly for another path to show up before using a potentially multipathable device directly. Removing the blacklist_exceptions section from the multipath.conf file that anaconda uses while installing could cause installations to take a few seconds longer, assuming that there is a non-multipathable device without a udev attribute that matches the exception criteria. However, without /lib/udev/rules.d/61-scsi-sg3_id.rules present, there is no guarantee that even multipathable devices will have a udev attribute that matches the exception criteria, so the installer might ignore a multipathable device, which is worse than taking a little bit longer. So, for the multipath.conf that is added to the installed system, the blacklist_exceptions block should definitely be removed (anaconda better be using "find_multipaths yes" there). For the multipath.conf that is used during installation, if /lib/udev/rules.d/61-scsi-sg3_id.rules exists in the installer, the blacklist_exceptions block can be kept. It may make installations go slightly faster. If that udev rule doesn't exist on the installer when it is detecting and making multipath devices, then the blacklist_exceptions block should definitely be removed from the installer's version of multipath.conf.
(In reply to Ben Marzinski from comment #20) > (In reply to Jan Stodola from comment #19) > > Thanks for the hint, Ben. > > Ben, can you please confirm that not writing that "blacklist_exceptions" > > block by lorax is a safe solution? > > Yes. As long as multipath.conf uses either: > find_multipaths yes > or > find_multipaths smart Yes, it does contain "find_multipaths smart" > > everything should be fine. That priority blacklist_exceptions block was > supposed to weed out devices that couldn't be multipathed. The two > find_multipaths settings above will not multipath devices if multiple paths > do not exist. If multiple paths exist, then obviously the devices are > multipathable. > > I know that while anaconda is installing the system, it makes use of > "find_multipaths smart". The way that the smart method works, systemd will > wait briefly for another path to show up before using a potentially > multipathable device directly. Removing the blacklist_exceptions section > from the multipath.conf file that anaconda uses while installing could cause > installations to take a few seconds longer, assuming that there is > a non-multipathable device without a udev attribute that matches the > exception criteria. However, without /lib/udev/rules.d/61-scsi-sg3_id.rules > present, there is no guarantee that even multipathable devices will have a > udev attribute that matches the exception criteria, so the installer might > ignore a multipathable device, which is worse than taking a little bit > longer. /lib/udev/rules.d/61-scsi-sg3_id.rules doesn't exist in the installation initramdisk, but it is present in stage2 of the installation process. And /lib/udev/rules.d/61-scsi-sg3_id.rules is not included in the initramdisk of the installed system either - actually the sg3_utils package is not installed at all as part of a minimal package set installation. Tested on a VM with a multipath disk. > > So, for the multipath.conf that is added to the installed system, the > blacklist_exceptions block should definitely be removed (anaconda better be > using "find_multipaths yes" there). Yes, multipath.conf on the installed system use "find_multipath yes" > > For the multipath.conf that is used during installation, if > /lib/udev/rules.d/61-scsi-sg3_id.rules exists in the installer, the > blacklist_exceptions block can be kept. It may make installations go > slightly faster. If that udev rule doesn't exist on the installer when it is > detecting and making multipath devices, then the blacklist_exceptions block > should definitely be removed from the installer's version of multipath.conf. I've submitted a lorax PR to update multipath.conf in the installer: https://github.com/weldr/lorax/pull/1329
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle. Changing version to 39.