Description of problem: When attempting to install an armhfp system, the boot hangs just before the installer with no indication why, with systemd debug enabled: zram-setup: Failed to load configuration: No such file or directory zram-setup: Failed to load configuration: No such file or directory zram-setup: Failed to load configuration: No such file or directory Full logs attached. Version-Release number of selected component (if applicable): systemd-246~rc2-2.fc33 How reproducible: Fails on hardware, working in qemu.
Created attachment 1702695 [details] zram systemd debug log
Created attachment 1702696 [details] Full boot log
cc'ing cmurf
>dev-zram0.mount: Failed to load configuration: No such file or directory I don't have a mount unit for this on any working setup. But maybe this is just a distraction. I'm not sure what it failed to load. Does /usr/lib/systemd/zram-generator.conf exist on this media? And is the single uncommented line merely [zram0] ?
The module loads super late... [ 174.399815] zram: Added device: zram0 succeeded. Missing from the journal is something like: [ 34.258490] fedora kernel: zram0: detected capacity change from 0 to 1515192320
(In reply to Chris Murphy from comment #4) > >dev-zram0.mount: Failed to load configuration: No such file or directory > > I don't have a mount unit for this on any working setup. But maybe this is > just a distraction. I'm not sure what it failed to load. > > Does /usr/lib/systemd/zram-generator.conf exist on this media? And is the > single uncommented line merely [zram0] ? This was a pxe installation attempt, the squashfs does include zram-generator.conf with '[zram0]' uncommented.
zram-setup@.service is not part of the zram-generator package. It seems something is pulling in this unit. I'm not sure what is the source is.
My proposal: 1. add Obsoletes: udisk2-zram in zram-generator-defaults 2. drop udisk2-zram
Sorry I missed this. I looked again at comps and kickstarts, and can't find any reference to udisks2-zram. I'm not sure why it's present in this case. Is there something in the 'arm-tools' group that might be pulling it in? On the one hand, since it's not installed by default anywhere else, I'm not sure how useful it is. On the other hand, I could see useful integration on server+cloud+desktop via Cockpit and Disk Utility, to (a) change the zram-based swap configuration (b) create generic non-swap ram disks. Perhaps in the short term all udisks2 needs is to not conflict with zram-generator by checking if there's already zram-based swap, and if so, then don't try to create another one. I think that might be all that's going on here, is zram-generator simply gets there first, and then anything else that assumes it can create a zram0 swap device fails. Noisy but non-fatal.
I wonder why this even fails the boot process... We can't drop udisks2-zram as it provides management API that others may use. Short-term we can add Conflicts: zram-generator into udisks2-zram and long-term we can port udisks2-zram to use zram-generator as a backend, keeping the public API.
(In reply to Tomáš Bžatek from comment #10) > I wonder why this even fails the boot process... I'm not sure if this is actually the cause of the problems. Those messages zram-setup: Failed to load configuration: No such file or directory would only be fatal if zram-setup was a dependency of something important. Is the unit configured to be Required or PartOf something? > We can't drop udisks2-zram as it provides management API that others may > use. Short-term we can add Conflicts: zram-generator into udisks2-zram and If anything, udisks2-zram could conflict with zram-generator-defaults. But conflicts can be problematic for users if those two units are both pulled in as dependencies. So I think it'd be better to make the two services aware of each other. Is there any simple check I could add in zram-generator to disable itself if udisk2-zram is going to be active? > long-term we can port udisks2-zram to use zram-generator as a backend, > keeping the public API. I'm not sure if this is worth the trouble. The actual process of creating a zram device is so simple. Instead, if both implementations have configurations, one of them should disable itself with a nice message.
I'm not sure what udisks2-zram even does. I thought arm installs were originally depending on 'zram' package to do this, not 'udisks2-zram'. zram-generator is likely to beat all other implementations, it just runs that much earlier in the startup process.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #11) > I'm not sure if this is actually the cause of the problems. Those messages > zram-setup: Failed to load configuration: No such file or > directory > would only be fatal if zram-setup was a dependency of > something important. > Is the unit configured to be Required or PartOf something? Nothing I'm aware of. https://raw.githubusercontent.com/storaged-project/udisks/master/modules/zram/data/zram-setup%40.service Apparently Anaconda has been shipping their own zram.service: https://github.com/rhinstaller/anaconda/pull/2727 (thanks to Vojtech Trefny for noticing that). > If anything, udisks2-zram could conflict with zram-generator-defaults. But > conflicts can be problematic for users if those two units are both pulled in > as > dependencies. So I think it'd be better to make the two services aware of > each other. > Is there any simple check I could add in zram-generator to disable itself if > udisk2-zram is going to be active? The udisks2-zram module creates a file "$(libdir)/zram.conf.d/zramN" for each configured zram device where N is an index. See related EnvironmentFile key in the mentioned service file. .... aaand looks like we found a bug. The udisks2-zram code is creating config files in /usr/lib64/zram.conf.d on x86_64 yet the service file hardcodes /usr/local/lib/zram.conf.d. Clearly nobody is using that code in reality. Besides that it creates /usr/lib/modules-load.d/zram.conf and /usr/lib/modprobe.d/zram.conf files (with no indication of being created by UDisks2). > I'm not sure if this is worth the trouble. The actual process of creating a > zram > device is so simple. Instead, if both implementations have configurations, > one of them should disable itself with a nice message. Right. There should at least be some degree of awareness since the udisks2-zram module exposes an extra D-Bus interface for management over any existing /dev/zram* block device even when created by a different framework. Filed upstream: https://github.com/storaged-project/udisks/issues/791
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle. Changing version to 33.
This continues to break installs on arm and aarch64 SBC's. Proposing as a blocker for Fedora 33 Beta, criteria: "The installer must run when launched normally from the release-blocking images."
Created attachment 1711215 [details] Full boot logs on aarch64 (Fedora-Rawhide-20200811.n.0)
I've pushed udisks2-2.9.1-1.fc33 (bodhi FEDORA-2020-e143a283fb) that brings couple of changes outlined above but probably nothing major that would help to fix this issue. The only notable change is a renamed udisks2-zram-setup@.service unit to help distinguish service origin. I haven't added any Conflicts: for the moment. The rest of the udisks2-zram issues are tracked upstream in the above mentioned ticket. I've check the comps and the Cockpit spec file, no traces of udisks2-zram Requires:. As far as I know no other package should explicitly require udisks2-zram. Kicking that package from installer image and default install should fix this bugzilla, however the reason it's pulled in needs to be found first. Please reassign as appropriate, this doesn't seem to be a udisks2 bug.
I don't see udisks2-zram package being brought in on either aarch64 or armv7hl image creation. These are from 20200816 https://kojipkgs.fedoraproject.org//work/tasks/9245/49389245/oz-aarch64.log https://kojipkgs.fedoraproject.org//work/tasks/9171/49389171/mock_output.log I also only find expected items in the root and initramfs on both Fedora-Workstation-33-20200816.n.0.aarch64.raw.xz Fedora-Workstation-armhfp-33-20200816.n.0-sda.raw.xz
So does that mean the original issue is fixed? I don't see anything fishy in the logs either.
I'm not sure, I don't have hardware. In the description it says the problem happens on hardware, not in qemu. That is also strange.
Testing the compose from Fedora-33-20200816.n.0, I am no longer hitting this on arm or aarch64, installs are completed as expected.
Based on comment 21, I'm unproposing this as a Beta blocker, the issue has disappeared. Paul, if it happens again, please nominate this bug again, thanks!
> I'm not sure, I don't have hardware. In the description it says the problem happens on hardware, not in qemu. That is also strange. I think this must be a false conclusion: there is nothing in either the udev rules or the unit installed by udisks2-zram that is hardware specific. Most likely there is some race condition and we just don't happen to see it on qemu. I installed udisks2-zram on my rpi3, and what happens now is that both zram-generator and udisk2-zram try to configure the same zram0 device: Aug 21 01:30:49 rpi3 sudo[2202]: zbyszek : TTY=pts/0 ; PWD=/home/zbyszek ; USER=root ; COMMAND=/usr/sbin/modprobe -v zram Aug 21 01:30:49 rpi3 kernel: zram: Added device: zram0 Aug 21 01:30:49 rpi3 systemd[1]: Starting Create swap on /dev/zram0... <-------------- this is zram-generator Aug 21 01:30:49 rpi3 systemd[1]: Starting UDisks2 managed ZRAM device zram0 setup... <-------------- this is udisk2-zram Aug 21 01:30:49 rpi3 kernel: zram0: detected capacity change from 0 to 489684992 Aug 21 01:30:49 rpi3 systemd[1]: udisks2-zram-setup: Succeeded. Aug 21 01:30:49 rpi3 systemd[1]: Finished UDisks2 managed ZRAM device zram0 setup. Aug 21 01:30:49 rpi3 zram-generator[2219]: Setting up swapspace version 1, size = 467 MiB (489680896 bytes) Aug 21 01:30:49 rpi3 zram-generator[2219]: no label, UUID=1545938f-92d5-477e-a02d-0e8d5856a56e Aug 21 01:30:49 rpi3 systemd[1]: Finished Create swap on /dev/zram0. Aug 21 01:30:50 rpi3 systemd[1]: Activating swap Compressed swap on /dev/zram0... Aug 21 01:30:50 rpi3 kernel: Adding 478204k swap on /dev/zram0. Priority:100 extents:1 across:478204k SSFS Aug 21 01:30:50 rpi3 systemd[1]: Activated swap Compressed swap on /dev/zram0. The order of messages should not be trusted. They two things are executing in parallel and happen "at the same time", so we cannot say which one succeeded. udisks2-zram-setup@.service ignores all errors, while swap-create@.service doesn't, so I think the latter must be winning the race here, but it's possible that they are both partially active. The kernel allows settings to be set multiple times before the device is activated. udisks2-zram-setup@.service will always execute because of the udev rule in /usr/lib/udev/rules.d/90-udisks2-zram.rules. I don't think we want to have both active, this can only lead to problems. Should I add a rule to zram-generator to disable itself if /usr/lib/udev/rules.d/90-udisks2-zram.rules exists? This is ugly, but probably better than status quo.
>Should I add a rule to zram-generator to disable itself if /usr/lib/udev/rules.d/90-udisks2-zram.rules exists? This is ugly, but probably better than status quo. I have no strong opinion. The conflict seems unlikely. I don't see udisks2-zram on either armv7hl or aarch64 media, and if it does pop up somewhere on media, that's a bug that definitely needs fixed. But to avoid a problem if someone intentionally installs udisks2-zram? Possibly.
This message is a reminder that Fedora 33 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '33'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 33 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.