Bug 2165433 - installer fails to boot in text mode or rescue mode with systemd 253
Summary: installer fails to boot in text mode or rescue mode with systemd 253
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 38
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Anaconda Maintenance Team
QA Contact: Fedora Extras Quality Assurance
Whiteboard: openqa
Depends On:
Blocks: F38BetaBlocker
TreeView+ depends on / blocked
Reported: 2023-01-30 02:40 UTC by lnie
Modified: 2023-02-14 07:29 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2023-02-13 16:52:04 UTC
Type: Bug

Attachments (Terms of Use)
screenshot (74.43 KB, image/png)
2023-01-30 02:40 UTC, lnie
no flags Details

System ID Private Priority Status Summary Last Updated
Github rhinstaller anaconda pull 4534 0 None open Fix the systemd generator for systemd 253 (#2165433) 2023-02-01 17:51:26 UTC

Description lnie 2023-01-30 02:40:33 UTC
Created attachment 1941036 [details]

Description of problem:

Boot Fedora-Server-dvd-x86_64-Rawhide-20230126.n.0.iso/Fedora-Server-netinst-x86_64-Rawhide-20230126.n.0.iso with inst.text command line added,system hangs at Finished plymouth-quit.service

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 1 Fedora Blocker Bugs Application 2023-01-30 05:12:17 UTC
Proposed as a Blocker for 38-beta by Fedora user lnie using the blocker tracking app because:

When using a dedicated installer image, the installer must be able to complete an installation using the text, graphical and VNC installation interfaces.

Comment 2 Adam Williamson 2023-01-30 18:25:28 UTC
This has been failing in openQA since Fedora-Rawhide-20230125.n.0 (I just didn't get around to reporting it yet, last week was kinda hectic). systemd 253~rc1 got tagged on 0124, so I'm betting this is a systemd 253 bug. Re-assigning.

Comment 3 Adam Williamson 2023-01-30 18:26:19 UTC
Rescue mode boot fails the same way.

Comment 4 Adam Williamson 2023-01-30 19:01:09 UTC
I can reproduce this locally. The journal does show anaconda running, but it's not actually visible on any VT, it should be on VT1. VT2, VT3, VT5 and VT6 show root consoles (this is normal, except for VT2, which should show a custom tmux setup). VT4 shows the system logs (this is normal).

To reproduce: download https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230130.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230130.n.0.iso , boot it with `inst.text`.

I do see a message:

/usr/lib/systemd/system-generators/anaconda-generator failed with exit status 1.

which may be related. It's very likely why VT2 doesn't have the custom tmux session that it ought to have. Running that script directly after system boot does not seem to fail, it exits 0. Before I run that script manually, /usr/lib/systemd/system/anaconda.target.wants does not exist. After running that script, it exists and contains several services.

Comment 5 Zbigniew Jędrzejewski-Szmek 2023-02-01 14:14:01 UTC
Oh, yes, this cannot work. As documented in systemd.generator(7):
> Each generator is called with three directory paths that are to be used for generator output.
Generators must not write outside of those paths.

I see that anaconda-generator does
> systemd_dir=/lib/systemd/system
> target_dir=$systemd_dir/anaconda.target.wants
> mkdir -p $target_dir
> ln -sf $systemd_dir/anaconda-nm-config.service $target_dir/anaconda-nm-config.service
> ln -sf $systemd_dir/anaconda-nm-disable-autocons.service $target_dir/anaconda-nm-disable-autocons.service
> ln -sf $systemd_dir/anaconda-pre.service $target_dir/anaconda-pre.service

This is broken on many levels:
1. Generators are called early, potentially before / has been remounted rw.
2. Generators are not supposed to create permanent configuration, their purpose is to translate foreign config into units on the fly.
3. Generator output is tied to the lifetime of systemd daemon-reload, i.e. potentially at any time a reload may happen, and if the generator is replaced or masked, its effects are supposed to vanish.

In systemd-253-rc1 we added a "sandbox" that provides a writable /tmp for
generators (so that here-docs work in bash even if /tmp is not available yet),
and makes most of the file system read-only. This obviously interferes with
the stuff that anaconda-generator was doing.

Please make the generator write in the expected locations.

Comment 6 Adam Williamson 2023-02-01 17:51:26 UTC
Zbigniew, does https://github.com/rhinstaller/anaconda/pull/4534 look good to you? It seems to work for me.

Comment 7 Jiri Konecny 2023-02-02 21:22:54 UTC
Setting needinfo for comment 6.

Comment 8 Adam Williamson 2023-02-03 19:28:54 UTC
The PR is merged now, setting MODIFIED.

Comment 9 Zbigniew Jędrzejewski-Szmek 2023-02-06 11:48:33 UTC

Comment 10 Ben Cotton 2023-02-07 15:13:51 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Comment 11 Radek Vykydal 2023-02-13 09:39:36 UTC
For the reference, the issue made "dracut-visible-warnings" kickstart test fail (https://github.com/rhinstaller/kickstart-tests/issues/875).
It is passing now.

Comment 12 Adam Williamson 2023-02-13 16:52:04 UTC
oh, yeah, this is confirmed fixed, openQA tests are passing now. Sorry, I forgot to update the bug.

Comment 13 Radek Vykydal 2023-02-14 07:16:39 UTC
No problem @awilliam and thank you for the fix!

Note You need to log in before you can comment on or make changes to this bug.