Bug 2165433

Summary: installer fails to boot in text mode or rescue mode with systemd 253
Product: [Fedora] Fedora Reporter: lnie <lnie>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 38CC: anaconda-maint-list, awilliam, fedoraproject, filbranden, jkonecny, lnykryn, msekleta, robatino, rvykydal, ryncsn, systemd-maint, vponcova, vslavik, w, yuwatana, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: openqa
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-13 16:52:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2083910    
Attachments:
Description Flags
screenshot none

Description lnie 2023-01-30 02:40:33 UTC
Created attachment 1941036 [details]
screenshot

Description of problem:

Boot Fedora-Server-dvd-x86_64-Rawhide-20230126.n.0.iso/Fedora-Server-netinst-x86_64-Rawhide-20230126.n.0.iso with inst.text command line added,system hangs at Finished plymouth-quit.service

Version-Release number of selected component (if applicable):
anaconda-38.18-1.fc38.x86_64.rpm  

How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Fedora Blocker Bugs Application 2023-01-30 05:12:17 UTC
Proposed as a Blocker for 38-beta by Fedora user lnie using the blocker tracking app because:

 affects:
When using a dedicated installer image, the installer must be able to complete an installation using the text, graphical and VNC installation interfaces.

Comment 2 Adam Williamson 2023-01-30 18:25:28 UTC
This has been failing in openQA since Fedora-Rawhide-20230125.n.0 (I just didn't get around to reporting it yet, last week was kinda hectic). systemd 253~rc1 got tagged on 0124, so I'm betting this is a systemd 253 bug. Re-assigning.

Comment 3 Adam Williamson 2023-01-30 18:26:19 UTC
Rescue mode boot fails the same way.

Comment 4 Adam Williamson 2023-01-30 19:01:09 UTC
I can reproduce this locally. The journal does show anaconda running, but it's not actually visible on any VT, it should be on VT1. VT2, VT3, VT5 and VT6 show root consoles (this is normal, except for VT2, which should show a custom tmux setup). VT4 shows the system logs (this is normal).

To reproduce: download https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230130.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230130.n.0.iso , boot it with `inst.text`.

I do see a message:

/usr/lib/systemd/system-generators/anaconda-generator failed with exit status 1.

which may be related. It's very likely why VT2 doesn't have the custom tmux session that it ought to have. Running that script directly after system boot does not seem to fail, it exits 0. Before I run that script manually, /usr/lib/systemd/system/anaconda.target.wants does not exist. After running that script, it exists and contains several services.

Comment 5 Zbigniew Jędrzejewski-Szmek 2023-02-01 14:14:01 UTC
Oh, yes, this cannot work. As documented in systemd.generator(7):
> Each generator is called with three directory paths that are to be used for generator output.
Generators must not write outside of those paths.

I see that anaconda-generator does
> systemd_dir=/lib/systemd/system
> target_dir=$systemd_dir/anaconda.target.wants
> mkdir -p $target_dir
...
> ln -sf $systemd_dir/anaconda-nm-config.service $target_dir/anaconda-nm-config.service
> ln -sf $systemd_dir/anaconda-nm-disable-autocons.service $target_dir/anaconda-nm-disable-autocons.service
> ln -sf $systemd_dir/anaconda-pre.service $target_dir/anaconda-pre.service

This is broken on many levels:
1. Generators are called early, potentially before / has been remounted rw.
2. Generators are not supposed to create permanent configuration, their purpose is to translate foreign config into units on the fly.
3. Generator output is tied to the lifetime of systemd daemon-reload, i.e. potentially at any time a reload may happen, and if the generator is replaced or masked, its effects are supposed to vanish.

In systemd-253-rc1 we added a "sandbox" that provides a writable /tmp for
generators (so that here-docs work in bash even if /tmp is not available yet),
and makes most of the file system read-only. This obviously interferes with
the stuff that anaconda-generator was doing.

Please make the generator write in the expected locations.

Comment 6 Adam Williamson 2023-02-01 17:51:26 UTC
Zbigniew, does https://github.com/rhinstaller/anaconda/pull/4534 look good to you? It seems to work for me.

Comment 7 Jiri Konecny 2023-02-02 21:22:54 UTC
Setting needinfo for comment 6.

Comment 8 Adam Williamson 2023-02-03 19:28:54 UTC
The PR is merged now, setting MODIFIED.

Comment 9 Zbigniew Jędrzejewski-Szmek 2023-02-06 11:48:33 UTC
The PR LGTM.

Comment 10 Ben Cotton 2023-02-07 15:13:51 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Comment 11 Radek Vykydal 2023-02-13 09:39:36 UTC
For the reference, the issue made "dracut-visible-warnings" kickstart test fail (https://github.com/rhinstaller/kickstart-tests/issues/875).
It is passing now.

Comment 12 Adam Williamson 2023-02-13 16:52:04 UTC
oh, yeah, this is confirmed fixed, openQA tests are passing now. Sorry, I forgot to update the bug.

Comment 13 Radek Vykydal 2023-02-14 07:16:39 UTC
No problem @awilliam and thank you for the fix!