Fedora-Minimal-38-20230222.n.0.aarch64.raw.xz initial-setup-0.3.97-1.fc38.aarch64 Raspberry Pi 4 arm-image-installer Headless, no keyboard attacched, serial communication. Initial setup start regularly. At least I'm able to configure the network. Also ntp server and timezone get configured. If I try to add a user, it is not created and initial-setup doesn't throw any error. Invoking /usr/libexec/initial-setup/run-initial-setup on a running system, looking at journalctl I can spot this: Mar 03 11:26:41 org.fedoraproject.Anaconda.Modules.Localization[1737]: ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-KeyboardInstallationTask-1 has failed: Traceback (most recent call last): And other Python stuff (attacched), but I don't spot anything related to user creation.
Proposed as a Blocker for 38-final by Fedora user alciregi using the blocker tracking app because: "If an initial setup utility is run or intended to be run after the first boot of the installed system, then it must start successfully and each page or panel of the initial setup utility should withstand a basic functionality test."
Created attachment 1947722 [details] log snippet when I hit "c" to end initial-setup
In addition, such errors appear also without creating a user. And last, initial-setup start at each reboot (probably obviously, because some completed bit are not set).
Ugh, yes - openQA seems to be running into the same thing, but I forgot to check the aarch64 results for a while so I didn't catch it :( sorry. See https://openqa.fedoraproject.org/tests/1799209 - logging in as the user created during initial-setup fails so the test fails. This seems to have broken between Fedora-Rawhide-20230202.n.0 and Fedora-Rawhide-20230202.n.1 , before f38 branched. anaconda and initial-setup did not change in that compose...python-blivet did, and rng-tools did, and systemd went from 253~rc1 to 253~rc2. Nothing else that changed looks obviously like it could be related. Promoting this to a Beta blocker per basic criterion "A working mechanism to create a user account must be clearly presented during installation and/or first boot of the installed system." - that criterion is violated for non-GNOME aarch64 disk images, which rely on initial-setup to create a user.
Huh, a funny thing: this actually seems to affect only the Minimal image. The same thing on the Server image works OK. So, maybe a missing dependency or something? Some package that's necessary for this to work is not being pulled into minimal installs?
Also proposing as an FE, since Ben pointed out minimal isn't actually on the blocking list ATM.
Discussed in ticket: https://pagure.io/fedora-qa/blocker-review/issue/1070 The decision to classify this bug as an RejectedBlocker was made: "Minimal image is not a blocking deliverable."
Something I notice here: anaconda uses `chpasswd` to set root and user passwords. That's part of shadow-utils. Neither anaconda nor initial-setup requires shadow-utils. I wonder if that's the issue here? A lot of things do depend on it so it may have been pulled in casually before, but some change made that stop happening on the ARM minimal image specifically?
Discussed during the 2023-03-06 blocker review meeting: [0] The decision to classify this bug as an "AcceptedFreezeException (Beta)" was made as this makes the image unusable for interactive installs, it's certainly serious enough to grant a freeze exception. [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2023-03-06/f38-blocker-review.2023-03-06-17.00.txt
No, that's not it. But I think I see what it is! I reproduced this in a VM (on my x86_64 laptop. Yes, it's slow) and mounted the root partition from the disk image. The journal shows this: Mar 06 14:55:41 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: INFO:anaconda.threading:Thread Done: AnaTaskThread-LanguageInstallationTask-1 (281473178485152) Mar 06 14:55:41 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: INFO:anaconda.threading:Running Thread: AnaTaskThread-KeyboardInstallationTask-1 (281473178485152) Mar 06 14:55:41 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: INFO:anaconda.modules.common.task.task:Configure keyboard Mar 06 14:55:41 fedora audit: BPF prog-id=66 op=LOAD Mar 06 14:55:41 fedora audit: BPF prog-id=67 op=LOAD Mar 06 14:55:41 fedora audit: BPF prog-id=68 op=LOAD Mar 06 14:55:41 fedora systemd[1]: Starting systemd-localed.service - Locale Service... Mar 06 14:55:42 fedora systemd[1]: Started systemd-localed.service - Locale Service. Mar 06 14:55:42 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-localed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Mar 06 14:55:42 fedora systemd-localed[944]: Cannot compile XKB keymap for new x11 keyboard layout ('' / 'us' / '' / ''): Operation not supported Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: INFO:anaconda.threading:Thread Failed: AnaTaskThread-KeyboardInstallationTask-1 (281473178485152) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-KeyboardInstallationTask-1 has failed: Traceback (most recent call last): Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/threading.py", line 275, in run Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: threading.Thread.run(self) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/threading.py", line 975, in run Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._target(*self._args, **self._kwargs) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/common/task/task.py", line 96, in _thread_run_callback Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._task_run_callback() Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/common/task/task.py", line 109, in _task_run_callback Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._set_result(self.run()) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/localization/installation.py", line 152, in run Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: write_x_configuration( Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/localization/installation.py", line 197, in write_x_configuration Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: localed_wrapper.set_layouts(x_layouts, switch_options) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/localization/localed.py", line 183, in set_layouts Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._localed_proxy.SetX11Keyboard( Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 450, in _call_method Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: return self._get_method_reply( Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^^^^^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 483, in _get_method_reply Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: return self._handle_method_error(error) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 509, in _handle_method_error Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: raise exception from None Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: dasbus.error.DBusError: Local keyboard configuration not supported on this system. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: INFO:anaconda.threading:Thread Done: AnaTaskThread-KeyboardInstallationTask-1 (281473178485152) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: WARNING:dasbus.server.handler:The call org.fedoraproject.Anaconda.Task.Finish has failed with an exception: Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: Traceback (most recent call last): Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/server/handler.py", line 455, in _method_callback Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: result = self._handle_call( Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/server/handler.py", line 265, in _handle_call Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: return handler(*parameters, **additional_args) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/common/task/task_interface.py", line 114, in Finish Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self.implementation.finish() Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/common/task/task.py", line 175, in finish Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: threadMgr.raise_if_error(self._thread_name) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/threading.py", line 166, in raise_if_error Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: raise exc_info[1] Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/threading.py", line 275, in run Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: threading.Thread.run(self) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/threading.py", line 975, in run Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._target(*self._args, **self._kwargs) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/common/task/task.py", line 96, in _thread_run_callback Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._task_run_callback() Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/common/task/task.py", line 109, in _task_run_callback Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._set_result(self.run()) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/localization/installation.py", line 152, in run Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: write_x_configuration( Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/localization/installation.py", line 197, in write_x_configuration Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: localed_wrapper.set_layouts(x_layouts, switch_options) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/localization/localed.py", line 183, in set_layouts Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: self._localed_proxy.SetX11Keyboard( Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 450, in _call_method Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: return self._get_method_reply( Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^^^^^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 483, in _get_method_reply Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: return self._handle_method_error(error) Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 509, in _handle_method_error Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: raise exception from None Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: dasbus.error.DBusError: Local keyboard configuration not supported on this system. Mar 06 14:55:42 fedora initial-setup[673]: Initial Setup crashed due to unhandled exception: Traceback (most recent call last): File "/usr/libexec/initial-setup/initial-setup-text", line 10, in <module> is_instance.run() File "/usr/lib/python3.11/site-packages/initial_setup/__init__.py", line 459, in run self._apply() File "/usr/lib/python3.11/site-packages/initial_setup/__init__.py", line 313, in _apply sync_run_task(task_proxy) File "/usr/lib64/python3.11/site-packages/pyanaconda/modules/common/task/__init__.py", line 46, in sync_run_task task_proxy.Finish() File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 450, in _call_method return self._get_method_reply( ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 483, in _get_method_reply return self._handle_method_error(error) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/dasbus/client/handler.py", line 509, in _handle_method_error raise exception from None pyanaconda.modules.common.errors.general.AnacondaError: Local keyboard configuration not supported on this system. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:anaconda.modules.boss.module_manager.module_manager:Stop modules. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Network[835]: DEBUG:dasbus.connection:Disconnecting from the bus. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Network has quit. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Services[839]: DEBUG:dasbus.connection:Disconnecting from the bus. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Services has quit. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Users[841]: DEBUG:dasbus.connection:Disconnecting from the bus. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Users has quit. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Security[842]: DEBUG:dasbus.connection:Disconnecting from the bus. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Security has quit. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Modules.Localization[844]: DEBUG:dasbus.connection:Disconnecting from the bus. Mar 06 14:55:42 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Localization has quit. Mar 06 14:55:43 fedora org.fedoraproject.Anaconda.Modules.Timezone[847]: DEBUG:dasbus.connection:Disconnecting from the bus. Mar 06 14:55:43 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Timezone has quit. Mar 06 14:55:43 fedora org.fedoraproject.Anaconda.Boss[815]: DEBUG:dasbus.connection:Disconnecting from the bus. Mar 06 14:55:44 fedora initial-setup[946]: Initial Setup failed, keeping enabled Mar 06 14:55:44 fedora systemd[1]: initial-setup.service: Main process exited, code=exited, status=1/FAILURE Mar 06 14:55:45 fedora systemd[1]: initial-setup.service: Failed with result 'exit-code'. It looks like that happens *before* it applies the root password and user creation, so those things don't get done.
The fact that the error comes from systemd-localed , combined with the fact that systemd changed in the affected compose, makes me suspect systemd is the cause here. Re-assigning. Might well still be some kind of missing dependency.
OK, so, I think this ultimately fails because libxkbcommon isn't there. We're hitting a codepath in systemd-localed where it tries to 'verify' an xkb layout, this involves dlopen'ing libxkbcommon.so.0 , and the minimal image does not have that file. However, that codepath itself is quite old and hasn't changed recently. So what must be new is that we're *hitting it* now when we weren't before. The check that fails is `verify_xkb_rmlvo` in src/locale/localed.c , when called from the same file's `method_set_x11_keyboard`. So either we were previously hitting `method_set_x11_keyboard` but not getting to the `verify_xkb_rmlvo` call, ot previously we weren't hitting `method_set_x11_keyboard` at all. I believe the trigger is likely a series of changes by Yu Watanabe starting at b41ec10ae4073c92cb0487edcd9898e44f10ee11 , but I'm not sure precisely which one.
OK, there was a lot of messing around that confused things a bit, but I *suspect* this is likely down to 800f65f827c9828d4c872d44b19ca8a008505690 , "locale: always check input keyboard layout and friends earlier". The effect of that is to change the order of the checks in method_set_x11_keyboard around a bit. Notably, before that change, this: if (x11_context_equal(xc, &in)) return sd_bus_reply_method_return(m, NULL); happens *before* the `verify_xkb_rmlvo()` call. After that change, it happens *after* the `verify_xkb_rmlvo()` call. AIUI, that block is basically "return early with no error if we don't have to do anything". Subsequent commits make various further changes, but that I think is probably the key one. I suspect we were hitting that "return early" block before, so we never reached the `verify_xkb_rmlvo()` call. One question that triggers for me is, would you actually have this problem with *earlier* images/releases if you do try to change the keyboard layout on the minimal image? I'll have to try that and see. The answer to that might determine how we want to fix it.
OK, practically speaking, it seems like you can't really change the keyboard layout in initial-setup, at least not the TUI one. You can change the locale setting, but doing so doesn't set the matching keyboard layout, it doesn't seem (e.g. I changed it to French, but my keyboard layout is still US). So if I'm right, I think it wasn't possible to trigger this before, because localed would always conclude it wasn't going to do anything to the keyboard layout and bail out early. So...if I'm right, one way to fix this would be to have systemd depend on libxkbcommon, but I imagine that might not be popular with dependency minimalists? We could also have initial-setup or anaconda depend on it. I'd like to get a 38 Beta candidate built today, so I might have to do one of those myself at least as a short-term option.
aha, systemd already *recommends* libxkbcommon , but I guess this intentionally isn't a requires because some folks want very minimal deps? Zbigniew added that dep in 2016 with the changelog entry "Require libxkbcommon explictly, since the automatic dependency will not be generated anymore", but despite that text it is a recommends: not a requires: . Also, in fedora-kickstarts, we explicitly exclude xkeyboard-config from the minimal image in `fedora-minimal-common.ks`. libxbcommon requires xkeyboard-config. So I suspect that since libxkbcommon is only *recommended* by systemd (not required), whatever ultimately does the depsolving here decides to honor the request to exclude xkeyboard-config , and therefore also excludes libxkbcommon , which it thinks is OK because it's not a *hard* dep. So we could attack this at several points, I guess. Make the systemd dep a hard requires rather than a recommends. Or drop the exclusion of xkeyboard-config in fedora-minimal-common.ks; it only saves 1MB anyhow. I'm going to see if I can get a 'scratch' image build done with that exclusion dropped and see what the result is.
So, good news, dropping the xkeyboard-config exclusion from fedora-minimal-common.ks does solve the bug. Kinda bad news, it seemed to make the image much bigger. Today's Rawhide nightly image was 1088891096 bytes compressed. The test image we built with the kickstart change is 1271802708 bytes compressed. This doesn't make much sense to me, because according to dnf and rpm, xkeyboard-config and libxkbcommon combined are only 7MB uncompressed. But them's the numbers. Still, I don't think I can come up with a safe way to fix this in systemd on short notice, so for a compose today, putting those packages on the image seems like the only option. So we may have to go with it at least for now.
We merged the PR for the Beta candidate compose. Just need to confirm the fix (openQA will tell us once the compose is done).
Gah. Somehow the minimal image build for the Beta candidate compose failed entirely. I don't think this change was the cause, because our test image built fine, and so did the Rawhide image for today's Rawhide nightly (which also passed tests, confirming the fix works).
I also filed https://github.com/systemd/systemd/pull/26714 on systemd's side.
Tested Fedora-Minimal-38_Beta-1.3.aarch64.raw.xz on RPi4 and I'm able to create a user. In the subsequent reboot, the initial setup is not presented again.
Can we close this? Last comment says the issue is fixed.
Oh, yeah, we can. I guess we could drop xkeyboard-config from the minimal image again once the upstream systemd fix is in Fedora, too...