Bug 2242219 - systemctl daemon-reexec breaks Alt-Fn key handling in Xorg
Summary: systemctl daemon-reexec breaks Alt-Fn key handling in Xorg
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 37
Hardware: Unspecified
OS: Linux
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
: 2249991 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2023-10-05 02:29 UTC by Radu Rendec
Modified: 2023-12-05 01:09 UTC (History)
10 users (show)

Fixed In Version: systemd-251.19-1.fc37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2023-12-05 01:09:43 UTC
Type: ---

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github systemd systemd-stable pull 277 0 None Merged v251 batch 2023-10-24 13:43:07 UTC
Github systemd systemd pull 27107 0 None Merged core: do early setup check for arguments with '=' too 2023-10-24 13:43:07 UTC

Description Radu Rendec 2023-10-05 02:29:09 UTC
I noticed that kernel or glibc package updates started to break Alt-Fn key handling. Normally, while Xorg is running, Alt-Fn key combinations do *not* switch the virtual console, and they can be used as shortcuts in the desktop environment (e.g. Alt-F4 to close a window). To switch the console while Xorg is active on the current console, one would have to press Ctrl-Alt-Fn, while Alt-Fn combinations still work on text mode console.

However, starting a few weeks ago, after a kernel or glibc package update, Alt-Fn combinations are handled by Xorg and at the same time they switch the virtual console.

I narrowed down the problem to the `systemctl daemon-reexec` command being run from the RPM scriptlets. For example, the glibc package runs this command at the end of the postinstall scriptlet.

Reproducible: Always

Steps to Reproduce:
From a desktop environment, run `systemctl daemon-reexec` as root in a terminal.

Actual Results:  
While the active virtual console is the Xorg session, Alt-Fn key combinations are handled both by Xorg and by the kernel (to switch the active console).

Expected Results:  
While the active virtual console is the Xorg session, Alt-Fn key combinations are handled by Xorg only. A Ctrl-Alt-Fn key combination is required to switch the virtual console away from the Xorg session.

I noticed that there is some interaction with the `kbd_mode` utility:

[root@thinkpad-p1 ~]# kbd_mode
The keyboard is in some unknown mode
[root@thinkpad-p1 ~]# systemctl daemon-reexec
[root@thinkpad-p1 ~]# kbd_mode
The keyboard is in Unicode (UTF-8) mode

Also, at this point, running `kbd_mode -k` seems to fix the issue. However, running `kbd_mode` again (with no parameters, to display the current mode) seems to crash Xorg. This could be a different bug.

Comment 1 David Tardon 2023-10-10 08:57:48 UTC
systemd initializes /dev/console, but only on first boot, not on reexec. Could you run `perf trace -e ioctl 2>&1 | grep -F KDSKBMODE` alongside your reproducer? That should list all processes that change the keyboard mode of any console.

Comment 2 Radu Rendec 2023-10-11 18:46:30 UTC
As it turns out, it's actually systemd :-/
For some reason, perf doesn't recognize KDSKBMODE, but still shows the ioctl request number in hex:

[root@thinkpad-p1 ~]# perf trace -e ioctl 2>&1 | grep '0x4b.*0x45' 
  8466.001 ( 0.002 ms): systemd/1 ioctl(fd: 3, cmd: (NONE, 0x4b, 0x45, 0), arg: 0x3)                    = 0

[root@thinkpad-p1 ~]# grep -rn KDSKBMODE /usr/include/
/usr/include/linux/kd.h:87:#define KDSKBMODE	0x4B45	/* sets current keyboard mode */

(and I got similar results with kbd_mode, which I could match against the strace log on kbd_mode).

[root@thinkpad-p1 ~]# strace -e ioctl -o log5 -p 1
[root@thinkpad-p1 ~]# grep KB log5
ioctl(3, KDSKBMODE, K_UNICODE)          = 0

Then I tried something really bold, like:
- install the debuginfo packages for glibc and systemd
- attach with gdb to pid 1 and set a conditional breakpoint in ioctl()
- run `systemctl daemon-reexec` (the reproducer)

Unfortunately, gdb catches an internal error and crashes when systemd re-executes itself, and that's before it tries to run ioctl. Apparently debugging systemd is trickier than a regular program. If you have a better idea, please let me know. Thanks!

Comment 3 David Tardon 2023-10-19 09:14:24 UTC
The only thing I can think of is that the command line filtering we do misbehaves somehow, thus early_skip_setup_check() fails to detect that systemd is just reexecuting. I see systemd in F37 is lagging a bit: there's v251.14 there but v251.18 is already available. In particular, v215.14 is missing this commit: https://github.com/systemd/systemd-stable/commit/a6ae06c41e50e458e7203e7e8806e58ed0837670 , which could be relevant if my assumption is correct.

@rrendec Could you show us the content of /proc/1/cmdline? And if you're up for some experimenting, could you rebuild systemd with the patch mentioned above and try again? (Note that you'd have to reexec once, fixup the keyboard mode (as the reexec command line would still have been created by the old systemd) and then reexec again to check if it works.)

@zbyszek Maybe an update is in order? There's not that much time to EOL left...

Comment 4 Radu Rendec 2023-10-19 12:59:08 UTC
Thanks for the update! Sure, the systemd command line is included below.

[root@thinkpad-p1 ~]# xargs -0 < /proc/1/cmdline 
/usr/lib/systemd/systemd rhgb --switched-root --system --deserialize=31

As for the missing commit, I'll rebuild later today and post an update here.
I need to reboot my system anyway to take a kernel update, so I'll boot straight with the new systemd.

Comment 5 Radu Rendec 2023-10-20 15:43:29 UTC
I tried the missing commit. No dice :(

[root@thinkpad-p1 ~]# kbd_mode 
The keyboard is in some unknown mode
[root@thinkpad-p1 ~]# systemctl daemon-reexec 
[root@thinkpad-p1 ~]# kbd_mode 
The keyboard is in Unicode (UTF-8) mode
[root@thinkpad-p1 ~]# 

I rebooted the system *after* updating systemd, and I double-checked that the patch had been applied by looking at /usr/src/debug/systemd-251.14-2.fc37.x86_64/src/core/main.c (from systemd-debugsource-251.14-2.fc37.x86_64). The filter_args() call comes after the code that injects the args, as expected.

Comment 6 David Tardon 2023-10-24 09:08:16 UTC
It turns out I can reproduce this... It is a regression between systemd-251.10-588.fc37 and systemd-251.14-2.fc37 .

Comment 8 David Tardon 2023-10-24 13:43:07 UTC
Further bisection shows this has been fixed by https://github.com/systemd/systemd/commit/4f44d2c4f76922a4f48dd4473e6abaca40d7e555 (https://github.com/systemd/systemd-stable/commit/8c741666fa49a8c87910e2da7832dcf1f414e286 in v251-stable). Hence, all we need is to update to v251.18.

Comment 9 David Tardon 2023-11-20 14:35:46 UTC
*** Bug 2249991 has been marked as a duplicate of this bug. ***

Comment 10 Aoife Moloney 2023-11-23 01:50:48 UTC
This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 11 Fedora Update System 2023-11-27 14:11:11 UTC
FEDORA-2023-1582a08b60 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-1582a08b60

Comment 12 Fedora Update System 2023-11-28 02:50:58 UTC
FEDORA-2023-1582a08b60 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-1582a08b60`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-1582a08b60

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2023-12-05 01:09:43 UTC
FEDORA-2023-1582a08b60 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.