The PATH environment variable is different in Bash instances running inside the Ptyxis terminal emulation application on freshly installed Fedora 42 Silverblue and Workstation virtual machines with clean home directories. On Fedora 42 Silverblue: rishi@fedora:~$ systemd-run --user -t env | grep "^PATH=" ... PATH=/var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin rishi@fedora:~$ echo "$PATH" /var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin On Fedora 42 Workstation, the 'sbin' directories are absent, which is what I expected because of the Unify_bin_and_sbin Change: https://fedoraproject.org/wiki/Changes/Unify_bin_and_sbin I don't think this is because of GDM's default-path option, because the relative order of the 'sbin' and 'bin' directories don't match. Here 'sbin' precedes 'bin' and GDM puts them the other way round. It's not /etc/profile. It would have skipped adding /usr/sbin because it's a symbolic link. As an aside, /usr/local/sbin is not a symlink on Silverblue. I have found that pkexec(1) and su(1) have their own definitions of PATH, but at least I am not explicitly using them here. I see that PATH is different in the 'systemd --user' and dbus-broker-launch(1) processes. The former has: PATH=/usr/local/bin:/usr/bin ... and the latter has: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin I think PATH in Bash running inside Ptyxis picks up the 'sbin' directories from dbus-broker(1) because it's launched by D-Bus. To verify this, I logged in through the textual console: rishi@fedora:~$ systemd-run --user -t env | grep "^PATH=" ... PATH=/var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin rishi@fedora:~$ echo "$PATH" /var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/bin:/usr/bin :/usr/local/sbin Notice how the 'systemd-run --user' output didn't change, but PATH in Bash running in the console launched without D-Bus changed to not have /usr/sbin because of /etc/profile. I have these OSTree deployments, and I can reproduce this with all of them: rishi@fedora:~$ rpm-ostree status State: idle Deployments: ● fedora:fedora/42/x86_64/silverblue Version: 42.20250928.0 (2025-09-28T01:32:25Z) Commit: 47be64ee847137938e489154856afb522d20e4e7e78aa7fb923352a5e4262667 GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944 fedora:fedora/42/x86_64/silverblue Version: 42.20250925.0 (2025-09-25T00:33:27Z) Commit: 4d1f2084255f975c5c26bcc2c2f13b53f1fc83fc9f9e0ac0837f7a667f9f0f27 GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944 Pinned: yes fedora:fedora/42/x86_64/silverblue Version: 42.1.1 (2025-04-09T11:03:58Z) Commit: 968168b643f6123c39e2fbd59b238c85a0fe29a2720da9f185bd7106ae27497b GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944 Pinned: yes The corresponding systemd NEVRAs are: systemd-257.9-2.fc42.x86_64 systemd-257.9-2.fc42.x86_64 systemd-257.3-7.fc42.x86_64 Reading the sources of the systemd RPM and the code (ie., default_user_PATH(), default_PATH(), manager_default_environment()), it's not obvious to me how the 'sbin' directories can show up. I am running out of time to debug this further on my own, so I thought I'd file a bug in case somebody knows what's going on. Reproducible: Always
This seems to be an ostree thing. The code in systemd will use $PATH with /usr/sbin and /usr/local/sbin if it sees separate /usr/sbin or /usr/local/sbin. In this case, one of the directories is separate. The user manager ends up with the expected split $PATH. The system manager does not. But when I do 'systemctl daemon-reexec', it gets the split $PATH. So it seems that somehow during boot, the directory does not exist or is not visible. Anyway, the solution is for rpm-ostree to not create the split path, but a symlink for /usr/local/sbin.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #1) > This seems to be an ostree thing. Thanks for looking into this, Zbigniew! > The code in systemd will use $PATH with /usr/sbin and /usr/local/sbin if it > sees separate /usr/sbin or /usr/local/sbin. In this case, one of the > directories is separate. > > The user manager ends up with the expected split $PATH. > > The system manager does not. But when I do 'systemctl daemon-reexec', it > gets the split $PATH. So it seems that somehow during boot, the directory > does not exist or is not visible. > > Anyway, the solution is for rpm-ostree to not create the split path, but a > symlink for /usr/local/sbin. Just to be sure that I understood you correctly. Are you saying that systemd will include both /usr/local/sbin and /usr/sbin, if only one of them is a separate directory instead of a symbolic link to its 'bin' counterpart? I am asking because I knew that /usr/local/sbin is not a symlink to /usr/local/bin on Fedora Silverblue, because it handles /usr/local differently, and /usr/sbin is. My original assumption was to treat this as a separate issue, and I was expecting at least /usr/sbin to not be in PATH similar to how /etc/profile behaves. I am not trying to imply that systemd is right or wrong - just trying to be sure that I understood correctly. :)
The `/usr/local/sbin` directory is created via https://github.com/coreos/rpm-ostree/blob/main/src/app/rpm-ostree-0-integration.conf#L15 and is stored in `/var` (`/usr/local` points to `/var/usrlocal`). We can replace this entry with a symlink instead but we probably can not find existing systems.
> it seems that somehow during boot, the directory does not exist or is not visible. /usr/local is a symlink to /var/usrlocal for Silverblue/CoreOS style appliances where it's intended to support writing content there per-machine. It's created by tmpfiles.d, so yes it won't be there early on boot. > Anyway, the solution is for rpm-ostree to not create the split path, but a symlink for /usr/local/sbin. Symlink to where? I think it's tricky as in the general case here `/var` could be a mount point; when is systemd computing this path?
(In reply to Colin Walters from comment #4) > > Anyway, the solution is for rpm-ostree to not create the split path, but a symlink for /usr/local/sbin. > > Symlink to where? I believe it should be something like this: ls -ld /usr/local/sbin lrwxrwxrwx. 1 root root 3 May 16 23:24 /usr/local/sbin -> bin ... to match with the Unify_bin_and_sbin Change: https://fedoraproject.org/wiki/Changes/Unify_bin_and_sbin
> Just to be sure that I understood you correctly. Are you saying that systemd will include both /usr/local/sbin and /usr/sbin, if only one of them is a separate directory instead of a symbolic link to its 'bin' counterpart? Yes. (Longer explanation: we do one check, and if any of the conditions are met, we fall into the "split bin-sbin" state. The $PATH value is actually set through a set of parametrized macros, and we just flip between one value or the other.) We can probably change the code to add /usr/sbin and /usr/local/sbin separately to $PATH. This would have the advantage that it'd be more self-explanatory for users. I'll see how complicated it would be to change the code to do that. > lrwxrwxrwx. 1 root root 3 May 16 23:24 /usr/local/sbin -> bin Yes.
https://github.com/systemd/systemd/pull/39275 will make the $PATH handle merged-usr-sbin and split-usr-local-sbin.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #7) > https://github.com/systemd/systemd/pull/39275 will make the $PATH handle > merged-usr-sbin and split-usr-local-sbin. Thanks, Zbigniew! Note to myself: I should have stopped to think that STRV_FOREACH_PAIR in default_PATH() is a loop - no idea why I missed that. :)
(In reply to Timothée Ravier from comment #3) > The `/usr/local/sbin` directory is created via > https://github.com/coreos/rpm-ostree/blob/main/src/app/rpm-ostree-0- > integration.conf#L15 and is stored in `/var` (`/usr/local` points to > `/var/usrlocal`). > > We can replace this entry with a symlink instead but we probably can not > find existing systems. I tried to put together a pull request to update rpm-ostree: https://github.com/coreos/rpm-ostree/pull/5507