Bug 2400220

Summary: 'systemd --user' sets PATH with 'sbin' directories on Silverblue
Product: [Fedora] Fedora Reporter: Debarshi Ray <debarshir>
Component: rpm-ostreeAssignee: Colin Walters <walters>
Status: POST --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rawhideCC: amurdaca, coreos-sig, daan.j.demeyer, dustymabe, fedoraproject, jmarrero, jonathan, lnykryn, miabbott, msekleta, suraj.ghimire7, systemd-maint, travier, walters, yuwatana, zbyszek
Target Milestone: ---Flags: fedora-admin-xmlrpc: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Debarshi Ray 2025-09-29 20:50:31 UTC 
The PATH environment variable is different in Bash instances running inside the Ptyxis terminal emulation application on freshly installed Fedora 42 Silverblue and Workstation virtual machines with clean home directories.

On Fedora 42 Silverblue:

rishi@fedora:~$ systemd-run --user -t env | grep "^PATH="
...
PATH=/var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

rishi@fedora:~$ echo "$PATH"
/var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

On Fedora 42 Workstation, the 'sbin' directories are absent, which is what I expected because of the Unify_bin_and_sbin Change:
https://fedoraproject.org/wiki/Changes/Unify_bin_and_sbin

I don't think this is because of GDM's default-path option, because the relative order of the 'sbin' and 'bin' directories don't match.  Here 'sbin' precedes 'bin' and GDM puts them the other way round.

It's not /etc/profile.  It would have skipped adding /usr/sbin because it's a symbolic link.  As an aside, /usr/local/sbin is not a symlink on Silverblue.

I have found that pkexec(1) and su(1) have their own definitions of PATH, but at least I am not explicitly using them here.

I see that PATH is different in the 'systemd --user' and dbus-broker-launch(1) processes.  The former has:
PATH=/usr/local/bin:/usr/bin

... and the latter has:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

I think PATH in Bash running inside Ptyxis picks up the 'sbin' directories from dbus-broker(1) because it's launched by D-Bus.

To verify this, I logged in through the textual console:

rishi@fedora:~$ systemd-run --user -t env | grep "^PATH="
...
PATH=/var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

rishi@fedora:~$ echo "$PATH"
/var/home/rishi/.local/bin:/var/home/rishi/bin:/usr/local/bin:/usr/bin
:/usr/local/sbin

Notice how the 'systemd-run --user' output didn't change, but PATH in Bash running in the console launched without D-Bus changed to not have /usr/sbin because of /etc/profile.

I have these OSTree deployments, and I can reproduce this with all of them:

rishi@fedora:~$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/42/x86_64/silverblue
                  Version: 42.20250928.0 (2025-09-28T01:32:25Z)
                   Commit: 47be64ee847137938e489154856afb522d20e4e7e78aa7fb923352a5e4262667
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944

  fedora:fedora/42/x86_64/silverblue
                  Version: 42.20250925.0 (2025-09-25T00:33:27Z)
                   Commit: 4d1f2084255f975c5c26bcc2c2f13b53f1fc83fc9f9e0ac0837f7a667f9f0f27
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
                   Pinned: yes

  fedora:fedora/42/x86_64/silverblue
                  Version: 42.1.1 (2025-04-09T11:03:58Z)
                   Commit: 968168b643f6123c39e2fbd59b238c85a0fe29a2720da9f185bd7106ae27497b
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
                   Pinned: yes

The corresponding systemd NEVRAs are:
systemd-257.9-2.fc42.x86_64
systemd-257.9-2.fc42.x86_64
systemd-257.3-7.fc42.x86_64

Reading the sources of the systemd RPM and the code (ie., default_user_PATH(), default_PATH(), manager_default_environment()), it's not obvious to me how the 'sbin' directories can show up.  I am running out of time to debug this further on my own, so I thought I'd file a bug in case somebody knows what's going on.

Reproducible: Always
Comment 1 Zbigniew Jędrzejewski-Szmek 2025-10-02 13:58:38 UTC 
This seems to be an ostree thing.

The code in systemd will use $PATH with /usr/sbin and /usr/local/sbin if it sees
separate /usr/sbin or /usr/local/sbin. In this case, one of the directories is separate.
The user manager ends up with the expected split $PATH.
The system manager does not. But when I do 'systemctl daemon-reexec', it gets the
split $PATH. So it seems that somehow during boot, the directory does not exist
or is not visible.

Anyway, the solution is for rpm-ostree to not create the split path, but a symlink
for /usr/local/sbin.
Comment 2 Debarshi Ray 2025-10-02 14:08:46 UTC 
(In reply to Zbigniew Jędrzejewski-Szmek from comment #1)
> This seems to be an ostree thing.

Thanks for looking into this, Zbigniew!

> The code in systemd will use $PATH with /usr/sbin and /usr/local/sbin if it
> sees separate /usr/sbin or /usr/local/sbin. In this case, one of the
> directories is separate.
>
> The user manager ends up with the expected split $PATH.
>
> The system manager does not. But when I do 'systemctl daemon-reexec', it
> gets the split $PATH. So it seems that somehow during boot, the directory
> does not exist or is not visible.
> 
> Anyway, the solution is for rpm-ostree to not create the split path, but a
> symlink for /usr/local/sbin.

Just to be sure that I understood you correctly.  Are you saying that systemd will include both /usr/local/sbin and /usr/sbin, if only one of them is a separate directory instead of a symbolic link to its 'bin' counterpart?

I am asking because I knew that /usr/local/sbin is not a symlink to /usr/local/bin on Fedora Silverblue, because it handles /usr/local differently, and /usr/sbin is.  My original assumption was to treat this as a separate issue, and I was expecting at least /usr/sbin to not be in PATH similar to how /etc/profile behaves.

I am not trying to imply that systemd is right or wrong - just trying to be sure that I understood correctly.  :)
Comment 3 Timothée Ravier 2025-10-02 14:16:51 UTC 
The `/usr/local/sbin` directory is created via https://github.com/coreos/rpm-ostree/blob/main/src/app/rpm-ostree-0-integration.conf#L15 and is stored in `/var` (`/usr/local` points to `/var/usrlocal`).

We can replace this entry with a symlink instead but we probably can not find existing systems.
Comment 4 Colin Walters 2025-10-02 14:38:36 UTC 
> it seems that somehow during boot, the directory does not exist or is not visible.

/usr/local is a symlink to /var/usrlocal for Silverblue/CoreOS style appliances where it's intended to support writing content there per-machine.

It's created by tmpfiles.d, so yes it won't be there early on boot.

> Anyway, the solution is for rpm-ostree to not create the split path, but a symlink for /usr/local/sbin.

Symlink to where?

I think it's tricky as in the general case here `/var` could be a mount point; when is systemd computing this path?
Comment 5 Debarshi Ray 2025-10-02 14:48:08 UTC 
(In reply to Colin Walters from comment #4)
> > Anyway, the solution is for rpm-ostree to not create the split path, but a symlink for /usr/local/sbin.
> 
> Symlink to where?

I believe it should be something like this:

ls -ld /usr/local/sbin
lrwxrwxrwx. 1 root root 3 May 16 23:24 /usr/local/sbin -> bin

... to match with the Unify_bin_and_sbin Change:
https://fedoraproject.org/wiki/Changes/Unify_bin_and_sbin
Comment 6 Zbigniew Jędrzejewski-Szmek 2025-10-04 13:56:01 UTC 
> Just to be sure that I understood you correctly.  Are you saying that systemd will include both /usr/local/sbin and /usr/sbin, if only one of them is a separate directory instead of a symbolic link to its 'bin' counterpart?

Yes. (Longer explanation: we do one check, and if any of the conditions are met, we fall into the "split bin-sbin" state. The $PATH value is actually set through a set of parametrized macros, and we just flip between one value or the other.)

We can probably change the code to add /usr/sbin and /usr/local/sbin separately to $PATH. This would have the advantage that it'd be more self-explanatory for users. I'll see how complicated it would be to change the code to do that.

> lrwxrwxrwx. 1 root root 3 May 16 23:24 /usr/local/sbin -> bin

Yes.
Comment 7 Zbigniew Jędrzejewski-Szmek 2025-10-10 12:43:55 UTC 
https://github.com/systemd/systemd/pull/39275 will make the $PATH handle merged-usr-sbin and split-usr-local-sbin.
Comment 8 Debarshi Ray 2025-10-10 14:54:11 UTC 
(In reply to Zbigniew Jędrzejewski-Szmek from comment #7)
> https://github.com/systemd/systemd/pull/39275 will make the $PATH handle
> merged-usr-sbin and split-usr-local-sbin.

Thanks, Zbigniew!

Note to myself: I should have stopped to think that STRV_FOREACH_PAIR in default_PATH() is a loop - no idea why I missed that.  :)
Comment 9 Debarshi Ray 2025-10-10 19:43:07 UTC 
(In reply to Timothée Ravier from comment #3)
> The `/usr/local/sbin` directory is created via
> https://github.com/coreos/rpm-ostree/blob/main/src/app/rpm-ostree-0-
> integration.conf#L15 and is stored in `/var` (`/usr/local` points to
> `/var/usrlocal`).
> 
> We can replace this entry with a symlink instead but we probably can not
> find existing systems.

I tried to put together a pull request to update rpm-ostree:
https://github.com/coreos/rpm-ostree/pull/5507
Comment 10 Fedora Release Engineering 2026-05-06 14:27:13 UTC 
This message is a reminder that Fedora Linux 42 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 42 on 2026-05-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '42'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 42 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.