Bug 2044100 - Fedora Silvervlue kind of broken on recent updates: /run/user/1000: no such file or directory
Summary: Fedora Silvervlue kind of broken on recent updates: /run/user/1000: no such f...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: selinux-policy
Version: 35
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Zdenek Pytela
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-23 21:45 UTC by slt
Modified: 2022-12-13 16:25 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-12-13 16:25:40 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description slt 2022-01-23 21:45:30 UTC
Description of problem:
 
Many things are not working, for example toolbox:
 $ toolbox enter
 Error: failed to get the Podman version
 $ podman version
 Error: error creating tmpdir: mkdir /run/user/1000: permission denied
 $ podman -v 
 podman version 3.4.4

Then, here is what I found when trying to find the source of the problem:

 $ sudo systemctl list-units --failed 
   UNIT                          LOAD   ACTIVE SUB    DESCRIPTION                          
 ● user-runtime-dir loaded failed failed User Runtime Directory /run/user/1000
 ● user-runtime-dir   loaded failed failed User Runtime Directory /run/user/42



 $ sudo systemctl status user-runtime-dir
 × user-runtime-dir - User Runtime Directory /run/user/1000
      Loaded: loaded (/usr/lib/systemd/system/user-runtime-dir@.service; static)
      Active: failed (Result: exit-code) since Sun 2022-01-23 22:04:31 CET; 18min ago
        Docs: man:user@.service(5)
     Process: 1660 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir start 1000 (code=exited, status=1/FAILURE)
    Main PID: 1660 (code=exited, status=1/FAILURE)
         CPU: 15ms
 
 janv. 23 22:04:31 nsfw systemd[1]: Starting User Runtime Directory /run/user/1000...
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: mmap: Permission denied
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: Failed to mount per-user tmpfs directory /run/user/1000: No such file or directory
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir: Main process exited, code=exited, status=1/FAILURE
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir: Failed with result 'exit-code'.
 janv. 23 22:04:31 nsfw systemd[1]: Failed to start User Runtime Directory /run/user/1000.


 journalctl -u user-runtime-dir
 [....]
 -- Boot 6bde8538db544eb48470f956f7f15aa9 --           <= THIS WAS ON MY LAST WORKING SYSTEM: Silverblue 35.20220115.0
 janv. 23 21:17:41 nsfw systemd[1]: Starting User Runtime Directory /run/user/1000...
 janv. 23 21:17:41 nsfw systemd-user-runtime-dir[1600]: mmap: Permission denied
 janv. 23 21:17:41 nsfw systemd[1]: Finished User Runtime Directory /run/user/1000.
 janv. 23 22:03:41 nsfw systemd[1]: Stopping User Runtime Directory /run/user/1000...
 janv. 23 22:03:41 nsfw systemd-user-runtime-dir[19789]: mmap: Permission denied
 janv. 23 22:03:41 nsfw systemd[1]: user-runtime-dir: Deactivated successfully.
 janv. 23 22:03:41 nsfw systemd[1]: Stopped User Runtime Directory /run/user/1000.
 -- Boot 96ddbd02ba1645fc9d4b4e20f9597171 --            <= THIS IS ON THE LATEST SYSTEM: Silverblue 35.20220123.0)
 janv. 23 22:04:31 nsfw systemd[1]: Starting User Runtime Directory /run/user/1000...
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: mmap: Permission denied
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: Failed to mount per-user tmpfs directory /run/user/1000: No such file or directory
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir: Main process exited, code=exited, status=1/FAILURE
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir: Failed with result 'exit-code'.
 janv. 23 22:04:31 nsfw systemd[1]: Failed to start User Runtime Directory /run/user/1000.


The output is similar with user-runtime-dir.



Version-Release number of selected component (if applicable):


Things got wrong a few days ago, my last working version is 35.20220115.0 (see below).

 $ rpm-ostree status
 State: idle
 Deployments:
 ● fedora:fedora/35/x86_64/silverblue
                   Version: 35.20220123.0 (2022-01-23T01:17:10Z)
                BaseCommit: d3f36f449b3e3a7de2a24187caeb3780407d71958cb11ed751e561d489766710
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
           LayeredPackages: aircrack-ng gnome-shell-extension-gsconnect gnome-tweaks gocryptfs gstreamer1-plugin-openh264 java-latest-openjdk kernel-tools lxc mozilla-openh264 neovim nmap p7zip reaver socat systemd-container tcpdump tmux
                            w3m wgctrl wireguard-tools

   fedora:fedora/35/x86_64/silverblue
                   Version: 35.20220115.0 (2022-01-15T00:49:28Z)
                BaseCommit: 4bb8301ce5639227589617b788943b1f95f93889407eec1ee63a762cb0a94c01
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
           LayeredPackages: aircrack-ng gnome-shell-extension-gsconnect gnome-tweaks gocryptfs gstreamer1-plugin-openh264 java-latest-openjdk kernel-tools lxc mozilla-openh264 neovim nmap p7zip reaver socat systemd-container tcpdump tmux
                            w3m wgctrl wireguard-tools


How reproducible:


Steps to Reproduce:
1. update system
2. reboot to new system
3. 

Actual results:

Many things are not working

Expected results:

everything works as with Silverblue 35.20220115.0

Additional info:

Here is an extract from journalctl with more context, and especially with some auditd messages:
-------------
 janv. 23 22:04:23 nsfw systemd[1]: Starting User Runtime Directory /run/user/42...
 janv. 23 22:04:23 nsfw systemd-logind[953]: New session c1 of user gdm.
 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { map } for  pid=1087 comm="systemd-user-ru" path="/etc/selinux/targeted/contexts/files/file_contexts.bin" dev="dm-0" ino=6103090 scontext=system_u:system_r:systemd_logind_t:s0 tconte>
 janv. 23 22:04:23 nsfw audit[1087]: SYSCALL arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=8dd7e a2=1 a3=2 items=0 ppid=1 pid=1087 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 com>
 janv. 23 22:04:23 nsfw audit: PROCTITLE proctitle=2F7573722F6C69622F73797374656D642F73797374656D642D757365722D72756E74696D652D646972007374617274003432
 janv. 23 22:04:23 nsfw systemd-user-runtime-dir[1087]: mmap: Permission denied
 janv. 23 22:04:23 nsfw systemd[1]: Started User resource assignment daemon.
 janv. 23 22:04:23 nsfw audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=uresourced comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
 janv. 23 22:04:23 nsfw uresourced[1086]: Setting resources on user.slice (MemoryMin: 262144000, MemoryLow: 0, CPUWeight: -, IOWeight: -)
 janv. 23 22:04:23 nsfw uresourced[1086]: Setting resources on user-42.slice (MemoryMin: 262144000, MemoryLow: 0, CPUWeight: 500, IOWeight: 500)
 janv. 23 22:04:23 nsfw uresourced[1086]: Setting resources on user (MemoryMin: 0, MemoryLow: 0, CPUWeight: 100, IOWeight: 100)
 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { create } for  pid=1087 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
 janv. 23 22:04:23 nsfw audit[1087]: SYSCALL arch=c000003e syscall=83 success=no exit=-13 a0=7ffe05a517f0 a1=1c0 a2=5642b97e4800 a3=0 items=0 ppid=1 pid=1087 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(non>
 janv. 23 22:04:23 nsfw audit: PROCTITLE proctitle=2F7573722F6C69622F73797374656D642F73797374656D642D757365722D72756E74696D652D646972007374617274003432
 janv. 23 22:04:23 nsfw systemd-user-runtime-dir[1087]: Failed to mount per-user tmpfs directory /run/user/42: No such file or directory
 janv. 23 22:04:23 nsfw systemd[1]: user-runtime-dir: Main process exited, code=exited, status=1/FAILURE
 janv. 23 22:04:23 nsfw systemd[1]: user-runtime-dir: Failed with result 'exit-code'.
 janv. 23 22:04:23 nsfw systemd[1]: Failed to start User Runtime Directory /run/user/42.
 janv. 23 22:04:23 nsfw systemd[1]: Dependency failed for User Manager for UID 42.
 janv. 23 22:04:23 nsfw systemd[1]: user: Job user/start failed with result 'dependency' .
 janv. 23 22:04:23 nsfw audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user-runtime-dir@42 comm="systemd"  exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=faile>
 ------------------



Please tell me what additional info you might need!

Any help appreciated,

Thanks in advance,

Samuel

Comment 1 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:15:49 UTC
This looks similar to #2020977, I hope the underlying cause is the same.

*** This bug has been marked as a duplicate of bug 2020977 ***

Comment 2 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:19:31 UTC
Or maybe not. Let's keep them separate for now.

Comment 3 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:43:38 UTC
 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { map } for  pid=1087 comm="systemd-user-ru" path="/etc/selinux/targeted/contexts/files/file_contexts.bin" dev="dm-0" ino=6103090 scontext=system_u:system_r:systemd_logind_t:s0 tconte>

systemd-user-runtime-dir loads the selinux config because it wants to assign appropriate labels
to the directories is tries to create (/run/user and /run/user/<UID>).

 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { create } for  pid=1087 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0

So this could be either /run/user or /run/user/<UID>.
The code emits a warning for the first one, but just continues for the second one, based
on the assumption that the mount will fail anyway. I'll submit a cosmetic patch to also emit a warning
to make this clearer.

I have no idea why this starting erroring.

Comment 4 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:57:59 UTC
https://github.com/systemd/systemd/pull/22235 is the cosmetic patch for systemd-user-runtime-dir.

Comment 5 slt 2022-01-26 21:34:37 UTC
Hello again,

I am not really able to test this cosmetic patch by myself.

Is there something else I could do to understand what's going on and eventuall fix the problem?

Thank you in advance!

samuel

Comment 6 Danut Enachioiu 2022-01-29 23:14:47 UTC
Hello!

I think I'm having the same problem after upgrading selinux-policy on Silverblue from 35.8-1 to 35.9-1. It causes GDM to fail to start on Wayland (for some reason it starts fine after that on X11), meaning I can't use Wayland anymore.

Here are some notes from my investigation, by comparing `journalctl -b` before and after the selinux-policy change. After the change, I see

Jan 29 23:10:18 fedora systemd[1]: Starting User Runtime Directory /run/user/42...
Jan 29 23:10:18 fedora systemd[1]: Started Network Manager Script Dispatcher Service.
Jan 29 23:10:18 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd>
Jan 29 23:10:18 fedora systemd-user-runtime-dir[1093]: Regex version mismatch, expected: 10.39 2021-10-29 actual: 10.37 2021-05-26
Jan 29 23:10:18 fedora audit[1093]: AVC avc:  denied  { create } for  pid=1093 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_>
Jan 29 23:10:18 fedora systemd-user-runtime-dir[1093]: Failed to mount per-user tmpfs directory /run/user/42: No such file or directory
Jan 29 23:10:18 fedora systemd[1]: user-runtime-dir: Main process exited, code=exited, status=1/FAILURE
Jan 29 23:10:18 fedora systemd[1]: user-runtime-dir: Failed with result 'exit-code'.
Jan 29 23:10:18 fedora systemd[1]: Failed to start User Runtime Directory /run/user/42.
Jan 29 23:10:18 fedora systemd[1]: Dependency failed for User Manager for UID 42.
Jan 29 23:10:18 fedora systemd[1]: user: Job user/start failed with result 'dependency'.

Which then causes GDM to fail like this:

Jan 29 23:10:18 fedora gdm-launch-environment][1080]: pam_systemd(gdm-launch-environment:session): Failed to stat() runtime directory '/run/user/42': No such file or directory
Jan 29 23:10:18 fedora gdm-launch-environment][1080]: pam_systemd(gdm-launch-environment:session): Not setting $XDG_RUNTIME_DIR, as the directory is not in order.
[...]
Jan 29 23:10:19 fedora gnome-shell[1145]: Using public X11 display :1024, (using :1025 for managed services)
Jan 29 23:10:19 fedora gnome-shell[1145]: WL: error: XDG_RUNTIME_DIR not set in the environment
Jan 29 23:10:19 fedora audit[1145]: ANOM_ABEND auid=4294967295 uid=42 gid=42 ses=4294967295 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 pid=1145 comm="gnome-shell" exe="/usr/bin/gnome-shell" sig=>
Jan 29 23:10:19 fedora org.gnome.Shell.desktop[1145]: == Stack trace for context 0x55a82c2301f0 ==
Jan 29 23:10:19 fedora gnome-shell[1145]: Failed to create socket
Jan 29 23:10:19 fedora systemd[1]: Created slice Slice /system/systemd-coredump.
Jan 29 23:10:19 fedora systemd[1]: Started Process Core Dump (PID 1216/UID 0).
Jan 29 23:10:19 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-1216-0 comm="systemd" exe="/usr/lib/systemd>
Jan 29 23:10:19 fedora systemd-coredump[1217]: [🡕] Process 1145 (gnome-shell) of user 42 dumped core.

To be clear, none of these errors are present in the logs in version 35.8-1. It seems very likely to be caused by commit b3c7810107cfa023cab14b16eaf7de4f8401142c, which includes:

"- Change /run/user/[0-9]+ to /run/user/%{USERID} for proper labeling"

Unfortunately, I don't understand enough about selinux to be able to dig deeper but let me know if I can help with anything else.

Comment 7 Josh Belanich 2022-02-07 05:10:31 UTC
Hi all! I'm running into the same issue as Danut -- GDM fails, after which it loads into X11 and things like toolbox and running flatpaks appear broken. I'm also seeing the same boot log pattern as Danut. I was able to pinpoint the breaking change to Fedora Silverblue 35.20220118.0 (35.20220117.0 works fine). Let me know if there is anything information I can give that can help. Thanks!

Comment 8 slt 2022-03-16 10:18:43 UTC
Hello all,

I'm still kind of trapped in Silverblue 35.20220115.0 (yes from January!)

Did some of you find a way out of the issue? I would be pleased to learn about,

Thank you

Comment 9 Danut Enachioiu 2022-03-16 23:50:56 UTC
Well, I disabled SELinux enforcement. Instructions here: https://docs.fedoraproject.org/en-US/quick-docs/changing-selinux-states-and-modes/#selinux-changing-to-permissive-mode

I'd rather have up-to-date packages and no SELinux, probably makes sense even for security.

Comment 10 Daniel Stiner 2022-04-25 07:15:26 UTC
Seeing this also in Silverblue 35.20220423.0, does not repro on "regular" Fedora 35. I agree with Danut on the labeling change is the likely cause, link: https://github.com/fedora-selinux/selinux-policy/commit/925e0fcfc3747fa0d8bae5c6f266b4f2d754b6f5

I'm trying to confirm that commit is the issue by applying a patched selinux policy that reverts the commit, but it is taking time to understand how to do that in Silverblue. Also unsure if I should open a Github issue in the selinux-policy repo, or if this bug tracker is the better place to track this issue.

What I have figured out from my local logs is that the issue is a denial to create a per-user directory under `/var/run/user`. This is attempted by the systemd service `user-runtime-dir@` running `/usr/lib/systemd/systemd-user-runtime-dir start {USERNAME}`. This causes a number of downstream issues in things that depend on that per-user temp directory, such as systemd user services (e.g. `systemctl --user` fails)

```
Apr 03 19:03:41 xps systemd[1]: Starting User Runtime Directory /run/user/1000...
Apr 03 19:03:41 xps audit[16446]: AVC avc:  denied  { create } for  pid=16446 comm="systemd-user-ru" name="1000" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
Apr 03 19:03:41 xps systemd-user-runtime-dir[16446]: Failed to mount per-user tmpfs directory /run/user/1000: No such file or directory
Apr 03 19:03:41 xps systemd[1]: user-runtime-dir: Main process exited, code=exited, status=1/FAILURE
Apr 03 19:03:41 xps systemd[1]: user-runtime-dir: Failed with result 'exit-code'.
Apr 03 19:03:41 xps systemd[1]: Failed to start User Runtime Directory /run/user/1000.
```

Comment 11 Daniel Stiner 2022-04-28 02:51:20 UTC
While experimenting the issue went away for me.

I first set the relevant security domain to permissive using `# semanage permissive -a systemd_logind_t` and then rebooted. This of course fixed the issue, but strangely I didn't get any denials logged. So I removed that permissive rule using `# semanage permissive -d systemd_logind_t`. and then rebooted. At this point the issue was still fixed, no denials. My Silverblue version stayed at 35.20220426.0 throughout this. The only changes made were adding/removing the permissive override and install packages `policycoreutils-devel setroubleshoot setools-console` to aid in troubleshooting. SELinux was enabled and in enforcing mode at all times.

I then looked through my audit.log using `# /usr/bin/audit2why < /var/log/audit/audit.log`, all the denials I saw related to this issue looked like:

```
type=AVC msg=audit(1651103861.263:1022): avc:  denied  { create } for  pid=21268 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.
```

I think this implies the in-memory settings/policy somehow doesn't match the policy on disk? I'm wondering if messing with semanage forced some kind of reload that fixed the issue. I don't know enough about SELinux to say more, except I spent a fair amount of time looking at the relevant policy files and they all look correct. The source and target contexts are labeled correctly I think and there is an allow rule that *should* allow the directory to be created:
```
$ sesearch -A -s systemd_logind_t -t user_tmp_t -c dir
allow systemd_logind_t user_tmp_type:dir { add_name create getattr ioctl link lock mounton open read relabelfrom relabelto remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
```

The change Danut and I were looking at does seem related, but at this point I don't think the code change is wrong. Instead my best guess is that change was somehow not applied correctly in-memory due to some quirk of how Silverblue applies updates, a quirk that doesn't affect regular Fedora.

Given this issue no longer reproduces for me, I can't really go further in investigating what the actual root cause is, so I'll leave it here.

Comment 12 reto 2022-04-28 13:11:33 UTC
I was affected by this bug, and the steps described by 'Daniel Stiner' also solved the problem for me:

1. `# semanage permissive -a systemd_logind_t`
2. reboot
3. `# semanage permissive -d systemd_logind_t`
4. reboot

The audit.log entries look different for me though:

```
type=AVC msg=audit(1651151050.692:286): avc:  denied  { read } for  pid=3995 comm="systemd-user-ru" name="secrets" dev="tmpfs" ino=265 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=0

	Was caused by:
		Missing type enforcement (TE) allow rule.
```

Comment 13 Nikola Knazekova 2022-09-07 13:02:57 UTC
Is it actual?

I tried to reproduce it on Fedora 36 and no AVC.

Thanks

Nikola

Comment 14 slt 2022-09-08 12:11:09 UTC
I can't really say, I disabled selinux as suggested above and everything has been fine since :-O

Comment 15 Ben Cotton 2022-11-29 17:43:40 UTC
This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 16 Ben Cotton 2022-12-13 16:25:40 UTC
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13.

Fedora Linux 35 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.