Bug 2044100 - Fedora Silvervlue kind of broken on recent updates: /run/user/1000: no such file or directory
Summary: Fedora Silvervlue kind of broken on recent updates: /run/user/1000: no such f...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: selinux-policy
Version: 35
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Zdenek Pytela
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-23 21:45 UTC by Samuel Le Thiec
Modified: 2022-04-28 13:11 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-24 09:15:49 UTC
Type: Bug


Attachments (Terms of Use)

Description Samuel Le Thiec 2022-01-23 21:45:30 UTC
Description of problem:
 
Many things are not working, for example toolbox:
 $ toolbox enter
 Error: failed to get the Podman version
 $ podman version
 Error: error creating tmpdir: mkdir /run/user/1000: permission denied
 $ podman -v 
 podman version 3.4.4

Then, here is what I found when trying to find the source of the problem:

 $ sudo systemctl list-units --failed 
   UNIT                          LOAD   ACTIVE SUB    DESCRIPTION                          
 ● user-runtime-dir@1000.service loaded failed failed User Runtime Directory /run/user/1000
 ● user-runtime-dir@42.service   loaded failed failed User Runtime Directory /run/user/42



 $ sudo systemctl status user-runtime-dir@1000.service
 × user-runtime-dir@1000.service - User Runtime Directory /run/user/1000
      Loaded: loaded (/usr/lib/systemd/system/user-runtime-dir@.service; static)
      Active: failed (Result: exit-code) since Sun 2022-01-23 22:04:31 CET; 18min ago
        Docs: man:user@.service(5)
     Process: 1660 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir start 1000 (code=exited, status=1/FAILURE)
    Main PID: 1660 (code=exited, status=1/FAILURE)
         CPU: 15ms
 
 janv. 23 22:04:31 nsfw systemd[1]: Starting User Runtime Directory /run/user/1000...
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: mmap: Permission denied
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: Failed to mount per-user tmpfs directory /run/user/1000: No such file or directory
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir@1000.service: Main process exited, code=exited, status=1/FAILURE
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir@1000.service: Failed with result 'exit-code'.
 janv. 23 22:04:31 nsfw systemd[1]: Failed to start User Runtime Directory /run/user/1000.


 journalctl -u user-runtime-dir@1000.service
 [....]
 -- Boot 6bde8538db544eb48470f956f7f15aa9 --           <= THIS WAS ON MY LAST WORKING SYSTEM: Silverblue 35.20220115.0
 janv. 23 21:17:41 nsfw systemd[1]: Starting User Runtime Directory /run/user/1000...
 janv. 23 21:17:41 nsfw systemd-user-runtime-dir[1600]: mmap: Permission denied
 janv. 23 21:17:41 nsfw systemd[1]: Finished User Runtime Directory /run/user/1000.
 janv. 23 22:03:41 nsfw systemd[1]: Stopping User Runtime Directory /run/user/1000...
 janv. 23 22:03:41 nsfw systemd-user-runtime-dir[19789]: mmap: Permission denied
 janv. 23 22:03:41 nsfw systemd[1]: user-runtime-dir@1000.service: Deactivated successfully.
 janv. 23 22:03:41 nsfw systemd[1]: Stopped User Runtime Directory /run/user/1000.
 -- Boot 96ddbd02ba1645fc9d4b4e20f9597171 --            <= THIS IS ON THE LATEST SYSTEM: Silverblue 35.20220123.0)
 janv. 23 22:04:31 nsfw systemd[1]: Starting User Runtime Directory /run/user/1000...
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: mmap: Permission denied
 janv. 23 22:04:31 nsfw systemd-user-runtime-dir[1660]: Failed to mount per-user tmpfs directory /run/user/1000: No such file or directory
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir@1000.service: Main process exited, code=exited, status=1/FAILURE
 janv. 23 22:04:31 nsfw systemd[1]: user-runtime-dir@1000.service: Failed with result 'exit-code'.
 janv. 23 22:04:31 nsfw systemd[1]: Failed to start User Runtime Directory /run/user/1000.


The output is similar with user-runtime-dir@42.service.



Version-Release number of selected component (if applicable):


Things got wrong a few days ago, my last working version is 35.20220115.0 (see below).

 $ rpm-ostree status
 State: idle
 Deployments:
 ● fedora:fedora/35/x86_64/silverblue
                   Version: 35.20220123.0 (2022-01-23T01:17:10Z)
                BaseCommit: d3f36f449b3e3a7de2a24187caeb3780407d71958cb11ed751e561d489766710
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
           LayeredPackages: aircrack-ng gnome-shell-extension-gsconnect gnome-tweaks gocryptfs gstreamer1-plugin-openh264 java-latest-openjdk kernel-tools lxc mozilla-openh264 neovim nmap p7zip reaver socat systemd-container tcpdump tmux
                            w3m wgctrl wireguard-tools

   fedora:fedora/35/x86_64/silverblue
                   Version: 35.20220115.0 (2022-01-15T00:49:28Z)
                BaseCommit: 4bb8301ce5639227589617b788943b1f95f93889407eec1ee63a762cb0a94c01
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
           LayeredPackages: aircrack-ng gnome-shell-extension-gsconnect gnome-tweaks gocryptfs gstreamer1-plugin-openh264 java-latest-openjdk kernel-tools lxc mozilla-openh264 neovim nmap p7zip reaver socat systemd-container tcpdump tmux
                            w3m wgctrl wireguard-tools


How reproducible:


Steps to Reproduce:
1. update system
2. reboot to new system
3. 

Actual results:

Many things are not working

Expected results:

everything works as with Silverblue 35.20220115.0

Additional info:

Here is an extract from journalctl with more context, and especially with some auditd messages:
-------------
 janv. 23 22:04:23 nsfw systemd[1]: Starting User Runtime Directory /run/user/42...
 janv. 23 22:04:23 nsfw systemd-logind[953]: New session c1 of user gdm.
 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { map } for  pid=1087 comm="systemd-user-ru" path="/etc/selinux/targeted/contexts/files/file_contexts.bin" dev="dm-0" ino=6103090 scontext=system_u:system_r:systemd_logind_t:s0 tconte>
 janv. 23 22:04:23 nsfw audit[1087]: SYSCALL arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=8dd7e a2=1 a3=2 items=0 ppid=1 pid=1087 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 com>
 janv. 23 22:04:23 nsfw audit: PROCTITLE proctitle=2F7573722F6C69622F73797374656D642F73797374656D642D757365722D72756E74696D652D646972007374617274003432
 janv. 23 22:04:23 nsfw systemd-user-runtime-dir[1087]: mmap: Permission denied
 janv. 23 22:04:23 nsfw systemd[1]: Started User resource assignment daemon.
 janv. 23 22:04:23 nsfw audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=uresourced comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
 janv. 23 22:04:23 nsfw uresourced[1086]: Setting resources on user.slice (MemoryMin: 262144000, MemoryLow: 0, CPUWeight: -, IOWeight: -)
 janv. 23 22:04:23 nsfw uresourced[1086]: Setting resources on user-42.slice (MemoryMin: 262144000, MemoryLow: 0, CPUWeight: 500, IOWeight: 500)
 janv. 23 22:04:23 nsfw uresourced[1086]: Setting resources on user@42.service (MemoryMin: 0, MemoryLow: 0, CPUWeight: 100, IOWeight: 100)
 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { create } for  pid=1087 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
 janv. 23 22:04:23 nsfw audit[1087]: SYSCALL arch=c000003e syscall=83 success=no exit=-13 a0=7ffe05a517f0 a1=1c0 a2=5642b97e4800 a3=0 items=0 ppid=1 pid=1087 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(non>
 janv. 23 22:04:23 nsfw audit: PROCTITLE proctitle=2F7573722F6C69622F73797374656D642F73797374656D642D757365722D72756E74696D652D646972007374617274003432
 janv. 23 22:04:23 nsfw systemd-user-runtime-dir[1087]: Failed to mount per-user tmpfs directory /run/user/42: No such file or directory
 janv. 23 22:04:23 nsfw systemd[1]: user-runtime-dir@42.service: Main process exited, code=exited, status=1/FAILURE
 janv. 23 22:04:23 nsfw systemd[1]: user-runtime-dir@42.service: Failed with result 'exit-code'.
 janv. 23 22:04:23 nsfw systemd[1]: Failed to start User Runtime Directory /run/user/42.
 janv. 23 22:04:23 nsfw systemd[1]: Dependency failed for User Manager for UID 42.
 janv. 23 22:04:23 nsfw systemd[1]: user@42.service: Job user@42.service/start failed with result 'dependency' .
 janv. 23 22:04:23 nsfw audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user-runtime-dir@42 comm="systemd"  exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=faile>
 ------------------



Please tell me what additional info you might need!

Any help appreciated,

Thanks in advance,

Samuel

Comment 1 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:15:49 UTC
This looks similar to #2020977, I hope the underlying cause is the same.

*** This bug has been marked as a duplicate of bug 2020977 ***

Comment 2 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:19:31 UTC
Or maybe not. Let's keep them separate for now.

Comment 3 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:43:38 UTC
 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { map } for  pid=1087 comm="systemd-user-ru" path="/etc/selinux/targeted/contexts/files/file_contexts.bin" dev="dm-0" ino=6103090 scontext=system_u:system_r:systemd_logind_t:s0 tconte>

systemd-user-runtime-dir loads the selinux config because it wants to assign appropriate labels
to the directories is tries to create (/run/user and /run/user/<UID>).

 janv. 23 22:04:23 nsfw audit[1087]: AVC avc:  denied  { create } for  pid=1087 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0

So this could be either /run/user or /run/user/<UID>.
The code emits a warning for the first one, but just continues for the second one, based
on the assumption that the mount will fail anyway. I'll submit a cosmetic patch to also emit a warning
to make this clearer.

I have no idea why this starting erroring.

Comment 4 Zbigniew Jędrzejewski-Szmek 2022-01-24 09:57:59 UTC
https://github.com/systemd/systemd/pull/22235 is the cosmetic patch for systemd-user-runtime-dir.

Comment 5 Samuel Le Thiec 2022-01-26 21:34:37 UTC
Hello again,

I am not really able to test this cosmetic patch by myself.

Is there something else I could do to understand what's going on and eventuall fix the problem?

Thank you in advance!

samuel

Comment 6 Danut Enachioiu 2022-01-29 23:14:47 UTC
Hello!

I think I'm having the same problem after upgrading selinux-policy on Silverblue from 35.8-1 to 35.9-1. It causes GDM to fail to start on Wayland (for some reason it starts fine after that on X11), meaning I can't use Wayland anymore.

Here are some notes from my investigation, by comparing `journalctl -b` before and after the selinux-policy change. After the change, I see

Jan 29 23:10:18 fedora systemd[1]: Starting User Runtime Directory /run/user/42...
Jan 29 23:10:18 fedora systemd[1]: Started Network Manager Script Dispatcher Service.
Jan 29 23:10:18 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd>
Jan 29 23:10:18 fedora systemd-user-runtime-dir[1093]: Regex version mismatch, expected: 10.39 2021-10-29 actual: 10.37 2021-05-26
Jan 29 23:10:18 fedora audit[1093]: AVC avc:  denied  { create } for  pid=1093 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_>
Jan 29 23:10:18 fedora systemd-user-runtime-dir[1093]: Failed to mount per-user tmpfs directory /run/user/42: No such file or directory
Jan 29 23:10:18 fedora systemd[1]: user-runtime-dir@42.service: Main process exited, code=exited, status=1/FAILURE
Jan 29 23:10:18 fedora systemd[1]: user-runtime-dir@42.service: Failed with result 'exit-code'.
Jan 29 23:10:18 fedora systemd[1]: Failed to start User Runtime Directory /run/user/42.
Jan 29 23:10:18 fedora systemd[1]: Dependency failed for User Manager for UID 42.
Jan 29 23:10:18 fedora systemd[1]: user@42.service: Job user@42.service/start failed with result 'dependency'.

Which then causes GDM to fail like this:

Jan 29 23:10:18 fedora gdm-launch-environment][1080]: pam_systemd(gdm-launch-environment:session): Failed to stat() runtime directory '/run/user/42': No such file or directory
Jan 29 23:10:18 fedora gdm-launch-environment][1080]: pam_systemd(gdm-launch-environment:session): Not setting $XDG_RUNTIME_DIR, as the directory is not in order.
[...]
Jan 29 23:10:19 fedora gnome-shell[1145]: Using public X11 display :1024, (using :1025 for managed services)
Jan 29 23:10:19 fedora gnome-shell[1145]: WL: error: XDG_RUNTIME_DIR not set in the environment
Jan 29 23:10:19 fedora audit[1145]: ANOM_ABEND auid=4294967295 uid=42 gid=42 ses=4294967295 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 pid=1145 comm="gnome-shell" exe="/usr/bin/gnome-shell" sig=>
Jan 29 23:10:19 fedora org.gnome.Shell.desktop[1145]: == Stack trace for context 0x55a82c2301f0 ==
Jan 29 23:10:19 fedora gnome-shell[1145]: Failed to create socket
Jan 29 23:10:19 fedora systemd[1]: Created slice Slice /system/systemd-coredump.
Jan 29 23:10:19 fedora systemd[1]: Started Process Core Dump (PID 1216/UID 0).
Jan 29 23:10:19 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-1216-0 comm="systemd" exe="/usr/lib/systemd>
Jan 29 23:10:19 fedora systemd-coredump[1217]: [🡕] Process 1145 (gnome-shell) of user 42 dumped core.

To be clear, none of these errors are present in the logs in version 35.8-1. It seems very likely to be caused by commit b3c7810107cfa023cab14b16eaf7de4f8401142c, which includes:

"- Change /run/user/[0-9]+ to /run/user/%{USERID} for proper labeling"

Unfortunately, I don't understand enough about selinux to be able to dig deeper but let me know if I can help with anything else.

Comment 7 Josh Belanich 2022-02-07 05:10:31 UTC
Hi all! I'm running into the same issue as Danut -- GDM fails, after which it loads into X11 and things like toolbox and running flatpaks appear broken. I'm also seeing the same boot log pattern as Danut. I was able to pinpoint the breaking change to Fedora Silverblue 35.20220118.0 (35.20220117.0 works fine). Let me know if there is anything information I can give that can help. Thanks!

Comment 8 Samuel Le Thiec 2022-03-16 10:18:43 UTC
Hello all,

I'm still kind of trapped in Silverblue 35.20220115.0 (yes from January!)

Did some of you find a way out of the issue? I would be pleased to learn about,

Thank you

Comment 9 Danut Enachioiu 2022-03-16 23:50:56 UTC
Well, I disabled SELinux enforcement. Instructions here: https://docs.fedoraproject.org/en-US/quick-docs/changing-selinux-states-and-modes/#selinux-changing-to-permissive-mode

I'd rather have up-to-date packages and no SELinux, probably makes sense even for security.

Comment 10 Daniel Stiner 2022-04-25 07:15:26 UTC
Seeing this also in Silverblue 35.20220423.0, does not repro on "regular" Fedora 35. I agree with Danut on the labeling change is the likely cause, link: https://github.com/fedora-selinux/selinux-policy/commit/925e0fcfc3747fa0d8bae5c6f266b4f2d754b6f5

I'm trying to confirm that commit is the issue by applying a patched selinux policy that reverts the commit, but it is taking time to understand how to do that in Silverblue. Also unsure if I should open a Github issue in the selinux-policy repo, or if this bug tracker is the better place to track this issue.

What I have figured out from my local logs is that the issue is a denial to create a per-user directory under `/var/run/user`. This is attempted by the systemd service `user-runtime-dir@` running `/usr/lib/systemd/systemd-user-runtime-dir start {USERNAME}`. This causes a number of downstream issues in things that depend on that per-user temp directory, such as systemd user services (e.g. `systemctl --user` fails)

```
Apr 03 19:03:41 xps systemd[1]: Starting User Runtime Directory /run/user/1000...
Apr 03 19:03:41 xps audit[16446]: AVC avc:  denied  { create } for  pid=16446 comm="systemd-user-ru" name="1000" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
Apr 03 19:03:41 xps systemd-user-runtime-dir[16446]: Failed to mount per-user tmpfs directory /run/user/1000: No such file or directory
Apr 03 19:03:41 xps systemd[1]: user-runtime-dir@1000.service: Main process exited, code=exited, status=1/FAILURE
Apr 03 19:03:41 xps systemd[1]: user-runtime-dir@1000.service: Failed with result 'exit-code'.
Apr 03 19:03:41 xps systemd[1]: Failed to start User Runtime Directory /run/user/1000.
```

Comment 11 Daniel Stiner 2022-04-28 02:51:20 UTC
While experimenting the issue went away for me.

I first set the relevant security domain to permissive using `# semanage permissive -a systemd_logind_t` and then rebooted. This of course fixed the issue, but strangely I didn't get any denials logged. So I removed that permissive rule using `# semanage permissive -d systemd_logind_t`. and then rebooted. At this point the issue was still fixed, no denials. My Silverblue version stayed at 35.20220426.0 throughout this. The only changes made were adding/removing the permissive override and install packages `policycoreutils-devel setroubleshoot setools-console` to aid in troubleshooting. SELinux was enabled and in enforcing mode at all times.

I then looked through my audit.log using `# /usr/bin/audit2why < /var/log/audit/audit.log`, all the denials I saw related to this issue looked like:

```
type=AVC msg=audit(1651103861.263:1022): avc:  denied  { create } for  pid=21268 comm="systemd-user-ru" name="42" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.
```

I think this implies the in-memory settings/policy somehow doesn't match the policy on disk? I'm wondering if messing with semanage forced some kind of reload that fixed the issue. I don't know enough about SELinux to say more, except I spent a fair amount of time looking at the relevant policy files and they all look correct. The source and target contexts are labeled correctly I think and there is an allow rule that *should* allow the directory to be created:
```
$ sesearch -A -s systemd_logind_t -t user_tmp_t -c dir
allow systemd_logind_t user_tmp_type:dir { add_name create getattr ioctl link lock mounton open read relabelfrom relabelto remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
```

The change Danut and I were looking at does seem related, but at this point I don't think the code change is wrong. Instead my best guess is that change was somehow not applied correctly in-memory due to some quirk of how Silverblue applies updates, a quirk that doesn't affect regular Fedora.

Given this issue no longer reproduces for me, I can't really go further in investigating what the actual root cause is, so I'll leave it here.

Comment 12 reto 2022-04-28 13:11:33 UTC
I was affected by this bug, and the steps described by 'Daniel Stiner' also solved the problem for me:

1. `# semanage permissive -a systemd_logind_t`
2. reboot
3. `# semanage permissive -d systemd_logind_t`
4. reboot

The audit.log entries look different for me though:

```
type=AVC msg=audit(1651151050.692:286): avc:  denied  { read } for  pid=3995 comm="systemd-user-ru" name="secrets" dev="tmpfs" ino=265 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=0

	Was caused by:
		Missing type enforcement (TE) allow rule.
```


Note You need to log in before you can comment on or make changes to this bug.