Created attachment 1524177 [details] Relevant snippets from journalctl from an affected system Description of problem: On my silverblue system, /home is a symlink to /var/home which is a mount point. After upgrading container-selinux 2:2.79-1.git6f01752.fc29 -> 2:2.80-1.git1b655d9.fc29 I get numerous selinux denials regarding denying read access to the /home symlink. This actually causes my system to fail to boot properly, and puts me into a rescue mode. Version-Release number of selected component (if applicable): container-selinux 2:2.80-1.git1b655d9.fc29 How reproducible: 100% of the time. Setting selinux to permissive resolves the failed boot. Reverting back to container-selinux 2:2.79-1.git6f01752.fc29 also resolves the failed boot. Steps to Reproduce: 1. Install Fedora silverblue 29 on a disk large enough that it splits /var/home to a separate mount (my disk is 500GB -- on a VM with 30GB, it is not separate and no noticeable failures occur). fstab will reference /home rather than /var/home (it works via the symlink instead). 2. Upgrade to the latest version with rpm-ostree upgrade (should include container-selinux 2:2.80-1.git1b655d9.fc29) 3. Reboot, notice that you get dropped to an emergency shell. Actual results: System fails to boot properly. If you either modify fstab to mount directly to /var/home, or disable selinux, mounting works fine. Rolling back my update prior to the container-selinux update also works fine. Expected results: The system should boot properly, and there should be no selinux denials for the /home symlink. Additional info: Walters asked me to file this in the silverblue IRC.
Created attachment 1524179 [details] the rpm-ostree upgrade logs for when the issue started
This is my first bugzilla report -- my apologies if I have missed anything. I marked it as "high" severity since it can cause some systems to fail to boot. Please feel free to change this if you wish. I'm reasonably sure the exact package update causing the problem is container-selinux since it's the only selinux related one in the update that caused me to experience the issue -- but it is possible it might be due to one of the other packages as well. I've included a full list of the updated packages between an unaffected system and an affected system.
This is a labeling issue. restorecom /home should fix. If you are not allowed to run restorecon on /home then this is a silverblue bug.
(In reply to Daniel Walsh from comment #3) > This is a labeling issue. > > restorecom /home should fix. > > If you are not allowed to run restorecon on /home then this is a silverblue > bug. You are right, the label is wrong. However, restorecon /home doesn't actually work. You're allowed to run it, but it has no output. ``` [root@silverblue ~]# stat -c %C /home system_u:object_r:default_t:s0 [root@silverblue ~]# restorecon -v /home [root@silverblue ~]# stat -c %C /home system_u:object_r:default_t:s0 [root@silverblue ~]# ``` Stracing restorecon shows that it checks this context and then does nothing. ``` 1548694209.645917 lgetxattr("/home", "security.selinux", "system_u:object_r:default_t:s0", 255) = 31 <0.000516> 1548694209.646662 munmap(0x7f7c63471000, 782336) = 0 <0.000072> 1548694209.646757 munmap(0x7f7c63ac5000, 19508) = 0 <0.000011> 1548694209.646783 munmap(0x7f7c63698000, 563271) = 0 <0.000016> 1548694209.646829 exit_group(0) = ? 1548694209.646987 +++ exited with 0 +++ ``` It's a bit weird to me that booting to the prior upgrade it has the correct selinux context and post upgrade it does not.
I've tracked it down to a difference in an selinux context file: On an unaffected system: ==================== [root@silverblue ~]# grep 'home_root_t' /etc/selinux/*/contexts/files/* /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/home -d system_u:object_r:home_root_t:s0 /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/home -l system_u:object_r:home_root_t:s0 /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/home/\-inst -d system_u:object_r:home_root_t:s0 /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/home/home-inst -d system_u:object_r:home_root_t:s0 Binary file /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin matches ==================== On an affected system: =================== [root@silverblue ~]# grep 'home_root_t' /etc/selinux/*/contexts/files/* /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/var/home -d system_u:object_r:home_root_t:s0 /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/var/home -l system_u:object_r:home_root_t:s0 /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/var/home/\-inst -d system_u:object_r:home_root_t:s0 /etc/selinux/targeted/contexts/files/file_contexts.homedirs:/var/home/home-inst -d system_u:object_r:home_root_t:s0 Binary file /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin matches =================== So for some reason, /home became /var/home in /etc/selinux/targeted/contexts/files/file_contexts.homedirs (in all lines, not just the ones in my grep). This definitely was not something I touched. It's not clear to me why this change happened.
[ this is also https://github.com/projectatomic/rpm-ostree/issues/1742 ] > So for some reason, /home became /var/home in /etc/selinux/targeted/contexts/files/file_contexts.homedirs (in all lines, not just the ones in my grep). That is odd. The linked PR https://github.com/projectatomic/rpm-ostree/pull/1726 I don't *think* should have changed this. Can you verify what `ostree admin config-diff | grep selinux` shows? (You can find the default file specifically in /usr/etc/selinux/targeted/contexts/files/file_contexts.homedirs for this one) I'm guessing here your policy got recompiled locally, and somehow when that happened the policy compiler picked up on the symlink?
(In reply to Colin Walters from comment #6) > Can you verify what `ostree admin config-diff | grep selinux` shows? > > (You can find the default file specifically in > /usr/etc/selinux/targeted/contexts/files/file_contexts.homedirs for this one) The command you requested gives no output: ################# [root@silverblue ~]# ostree admin config-diff | grep -i selinux [root@silverblue ~]# ################# If I just look at the output (no grep), there's nothing that looks relevant. The file /usr/etc/selinux/targeted/contexts/files/file_contexts.homedirs has the exact same contents as /etc/selinux/targeted/contexts/files/file_contexts.homedirs -- both only refer to /var/home I don't know if this is helpful, but I do notice that on a small test vm running silverblue (with no mounts, everything being a part of /) these same files use /home rather than /var/home -- the selinux context of the /home symlink works perfectly fine. I'm not sure why these two different machines would have different contents in these files.
For now, a temporary workaround to manually set the correct context on the /home symlink: ############### [root@silverblue ~]# chcon -v -h -t home_root_t /home changing security context of '/home' ############### This prevents the selinux denials. This persists across reboots, but it won't survive a relable, and if you run restorecon on the file it will go back to being incorrect. It would be best if the policies generated the correct labels though, and I'd rather not manually modify them by hand. Not sure what the best way of correcting that would be.
Would this also work for my case (https://github.com/projectatomic/rpm-ostree/issues/1742)? I don’t want to brick my system and then not being able to revert that… :/
(In reply to Frank from comment #9) > Would this also work for my case > (https://github.com/projectatomic/rpm-ostree/issues/1742)? > > I don’t want to brick my system and then not being able to revert that… :/ From your logs, your problem is exactly the same as mine -- the /home symlink has the wrong selinux label. Any of the solutions I've mentioned in here (disabling selinux, modifying your fstab file to mount to /var/home directly, or using chcon on the /home symlink) should work for you. Of them, chcon is the best IMO. We'll still want to fix the problem that's causing the labeling to be wrong, but that's what this bugreport is for. You won't brick your system. Worst case scenario: if you screw up, you can just select your prior ostree boot in grub and be fine. Though, you should know you'll have to do your fix from the rescue mode you get dropped into, and then reboot after.
> I don't know if this is helpful, but I do notice that on a small test vm running silverblue (with no mounts, everything being a part of /) these same files use /home rather than /var/home -- the selinux context of the /home symlink works perfectly fine. Can you post your `rpm-ostree status -v`? On my Silverblue, I'm at 29.20190202.0, and looking at the upstream OSTree commit of that file: ``` $ ostree cat d56cc3afd54f38a7184917e9d4c9c4719aab7d285a65a92d4aa01fcb8f098922 /usr/etc/selinux/targeted/contexts/files/file_contexts.homedirs | grep home_root_t /home -d system_u:object_r:home_root_t:s0 /home -l system_u:object_r:home_root_t:s0 /home/\-inst -d system_u:object_r:home_root_t:s0 /home/home-inst -d system_u:object_r:home_root_t:s0 ``` So this suggests to me there might be a layered package that's somehow affecting your SELinux policy.
> So this suggests to me there might be a layered package that's somehow affecting your SELinux policy. I did some testing -- I have Ceph installed, which includes a "ceph-selinux" package. Doing "rpm-ostree install ceph-selinux" on my unaffected test VM is enough to trigger the incorrect context of /home -- the boot doesn't fail since /var/home doesn't have a separate filesystem, but some selinux denials do occur. I have ceph installed on my non-silverblue machines as well, and they do not experience this issue. Similarly, https://discussion.fedoraproject.org/t/silverblue-fails-to-boot-since-29-20190124-0/1037 (who is apparently affected by this as well), does not have ceph installed. Instead, he has snapd which comes with snapd-selinux. rpm-ostree install snapd-selinux is another way of replicating the same problem (I tested this as well). Neither my system nor this other person's system encountered this issue until recently, though I had ceph and he had snapd installed previously without encountering any issues. On both of our systems, ceph-selinux/snapd-selinux were not updated when the problems were first encountered (though container-selinux was) -- so something outside of them changed and causes these packages to become problematic.
OK, so I *think* I got to the bottom of this. I opened: https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14 and https://github.com/projectatomic/rpm-ostree/pull/1754 TL;DR, you can work around this for now with: ``` # sed -i '/^\/var\/home/d' /etc/selinux/targeted/contexts/files/file_contexts.subs_dist # restorecon -Rv /var/home ``` Your `subs_dist` will show as modified in `ostree admin config-diff`, but if we go with the PR above, it'll converge back to the default eventually.
That command sadly didn’t work for me. I’ll try to find the responsible RPM(s), what else can I do to help?
(I entered that both in my running session after `rpm-ostree upgrade`, and in that emergency console.)
(In reply to Jonathan Lebon from comment #13) > TL;DR, you can work around this for now with: > > ``` > # sed -i '/^\/var\/home/d' > /etc/selinux/targeted/contexts/files/file_contexts.subs_dist > # restorecon -Rv /var/home > ``` > > Your `subs_dist` will show as modified in `ostree admin config-diff`, but if > we go with the PR above, it'll converge back to the default eventually. Similarly to Frank (I assume he meant to reply to you), this doesn't seem to resolve the problem on my test vm. We'd need to be running restorecon on the /home symlink, not /var/home Anyhow, after modifying /etc/selinux/targeted/contexts/files/file_contexts.subs_dist as instructed and running restorecon (both on /home and /var/home), the label for /home remains incorrect for me. ``` [root@localhost ~]# stat -c +%C /home +system_u:object_r:home_root_t:s0 [root@localhost ~]# rpm-ostree --reboot install ceph-selinux ...(wait for reboot) [root@localhost ~]# stat -c +%C /home +system_u:object_r:default_t:s0 [root@localhost ~]# sed -i.$(date +%s).bak '/^\/var\/home/d' /etc/selinux/targeted/contexts/files/file_contexts.subs_dist [root@localhost ~]# diff -u /etc/selinux/targeted/contexts/files/file_contexts.subs_dist.1549526855.bak /etc/selinux/targeted/contexts/files/file_contexts.subs_dist --- /etc/selinux/targeted/contexts/files/file_contexts.subs_dist.1549526855.bak 2019-02-07 02:05:12.330482069 -0600 +++ /etc/selinux/targeted/contexts/files/file_contexts.subs_dist 2019-02-07 02:07:35.057853265 -0600 @@ -12,7 +12,6 @@ /var/lib/xguest/home /home /var/named/chroot/usr/lib64 /usr/lib /var/named/chroot/lib64 /usr/lib -/var/home /home /home-inst /home /home/home-inst /home /var/roothome /root [root@localhost ~]# restorecon -Rv /var/home /home Relabeled /var/home from system_u:object_r:default_t:s0 to system_u:object_r:home_root_t:s0 [root@localhost ~]# stat -c +%C /home +system_u:object_r:default_t:s0 ``` This seems to fix a potential issue I didn't notice regarding the /var/home directory -- however, the label on /var/home is actually not what's causing our problems. /home still is wrong afterwards though, which continues to cause selinux denials (including the failed boot if /var/home is a separate mount while fstab uses the /home symlink, which it does by default). Am I missing/overlooking something here?
So for me `snapd` caused this. I deleted that crap and everything works fine again.
> Am I missing/overlooking something here? No, you're absolutely correct. I've reproduced the full scenario here with a separate /home mount. Fr now, you can just keep pointing fstab to /var/home. That way you don't have to `chcon` on each new deployment. To summarize, The full fix for this requires: 1. RPM-OSTree to generate the full policy so that the homedirs rules in the compose also use /var/home (this is https://github.com/projectatomic/rpm-ostree/pull/1754). Otherwise, the rules would only get updated when e.g. layering a package that triggers policy regeneration, like ceph-selinux. 2. We drop the equivalency rule (this is https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14). Otherwise, /var/home/... is interpreted as /home/..., which is unmapped, so new files there get mislabeled. 3. We label the /home symlink itself as home_root_t (this is https://github.com/fedora-selinux/selinux-policy/pull/249). Otherwise, with the new rules, /home is left unmapped and is labeled default_t, giving systemd a hard time. This only affects systems that have a /home mount. One does wonder if it wouldn't be easier to just revert https://github.com/projectatomic/rpm-ostree/pull/1726. But a lot of software out there gets confused by `$HOME` and `realpath $HOME` being different. So it's probably worth it overall to get this sorted out for good. (Heck I've been hacking around this for a while myself just so I can get a ~ in my shell: https://github.com/jlebon/files/blob/747537af5ad542deceb481969856071d75df30b8/dotfiles/.bashrc#L157).
(In reply to Jonathan Lebon from comment #18) > Fr now, you can just keep pointing fstab to /var/home. That way you don't have to `chcon` on each new deployment. > 3. ... This only affects systems that have a /home mount. Actually, even after modifying the fstab file, there's still some other (minor?) selinux issues, and systems without a separate mount will be affected due to stuff trying to use the symlink. You just probably won't notice it because AFAIK none of these cause any visible effect to the end user. Some examples from the vm without a separate mount: ``` [root@localhost ~]# journalctl | grep 'avc: denied' Feb 06 04:13:12 localhost.localdomain audit[1267]: AVC avc: denied { getattr } for pid=1267 comm="(geoclue)" path="/run/systemd/unit-root/home" dev="dm-0" ino=761857 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=lnk_file permissive=0 Feb 06 04:13:14 localhost.localdomain audit[1097]: AVC avc: denied { read } for pid=1097 comm="gnome-shell" name="home" dev="dm-0" ino=761857 scontext=system_u:system_r:xdm_t:s0-s0:c0.c1023 tcontext=system_u:object_r:default_t:s0 tclass=lnk_file permissive=0 Feb 06 04:13:51 localhost.localdomain audit[1510]: AVC avc: denied { read } for pid=1510 comm="gdm-session-wor" name="home" dev="dm-0" ino=761857 scontext=system_u:system_r:xdm_t:s0-s0:c0.c1023 tcontext=system_u:object_r:default_t:s0 tclass=lnk_file permissive=0 Feb 06 04:13:56 localhost.localdomain audit[2170]: AVC avc: denied { getattr } for pid=2170 comm="(boltd)" path="/run/systemd/unit-root/home" dev="dm-0" ino=761857 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=lnk_file permissive=0 Feb 06 04:13:56 localhost.localdomain audit[2170]: AVC avc: denied { search } for pid=2170 comm="boltd" name=".cache" dev="dm-0" ino=919638 scontext=system_u:system_r:boltd_t:s0 tcontext=system_u:object_r:cache_home_t:s0 tclass=dir permissive=0 Feb 07 02:01:37 localhost.localdomain audit[1355]: AVC avc: denied { write } for pid=1355 comm="grub2-set-bootf" name="grubenv" dev="vda1" ino=303 scontext=system_u:system_r:xdm_t:s0-s0:c0.c1023 tcontext=system_u:object_r:boot_t:s0 tclass=file permissive=0 Feb 07 02:01:37 localhost.localdomain systemd[947]: selinux: avc: denied { start } for auid=n/a uid=42 gid=42 path="/usr/lib/systemd/user/dbus-daemon.service" cmdline="/usr/libexec/gnome-session-binary --autostart /usr/share/gdm/greeter/autostart" scontext=system_u:system_r:xdm_t:s0-s0:c0.c1023 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=service permissive=0 ``` (there's more, I've just included a few of them) So, for now, chcon is still something I'd advise anyone do. Speaking of which -- Frank, it's not really snapd's fault, and if you had used chcon like I suggested earlier you wouldn't have had to uninstall it. You can install it again, and then do chcon in the emergency shell after you reboot, and I think you'll be fine.
(In reply to Jonathan Lebon from comment #18) Also, thanks very much for looking into all of this and putting in the pull requests. I really appreciate it.
rpm-ostree-2019.2-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-fff22d1645
rpm-ostree-2019.2-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-fff22d1645
This is fixed in https://bodhi.fedoraproject.org/updates/FEDORA-2019-fff22d1645, which I've just pushed to stable. Though note that it'll require a new compose built by that OSTree, so just doing `rpm-ostree override replace` won't work. Make sure also to undo the modifications to /etc/selinux so your system picks up all the fixes from the new compose. (`ostree admin config-diff | grep selinux` shows you modified files).
rpm-ostree-2019.2-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.