Description of problem: We are unable to ssh into machine using keys passed to the instance via cloud provider metadata. The SSH key is getting created/set but it they do not work. Upstream Test: https://github.com/coreos/coreos-assembler/blob/main/mantle/kola/tests/ignition/empty.go The test runs and fails on AWS, GCP, Azure and Openstack. Upstream Issue: https://github.com/coreos/fedora-coreos-tracker/issues/1630 Transition of selinux-policy that's caused it is selinux-policy-40.5-1.fc40 → 40.6-1.fc40 Version-Release number of selected component (if applicable): selinux-policy-40.6-1.fc40 Steps to Reproduce: 1. Run an instance using empty ignition config 2. Use ssh authorized keys file provided and try to login. Actual results: The machine shuts down after writing ssh key files via Afterburn. From console.log ``` Ignition: ran on 2023/12/09 16:41:53 UTC (this boot) Ignition: user-provided config was applied Afterburn: wrote ssh authorized keys file for user: core host-192-168-40-46 login: ``` No console logs after the above Expected results: It should be able to log into the machine using the SSH keys.
Can you share some data, like audit log or journal entries? I can't see any change which should effect in such a case: $ git log --oneline v40.5..v40.6 048e9da4d (tag: v40.6) Update cifs interfaces to include fs_search_auto_mountpoints() 0f7fb7624 Allow sudodomain read var auth files 5c556764c Allow spamd_update_t read hardware state information eba81d0e0 Allow virtnetworkd domain transition on tc command execution e138fb784 Allow sendmail MTA connect to sendmail LDA 41870434d Allow auditd read all domains process state 063a3d832 Allow rsync read network sysctls d60aa9929 Add dhcpcd bpf capability to run bpf programs fd52c7866 Dontaudit systemd-hwdb dac_override capability c7fd83772 Allow systemd-sleep create efivarfs files
I couldn't really get anything from the console logs i.e. no difference to when the test passes. Also when the test fails, we get an empty journal.txt. It fails before any journal entry. Also there's another change to that version that you probably missed by mistake in the above list. I am not sure if that has any changes. * Tue Nov 28 2023 Zdenek Pytela <zpytela> - 40.6-1 - Add afterburn to modules-targeted-contrib.conf
(In reply to Gursewak Singh from comment #2) > I couldn't really get anything from the console logs i.e. no difference to > when the test passes. Also when the test fails, we get an empty journal.txt. > It fails before any journal entry. > Also there's another change to that version that you probably missed by > mistake in the above list. I am not sure if that has any changes. > * Tue Nov 28 2023 Zdenek Pytela <zpytela> - 40.6-1 > - Add afterburn to modules-targeted-contrib.conf You are right, this is from a different log. Please disable dontaudit rules and collect audit data: semodule -DB <reproduce> semodule -B preferably with full auditing enabled: https://fedoraproject.org/wiki/SELinux/Debugging#Enable_full_auditing
Build fcos-rawhide with latest selinux-policy-40.9-1.fc40.noarch, set `selinux=permissive` and run on azure, get avc denied logs, hope this can be helpful. $ grep avc journal.txt Jan 16 12:08:42.688000 audit[1805]: AVC avc: denied { create } for pid=1805 comm="afterburn" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:system_r:afterburn_t:s0 tclass=unix_dgram_socket permissive=1 Jan 16 12:08:42.688000 audit[1805]: AVC avc: denied { ioctl } for pid=1805 comm="afterburn" path="socket:[7332]" dev="sockfs" ino=7332 ioctlcmd=0x8933 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:system_r:afterburn_t:s0 tclass=unix_dgram_socket permissive=1 Jan 16 12:08:44.953000 audit[1804]: USER_AVC pid=1804 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc: denied { send_msg } for scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:system_r:NetworkManager_t:s0 tclass=dbus permissive=1 exe="/usr/bin/dbus-broker" sauid=81 hostname=? addr=? terminal=?' Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { search } for pid=1805 comm="afterburn" name="core" dev="sdb4" ino=15728768 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { write } for pid=1805 comm="afterburn" name="core" dev="sdb4" ino=15728768 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { add_name } for pid=1805 comm="afterburn" name=".ssh" scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { create } for pid=1805 comm="afterburn" name=".ssh" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { search } for pid=1805 comm="afterburn" name=".ssh" dev="sdb4" ino=12583046 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { write } for pid=1805 comm="afterburn" name=".ssh" dev="sdb4" ino=12583046 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { add_name } for pid=1805 comm="afterburn" name="authorized_keys.d" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { create } for pid=1805 comm="afterburn" name=".afterburn-DQOoXH" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { read write open } for pid=1805 comm="afterburn" path="/var/home/core/.ssh/authorized_keys.d/.afterburn-DQOoXH" dev="sdb4" ino=13631623 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:08:45.698000 audit[1805]: AVC avc: denied { remove_name } for pid=1805 comm="afterburn" name=".afterburn-DQOoXH" dev="sdb4" ino=13631623 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.698000 audit[1805]: AVC avc: denied { rename } for pid=1805 comm="afterburn" name=".afterburn-DQOoXH" dev="sdb4" ino=13631623 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:08:45.698000 audit[1805]: AVC avc: denied { create } for pid=1805 comm="afterburn" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:system_r:afterburn_t:s0 tclass=unix_dgram_socket permissive=1 Jan 16 12:08:45.698000 audit[1805]: AVC avc: denied { write } for pid=1805 comm="afterburn" name="socket" dev="tmpfs" ino=55 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:syslogd_var_run_t:s0 tclass=sock_file permissive=1 Jan 16 12:08:45.698000 audit[1805]: AVC avc: denied { sendto } for pid=1805 comm="afterburn" path="/systemd/journal/socket" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket permissive=1 Jan 16 12:08:45.698000 audit[1805]: AVC avc: denied { read } for pid=1805 comm="afterburn" name="authorized_keys.d" dev="sdb4" ino=13631622 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:08:45.698000 audit[1805]: AVC avc: denied { open } for pid=1805 comm="afterburn" path="/var/home/core/.ssh/authorized_keys.d" dev="sdb4" ino=13631622 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 Jan 16 12:09:00.909000 audit[2074]: AVC avc: denied { getattr } for pid=2074 comm="ssh-key-dir" path="/var/home/core/.ssh/authorized_keys.d/afterburn" dev="sdb4" ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:09:00.909000 audit[2074]: AVC avc: denied { read } for pid=2074 comm="ssh-key-dir" name="afterburn" dev="sdb4" ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:09:00.909000 audit[2074]: AVC avc: denied { open } for pid=2074 comm="ssh-key-dir" path="/var/home/core/.ssh/authorized_keys.d/afterburn" dev="sdb4" ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:09:02.768000 audit[2099]: AVC avc: denied { getattr } for pid=2099 comm="ssh-key-dir" path="/var/home/core/.ssh/authorized_keys.d/afterburn" dev="sdb4" ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:09:02.768000 audit[2099]: AVC avc: denied { read } for pid=2099 comm="ssh-key-dir" name="afterburn" dev="sdb4" ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 Jan 16 12:09:02.768000 audit[2099]: AVC avc: denied { open } for pid=2099 comm="ssh-key-dir" path="/var/home/core/.ssh/authorized_keys.d/afterburn" dev="sdb4" ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1
Didn't get much additional information after enabling full auditing on AWS cat console.txt | grep audit [ 9.543221] audit: type=1404 audit(1702052949.176:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295 enabled=1 old-enabled=1 lsm=selinux res=1 [ 9.683798] audit: type=1403 audit(1702052949.316:3): auid=4294967295 ses=4294967295 lsm=selinux res=1 [ 10.713453] audit: type=1400 audit(1702052950.346:4): avc: denied { create } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:init_t:s0 tclass=netlink_netfilter_socket permissive=0 [ 10.829400] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket. [ OK ] Listening on systemd-journald-audit.socket - Journal Audit Socket. [ 10.962072] systemd-journald[1377]: Collecting audit messages is enabled. [ 10.973647] audit: type=1130 audit(1702052950.606:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.000985] audit: type=1130 audit(1702052950.633:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=coreos-printk-quiet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.009099] audit: type=1130 audit(1702052950.641:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.018918] audit: type=1130 audit(1702052950.651:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@configfs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.020971] audit: type=1131 audit(1702052950.651:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@configfs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.027111] audit: type=1130 audit(1702052950.659:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@dm_multipath comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.029211] audit: type=1131 audit(1702052950.659:11): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@dm_multipath comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Starting auditd.service - Security Auditing Service... [ OK ] Started auditd.service - Security Auditing Service. Stopping auditd.service - Security Auditing Service... [ OK ] Stopped auditd.service - Security Auditing Service.
I've submitted a Fedora PR to address the issue: https://github.com/fedora-selinux/selinux-policy/pull/2000 There will be a copr repo soon as an effect of the automation. I did not address though all denials related to the /var/home/core path: Jan 16 12:09:00.909000 audit[2074]: AVC avc: denied { getattr } for pid=2074 comm="ssh-key-dir" path="/var/home/core/.ssh/authorized_keys.d/afterburn" dev="sdb4" ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 What is the path used for? If /var/home is a root for home directories, there should be an equivalency defined and applied: semanage fcontext -a -e /home /var/home restorecon -R -v /var/home
(In reply to Zdenek Pytela from comment #6) > I've submitted a Fedora PR to address the issue: > https://github.com/fedora-selinux/selinux-policy/pull/2000 > > There will be a copr repo soon as an effect of the automation. > I did not address though all denials related to the /var/home/core path: > > Jan 16 12:09:00.909000 audit[2074]: AVC avc: denied { getattr } for > pid=2074 comm="ssh-key-dir" > path="/var/home/core/.ssh/authorized_keys.d/afterburn" dev="sdb4" > ino=13631623 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 > tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 > > What is the path used for? > If /var/home is a root for home directories, there should be an equivalency > defined and applied: > > semanage fcontext -a -e /home /var/home > restorecon -R -v /var/home On OSTree systems /home/ is a symlink to /var/home/: ``` $ readlink -f /home/ /var/home $ ls -l /home lrwxrwxrwx. 3 root root 8 Mar 22 2023 /home -> var/home ``` This is the case for Silverblue/IoT/CoreOS and has been for many years. Here's the fcontext Equivalence from a system I see: ``` $ sudo semanage fcontext -l | tail -n 28 SELinux Distribution fcontext Equivalence /run = /var/run /run/lock = /var/lock /run/systemd/system = /usr/lib/systemd/system /run/systemd/generator = /usr/lib/systemd/system /run/systemd/generator.early = /usr/lib/systemd/system /run/systemd/generator.late = /usr/lib/systemd/system /lib = /usr/lib /lib64 = /usr/lib /usr/lib64 = /usr/lib /usr/local/lib64 = /usr/lib /usr/local/lib32 = /usr/lib /etc/systemd/system = /usr/lib/systemd/system /var/lib/xguest/home = /home /var/named/chroot/usr/lib64 = /usr/lib /var/named/chroot/lib64 = /usr/lib /var/named/chroot/var = /var /home-inst = /home /home/home-inst = /home /var/roothome = /root /sbin = /usr/sbin /sysroot/tmp = /tmp /var/usrlocal = /usr/local /var/mnt = /mnt /home = /var/home /usr/etc = /etc ``` Note that I had to package layer in policycoreutils-python-utils on FCOS so I could run `semanage`: ``` sudo rpm-ostree install policycoreutils-python-utils -y -A ```
Where is this defined? /home = /var/home It should be the other way round. With the proper /var/home equivalency in place, /var/home/core/.ssh/ should have ssh_home_t
(In reply to Zdenek Pytela from comment #8) > Where is this defined? > /home = /var/home It's injected by rpm-ostree at compose time. > It should be the other way round. It used to be that way a long time ago. We flipped it because rpm-ostree also sets `HOME=/var/home` in `/etc/default/useradd` and so the generated `file_contexts.homedirs` has `/var/home` (that code keys off of `/etc/default/useradd`). And the reason we changed `/etc/default/useradd` is because some software get confused by the fact that `$HOME` contains symlinks. > With the proper /var/home equivalency in place, /var/home/core/.ssh/ should > have ssh_home_t It does have ssh_home_t: ``` $ ls -ldZ /var/home/core/.ssh drwx------. 3 core core unconfined_u:object_r:ssh_home_t:s0 31 Jan 22 19:28 /var/home/core/.ssh ``` Lots of things would be broken if home dirs weren't labeled properly. :) For more contexts, see: https://github.com/coreos/rpm-ostree/pull/1726 https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14 https://github.com/coreos/rpm-ostree/pull/1754 See especially https://src.fedoraproject.org/rpms/selinux-policy/pull-request/14#comment-20484. TL;DR, the equivalency rule we inject doesn't actually do much nowadays IIRC. Given that `file_contexts.homedirs` already has the right `/var/home` paths and userspace tools like restorecon resolve symlinks before applying the right label, they'd look up `/var/home/...` and not `/home/...` even if a user typed `restorecon -R /home`. But confusingly, `matchpathcon` just does a straight lookup, so without that rule it prints the wrong answer (whether that result is actually used in practice is a different question).
Jonathan, thank you for the information and links. So this reported problem: Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { add_name } for pid=1805 comm="afterburn" name=".ssh" scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 means the directory does not exist yet and will be created with an incorrect label: Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { read write open } for pid=1805 comm="afterburn" path="/var/home/core/.ssh/authorized_keys.d/.afterburn-DQOoXH" dev="sdb4" ino=13631623 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 but restorecon would help and relabeled it correctly. Am I right? In that case we should add the transition in selinux-policy for afterburn.
(In reply to Zdenek Pytela from comment #10) > Jonathan, thank you for the information and links. > > So this reported problem: > Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { add_name } for > pid=1805 comm="afterburn" name=".ssh" > scontext=system_u:system_r:afterburn_t:s0 > tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir permissive=1 > > means the directory does not exist yet and will be created with an incorrect > label: > > Jan 16 12:08:45.688000 audit[1805]: AVC avc: denied { read write open } > for pid=1805 comm="afterburn" > path="/var/home/core/.ssh/authorized_keys.d/.afterburn-DQOoXH" dev="sdb4" > ino=13631623 scontext=system_u:system_r:afterburn_t:s0 > tcontext=system_u:object_r:user_home_dir_t:s0 tclass=file permissive=1 > > but restorecon would help and relabeled it correctly. Am I right? Yes, I verified this: ``` $ restorecon -Rv .ssh Relabeled /var/home/core/.ssh from system_u:object_r:user_home_dir_t:s0 to system_u:object_r:ssh_home_t:s0 Relabeled /var/home/core/.ssh/authorized_keys.d from system_u:object_r:user_home_dir_t:s0 to system_u:object_r:ssh_home_t:s0 Relabeled /var/home/core/.ssh/authorized_keys.d/afterburn from system_u:object_r:user_home_dir_t:s0 to system_u:object_r:ssh_home_t:s0 ``` > In that case we should add the transition in selinux-policy for afterburn. I'm confused. Do I understand correctly that the recent change that caused this was that Afterburn no longer had access to that file transition?
(In reply to Jonathan Lebon from comment #11) > > In that case we should add the transition in selinux-policy for afterburn. > > I'm confused. Do I understand correctly that the recent change that caused > this was that Afterburn no longer had access to that file transition? It actually has never been allowed. I just wanted to have it confirmed, I can add to the current allow rules in existing PR a transition which will make afterburn create .ssh dir with the correct context.
Yes, let's get it added. Thanks!
It will be a part of the next build, but please note it was not tested exactly as reported, feel free to get back to us.