Description of problem: I am getting an error when trying to rebase to the updates-testing ostree. It is a similar fsetxattr error we have seen in the past: # Go to known reproducer starting point: -bash-4.3# rpm-ostree deploy 23.54 && reboot # After reboot: -bash-4.3# rpm-ostree rebase fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host Receiving objects: 92% (3773/4069) 859.0 kB/s 99.6 MB error: fsetxattr: Invalid argument From Giuseppe: """ I can reproduce the issue and the problem seems to be that the /usr/libexec/rpm-ostreed file is not properly labeled. It must have the same label as as rpm-ostree: bash-4.3# ls -Z -1 /usr/libexec/rpm-ostreed /usr/bin/rpm-ostree system_u:object_r:install_exec_t:s0 /usr/bin/rpm-ostree system_u:object_r:bin_t:s0 /usr/libexec/rpm-ostreed """ Version-Release number of selected component (if applicable): The version I am coming from: 2016-01-28 18:08:54 23.54 33166bf33b fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host The version I am going to: 2016-02-15 04:56:09 23.68 6c739b2306 fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host How reproducible: Steps to Reproduce: 1. rpm-ostree deploy 23.54 --reboot 2. rpm-ostree rebase fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host Since the HEAD of the updates-testing branch will move over time this reproducer may not work in the future. At the time of writing the version at the head of the branch is: 2016-02-15 04:56:09 23.68 6c739b2306 fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host Additional info: You can workaround by disabling selinux or using this workaround from giuseppe: # systemctl stop rpm-ostreed # cp /usr/libexec/rpm-ostreed /tmp # chcon system_u:object_r:install_exec_t:s0 /tmp/rpm-ostreed # /tmp/rpm-ostreed
This does also happen on atomic host upgrade: $ sudo atomic host status TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC * 2016-01-26 18:16:33 23.53 22f0b303da fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host $ sudo atomic host upgrade Updating from: fedora-atomic:fedora-atomic/f23/x86_64/docker-host Receiving objects: 25% (1/4) 2.7 MB/s 8.1 MB error: fsetxattr: Invalid argument But I guess its the same root cause?!
https://github.com/fedora-selinux/selinux-policy/commit/d70774079abc152ac233145a590e07bf3cbe77f7
So unfortunately, now that the selinux-policy update made it into stable with new types on disk, in order to fix this on the server side, we would have to: - Temporarily revert the update - Create a new package which has the policy fix for rpm-ostreed - Use that new policy only for Atomic Host content, and freeze it - don't pull in any other policy updates - Let time pass for clients to upgrade - Eventually unfreeze and return to policy updates - At this point anyone who wants to upgrade could use `atomic host deploy` to pick a fixed version, or alternatively setenforce 0.
There is no update for this issue. Just committed in dist-git.
(In reply to Colin Walters from comment #3) > So unfortunately, now that the selinux-policy update made it into stable > with new types on disk, in order to fix this on the server side, we would > have to: And it happens very often if we add new labels to the policy. > > - Temporarily revert the update > - Create a new package which has the policy fix for rpm-ostreed > - Use that new policy only for Atomic Host content, and freeze it - don't > pull in any other policy updates > - Let time pass for clients to upgrade > - Eventually unfreeze and return to policy updates > - At this point anyone who wants to upgrade could use `atomic host deploy` > to pick a fixed version, or alternatively setenforce 0.
I'll elaborate a bit here on how this bug occurred. See https://bugzilla.redhat.com/show_bug.cgi?id=1061064 for some background. Anything without specific policy runs as unconfined_t, where SELinux allows *almost* anything. The key here being "almost". One thing that's not allowed is for a process to write labels that are not known by the current policy. In order to do that, the install_t domain (via install_exec_t) is used, which has "mac_admin". The "unknown types" scenario can occur when we're downloading an updated tree commit with a new selinux policy that added types. But we're still running with the old policy. However, for an upgrade which doesn't add any new types, things work fine. Which is how this passed our testing. This bug will hopefully not reoccur, but just as a sanity check, it might help if rpm-ostree checked on startup that it has the mac_admin permission?
My understanding is /usr/libexec/rpm-ostreed is involved here and it runs as unconfined_service_t so we get this error because of new types in the policy and no "mac_admin" for unconfined_service_t. It is the same what we had for /usr/bin/rpm-ostree running as unconfined_service_t without mac_admin. Note: Another reason to have seatomic policy coming from Atomic where we could avoid it.
Yeah. We should have thought of this when creating the daemon, it's just a very subtle issue...things worked with local testing. Agreed on seatomic policy.
(In reply to Colin Walters from comment #8) > Yeah. We should have thought of this when creating the daemon, it's just a > very subtle issue...things worked with local testing. Ok the questions is what is our solution here. We will "fail" in both cases - with/without install_t policy for /usr/libexec/rpm-ostreed.
I'm hitting this issue with the latest "stable" Fedora Atomic Host https://download.fedoraproject.org/pub/alt/atomic/stable/Cloud_Atomic/x86_64/iso/Fedora-Cloud_Atomic-x86_64-23-20160308.iso
Like Luke in the comment #10, I'm seeing this error again. Going from 23.90 to 23.91. # rpm-ostree status TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC * 2016-03-25 01:15:28 23.90 648b1538c6 fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host 2016-03-07 03:45:47 23.79 1a68c7b9c0 fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host -bash-4.3# rpm-ostree deploy 23.91 Resolving version '23.91' error: fsetxattr: Invalid argument Relevant messages from journal: Mar 28 18:37:45 micah-fedora-ah-23-vm1.localdomain audit[1174]: AVC avc: denied { mac_admin } for pid=1174 comm="pool" capability=33 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0 Mar 28 18:37:45 micah-fedora-ah-23-vm1.localdomain audit: SELINUX_ERR op=setxattr invalid_context="system_u:object_r:systemd_hwdb_exec_t:s0" Mar 28 18:37:49 micah-fedora-ah-23-vm1.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' The workaround in comment #1 gets around the issue again: -bash-4.3# systemctl stop rpm-ostreed -bash-4.3# cp /usr/libexec/rpm-ostreed /tmp -bash-4.3# chcon system_u:object_r:install_exec_t:s0 /tmp/rpm-ostreed -bash-4.3# /tmp/rpm-ostreed & [1] 1208 -bash-4.3# rpm-ostree deploy 23.91 Resolving version '23.91' 15 metadata, 32 content objects fetched; 95955 KiB transferred in 4 seconds Copying /etc changes: 24 modified, 0 removed, 49 added Transaction complete; bootconfig swap: yes deployment count change: 0 Freed objects: 200.0 MB Changed: selinux-policy 3.13.1-158.9.fc23 -> 3.13.1-158.11.fc23 selinux-policy-targeted 3.13.1-158.9.fc23 -> 3.13.1-158.11.fc23 Run "systemctl reboot" to start a reboot
Looks like a patch was added in http://pkgs.fedoraproject.org/cgit/rpms/selinux-policy.git/commit/?h=f23&id=c582c0ebc168f89382178bda5867e18ca0f44dfe Then reverted in: http://pkgs.fedoraproject.org/cgit/rpms/selinux-policy.git/commit/?h=f23&id=f673707f7e90d4c9adf73e9fa3ad64d11f363aac But I find no mention of it being added again. Miroslav?
One thing we can do on the server side is (temporarily) revert selinux-policy back to what's in the last image, and add the patch to fix the label.
Ok, see your #c3 as the reaction on the change. If you are ok with that, no problem to add it back. This change has never been shipped.
(In reply to Miroslav Grepl from comment #14) > Ok, see your #c3 as the reaction on the change. If you are ok with that, no > problem to add it back. This change has never been shipped. Yes, we need the label change.
selinux-policy-3.13.1-158.14.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3fade554b0
selinux-policy-3.13.1-158.14.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-3fade554b0
selinux-policy-3.13.1-158.14.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
This still doesn't work. # rpm-ostree status TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC * 2016-06-20 20:15:43 23.141 e3e06c5186 fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host 2016-06-07 01:00:21 23.131 aadcfe51ac fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host # rpm -q selinux-policy selinux-policy-3.13.1-158.15.fc23.noarch # ps axZ|grep rpm-os system_u:system_r:init_t:s0 1137 ? Ssl 0:00 /usr/libexec/rpm-ostreed
somehow rpm-ostreed got back to init_t...
How is rpm-ostreed labeled
# rpm-ostree status TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC * 2016-06-20 20:15:43 23.141 e3e06c5186 fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host 2016-05-24 01:15:41 23.125 5668f0613e fedora-atomic fedora-atomic:fedora-atomic/f23/x86_64/docker-host # rpm -q rpm-ostree rpm-ostree-2016.1-3.fc23.x86_64 # ls -Z /usr/libexec/rpm-ostreed system_u:object_r:install_exec_t:s0 /usr/libexec/rpm-ostreed
*** Bug 1349045 has been marked as a duplicate of this bug. ***
That is the correct label. Do we have a transition rule from init_t -> install_t? sesearch -T -s init_t -t install_exec_t
Doesn't look that way. -bash-4.3# sesearch -T -s init_t -t install_exec_t -bash-4.3#
In Rawhide Policy I have sesearch -T -t install_exec_t Found 2 semantic te rules: type_transition unconfined_t install_exec_t : process install_t; type_transition init_t install_exec_t : process install_t; We need this back ported to F23/F24 I guess.
backported.
selinux-policy-3.13.1-158.21.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7bed6e7c72
selinux-policy-3.13.1-158.21.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
To work around this, just: setenforce 0 Then perform rpm-ostree upgrade/rebase etc. Then reboot, and if you have this new policy, you should still be in enforcing after the reboot and won't need to setenforce 0 again.