Bug 1309075 - [f23] atomic updates-testing rebase selinux error
[f23] atomic updates-testing rebase selinux error
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: selinux-policy (Show other bugs)
23
Unspecified Unspecified
urgent Severity high
: ---
: ---
Assigned To: Lukas Vrabec
Fedora Extras Quality Assurance
: Reopened
: 1349045 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-16 13:51 EST by Dusty Mabe
Modified: 2016-07-15 06:26 EDT (History)
13 users (show)

See Also:
Fixed In Version: selinux-policy-3.13.1-158.21.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-13 20:23:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dusty Mabe 2016-02-16 13:51:45 EST
Description of problem:
I am getting an error when trying to rebase to the updates-testing
ostree. It is a similar fsetxattr error we have seen in the past:


# Go to known reproducer starting point:
-bash-4.3# rpm-ostree deploy 23.54 && reboot

# After reboot:
-bash-4.3# rpm-ostree rebase fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host

Receiving objects: 92% (3773/4069) 859.0 kB/s 99.6 MB         
error: fsetxattr: Invalid argument


From Giuseppe:
"""
I can reproduce the issue and the problem seems to be that the
/usr/libexec/rpm-ostreed file is not properly labeled.  It must have the
same label as as rpm-ostree:

bash-4.3# ls -Z -1 /usr/libexec/rpm-ostreed  /usr/bin/rpm-ostree 
system_u:object_r:install_exec_t:s0 /usr/bin/rpm-ostree
         system_u:object_r:bin_t:s0 /usr/libexec/rpm-ostreed
"""



Version-Release number of selected component (if applicable):
The version I am coming from:
2016-01-28 18:08:54     23.54     33166bf33b     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host


The version I am going to:
2016-02-15 04:56:09     23.68     6c739b2306     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host

How reproducible:


Steps to Reproduce:
1. rpm-ostree deploy 23.54 --reboot
2. rpm-ostree rebase fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host

Since the HEAD of the updates-testing branch will move over time this reproducer may not work in the future. At the time of writing the version at the head of the branch is:
2016-02-15 04:56:09     23.68     6c739b2306     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host


Additional info:
You can workaround by disabling selinux or using this workaround from giuseppe:

# systemctl stop rpm-ostreed
# cp /usr/libexec/rpm-ostreed /tmp
# chcon system_u:object_r:install_exec_t:s0 /tmp/rpm-ostreed
# /tmp/rpm-ostreed
Comment 1 Christoph Görn 2016-02-17 11:01:25 EST
This does also happen on atomic host upgrade:

$ sudo atomic host status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME            REFSPEC                                                
* 2016-01-26 18:16:33     23.53     22f0b303da     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host     
$ sudo atomic host upgrade
Updating from: fedora-atomic:fedora-atomic/f23/x86_64/docker-host

Receiving objects: 25% (1/4) 2.7 MB/s 8.1 MB
error: fsetxattr: Invalid argument

But I guess its the same root cause?!
Comment 3 Colin Walters 2016-02-18 08:47:11 EST
So unfortunately, now that the selinux-policy update made it into stable with new types on disk, in order to fix this on the server side, we would have to:

 - Temporarily revert the update
 - Create a new package which has the policy fix for rpm-ostreed
 - Use that new policy only for Atomic Host content, and freeze it - don't pull in any other policy updates
 - Let time pass for clients to upgrade
 - Eventually unfreeze and return to policy updates
 - At this point anyone who wants to upgrade could use `atomic host deploy` to pick a fixed version, or alternatively setenforce 0.
Comment 4 Miroslav Grepl 2016-02-18 09:51:23 EST
There is no update for this issue. Just committed in dist-git.
Comment 5 Miroslav Grepl 2016-02-18 10:02:17 EST

(In reply to Colin Walters from comment #3)
> So unfortunately, now that the selinux-policy update made it into stable
> with new types on disk, in order to fix this on the server side, we would
> have to:

And it happens very often if we add new labels to the policy.

> 
>  - Temporarily revert the update
>  - Create a new package which has the policy fix for rpm-ostreed
>  - Use that new policy only for Atomic Host content, and freeze it - don't
> pull in any other policy updates
>  - Let time pass for clients to upgrade
>  - Eventually unfreeze and return to policy updates
>  - At this point anyone who wants to upgrade could use `atomic host deploy`
> to pick a fixed version, or alternatively setenforce 0.
Comment 6 Colin Walters 2016-02-18 10:06:34 EST
I'll elaborate a bit here on how this bug occurred.

See https://bugzilla.redhat.com/show_bug.cgi?id=1061064 for some background.

Anything without specific policy runs as unconfined_t, where SELinux allows *almost* anything.  The key here being "almost".  One thing that's not allowed is for a process to write labels that are not known by the current policy.  In order to do that, the install_t domain (via install_exec_t) is used, which has "mac_admin".

The "unknown types" scenario can occur when we're downloading an updated tree commit with a new selinux policy that added types.  But we're still running with the old policy.

However, for an upgrade which doesn't add any new types, things work fine.
Which is how this passed our testing.

This bug will hopefully not reoccur, but just as a sanity check, it might help if rpm-ostree checked on startup that it has the mac_admin permission?
Comment 7 Miroslav Grepl 2016-02-19 05:24:20 EST
My understanding is /usr/libexec/rpm-ostreed is involved here and it runs as unconfined_service_t so we get this error because of new types in the policy and no "mac_admin" for unconfined_service_t.

It is the same what we had for /usr/bin/rpm-ostree running as unconfined_service_t without mac_admin.


Note: Another reason to have seatomic policy coming from Atomic where we could avoid it.
Comment 8 Colin Walters 2016-02-19 10:22:58 EST
Yeah.  We should have thought of this when creating the daemon, it's just a very subtle issue...things worked with local testing.

Agreed on seatomic policy.
Comment 9 Miroslav Grepl 2016-02-19 10:34:13 EST
(In reply to Colin Walters from comment #8)
> Yeah.  We should have thought of this when creating the daemon, it's just a
> very subtle issue...things worked with local testing.

Ok the questions is what is our solution here. We will "fail" in both cases - with/without install_t policy for /usr/libexec/rpm-ostreed.
Comment 10 Luke Macken 2016-03-26 22:01:39 EDT
I'm hitting this issue with the latest "stable" Fedora Atomic Host

https://download.fedoraproject.org/pub/alt/atomic/stable/Cloud_Atomic/x86_64/iso/Fedora-Cloud_Atomic-x86_64-23-20160308.iso
Comment 11 Micah Abbott 2016-03-28 14:41:20 EDT
Like Luke in the comment #10, I'm seeing this error again.

Going from 23.90 to 23.91.


# rpm-ostree status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME            REFSPEC                                                
* 2016-03-25 01:15:28     23.90     648b1538c6     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host     
  2016-03-07 03:45:47     23.79     1a68c7b9c0     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host     
-bash-4.3# rpm-ostree deploy 23.91
Resolving version '23.91'
error: fsetxattr: Invalid argument


Relevant messages from journal:

Mar 28 18:37:45 micah-fedora-ah-23-vm1.localdomain audit[1174]: AVC avc:  denied  { mac_admin } for  pid=1174 comm="pool" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
Mar 28 18:37:45 micah-fedora-ah-23-vm1.localdomain audit: SELINUX_ERR op=setxattr invalid_context="system_u:object_r:systemd_hwdb_exec_t:s0"
Mar 28 18:37:49 micah-fedora-ah-23-vm1.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'


The workaround in comment #1 gets around the issue again:

-bash-4.3# systemctl stop rpm-ostreed
-bash-4.3# cp /usr/libexec/rpm-ostreed /tmp
-bash-4.3# chcon system_u:object_r:install_exec_t:s0 /tmp/rpm-ostreed
-bash-4.3# /tmp/rpm-ostreed &
[1] 1208
-bash-4.3# rpm-ostree deploy 23.91
Resolving version '23.91'

15 metadata, 32 content objects fetched; 95955 KiB transferred in 4 seconds
Copying /etc changes: 24 modified, 0 removed, 49 added
Transaction complete; bootconfig swap: yes deployment count change: 0
Freed objects: 200.0 MB
Changed:
  selinux-policy 3.13.1-158.9.fc23 -> 3.13.1-158.11.fc23
  selinux-policy-targeted 3.13.1-158.9.fc23 -> 3.13.1-158.11.fc23
Run "systemctl reboot" to start a reboot
Comment 13 Colin Walters 2016-03-29 14:54:54 EDT
One thing we can do on the server side is (temporarily) revert selinux-policy back to what's in the last image, and add the patch to fix the label.
Comment 14 Miroslav Grepl 2016-04-01 03:40:58 EDT
Ok, see your #c3 as the reaction on the change. If you are ok with that, no problem to add it back. This change has never been shipped.
Comment 15 Colin Walters 2016-04-01 08:27:51 EDT
(In reply to Miroslav Grepl from comment #14)
> Ok, see your #c3 as the reaction on the change. If you are ok with that, no
> problem to add it back. This change has never been shipped.

Yes, we need the label change.
Comment 16 Fedora Update System 2016-04-11 07:03:36 EDT
selinux-policy-3.13.1-158.14.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3fade554b0
Comment 17 Fedora Update System 2016-04-13 05:28:32 EDT
selinux-policy-3.13.1-158.14.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-3fade554b0
Comment 18 Fedora Update System 2016-04-18 21:21:39 EDT
selinux-policy-3.13.1-158.14.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 19 Colin Walters 2016-06-22 11:09:17 EDT
This still doesn't work.

# rpm-ostree status
  TIMESTAMP (UTC)         VERSION     ID             OSNAME            REFSPEC                                           
* 2016-06-20 20:15:43     23.141      e3e06c5186     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host
  2016-06-07 01:00:21     23.131      aadcfe51ac     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host
# rpm -q selinux-policy
selinux-policy-3.13.1-158.15.fc23.noarch
# ps axZ|grep rpm-os
system_u:system_r:init_t:s0      1137 ?        Ssl    0:00 /usr/libexec/rpm-ostreed
Comment 20 Eric Paris 2016-06-22 11:13:47 EDT
somehow rpm-ostreed got back to init_t...
Comment 21 Daniel Walsh 2016-06-22 11:19:01 EDT
How is rpm-ostreed labeled
Comment 22 Micah Abbott 2016-06-22 11:25:53 EDT
# rpm-ostree status
  TIMESTAMP (UTC)         VERSION     ID             OSNAME            REFSPEC                                           
* 2016-06-20 20:15:43     23.141      e3e06c5186     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host
  2016-05-24 01:15:41     23.125      5668f0613e     fedora-atomic     fedora-atomic:fedora-atomic/f23/x86_64/docker-host

# rpm -q rpm-ostree
rpm-ostree-2016.1-3.fc23.x86_64

# ls -Z /usr/libexec/rpm-ostreed
system_u:object_r:install_exec_t:s0 /usr/libexec/rpm-ostreed
Comment 23 Micah Abbott 2016-06-22 11:30:49 EDT
*** Bug 1349045 has been marked as a duplicate of this bug. ***
Comment 24 Daniel Walsh 2016-06-22 11:59:33 EDT
That is the correct label.  Do we have a transition rule from init_t -> install_t?

sesearch -T -s init_t -t install_exec_t
Comment 25 Micah Abbott 2016-06-22 12:10:54 EDT
Doesn't look that way.

-bash-4.3# sesearch -T -s init_t -t install_exec_t


-bash-4.3#
Comment 26 Daniel Walsh 2016-06-22 12:18:28 EDT
In Rawhide Policy I have

sesearch -T -t install_exec_t
Found 2 semantic te rules:
   type_transition unconfined_t install_exec_t : process install_t; 
   type_transition init_t install_exec_t : process install_t; 

We need this back ported to F23/F24 I guess.
Comment 27 Lukas Vrabec 2016-06-27 10:43:52 EDT
backported.
Comment 28 Fedora Update System 2016-07-02 16:54:25 EDT
selinux-policy-3.13.1-158.21.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7bed6e7c72
Comment 29 Fedora Update System 2016-07-13 20:23:13 EDT
selinux-policy-3.13.1-158.21.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 30 Colin Walters 2016-07-15 06:26:37 EDT
To work around this, just:

setenforce 0

Then perform rpm-ostree upgrade/rebase etc.   Then reboot, and if you have this new policy, you should still be in enforcing after the reboot and won't need to setenforce 0 again.

Note You need to log in before you can comment on or make changes to this bug.