Bug 2057497

The upstream PR is now merged, switching back to Colin.

OK, to repeat from discussion in https://github.com/coreos/fedora-coreos-tracker/issues/701#issuecomment-1104000536

To test this bug, we need to:

Boot into RHCOS version X with selinux-policy version 1
Run a command (or via Ignition) that mutates the selinux policy, e.g. setsebool
Upgrade in place to a newer RHCOS version that has selinux-policy version 2
Verify that the machine is actually using the updated policy version 2 *and* has the setsebool changes.

The log messages should be in `journalctl -u ostree-finalize-staged`.

Probably the simplest way to test a selinux-policy change is by going from RHCOS based on e.g. RHEL 8.4 to one based on 8.5 or 8.6.

This needs Verified:Tested to proceed.

I think `semodule --rebuild-if-modules-changed` will always succeed, but I think it should output some information about when the policy has changed.
Maybe try with `semodule -v --rebuild-if-modules-changed` to see if it's actually doing something?

Yes, it will succeed regardless of whether policy linking (the slow part) was skipped or not. You can however detect it from the output if you add `-vv` to the command line. If the policy linking was skipped, you should see just 2 lines; otherwise there will be lots of additional output. You can use e.g. the presence of the 'Building policy binary' text to reliably discern the two cases.

OK this is really embarrassing, I think I completely missed that this change wasn't in 2022.2.
https://github.com/ostreedev/ostree/pull/2596
i.e. 2022.3 should have it.

(In reply to Colin Walters from comment #15)
> OK this is really embarrassing, I think I completely missed that this change
> wasn't in 2022.2.
> https://github.com/ostreedev/ostree/pull/2596
> i.e. 2022.3 should have it.

Any concerns including this version in RHCOS for OCP 4.11 so that we can still include SPO which in 4.11?

> Any concerns including this version in RHCOS for OCP 4.11 so that we can still include SPO which in 4.11?

Can't think of any, should be easy to tag back into 4.11.


OK so ostree-2022.2-4.el8 actually has the patches now, but I had to make a rhel8 branch upstream and cherry pick things.

Test with ostree-2022.2-4.el8.x86_64, result is the same with `2022.1-2.el8` and `2022.2-4.el8`

[core@cosa-devsh ~]$ rpm -q ostree policycoreutils
ostree-2022.2-4.el8.x86_64
policycoreutils-2.9-19.el8.x86_64

[core@cosa-devsh ~]$ sudo semodule -vvv --rebuild-if-modules-changed
Committing changes:
Ok: transaction number 0.

OK I did a quick sanity check with the same versions; in particular I used:

auditctl -a always,exit -F exe=/usr/sbin/semodule -F arch=b64 -S execve -k execsemodule

Then I saw in the audit logs that we were running that binary as part of shutdown.

I didn't yet get to the point of testing a policy update though.

Huijing were you able to try that full flow?

Colin, here are the steps to verify. Are the steps and result expected?

Steps:
- Boot 411.85.202205101201-0 qcow2, upgrade policycoreutils and ostree to fixed version and reboot
- setsebool, exec `sha256sum /etc/selinux/targeted/policy/policy.31`
- do the rebase to rhel8.6
- reboot and check 

1) setsebool is changed
2) check /etc/selinux/targeted/policy/policy.31 content using `sha256sum` is the same as before rebase

[core@cosa-devsh ~]$ sha256sum /etc/selinux/targeted/policy/policy.31
47f200d67ee9cecb9b8be42412109a5deb02a6627c138b018943b5bc6e9c22a1  /etc/selinux/targeted/policy/policy.31

Hi @walters Just tested the patch on OCP 4.11 RHCOS cluster. It does not seem the patch is working. I installed the patch manually using:

https://coreos.slack.com/archives/C999USB0D/p1652323577003319?thread_ts=1652302218.453639&cid=C999USB0D

Then I installed SPO. This triggers deploying selinuxd which in turn installs policies. Waited until the spod DS is ready.

sh-4.4# ostree admin config-diff | grep selinux/targeted/policy
M    selinux/targeted/policy/policy.31
sh-4.4# rpm -q ostree
ostree-2022.2-4.el8.x86_64

> sh-4.4# ostree admin config-diff | grep selinux/targeted/policy
> M    selinux/targeted/policy/policy.31
> sh-4.4# rpm -q ostree
> ostree-2022.2-4.el8.x86_64

Well yes, you're still expected to have a modified policy.  The original bug was that you'd have a modified version of the *old* policy.

An important comment on this was this one https://bugzilla.redhat.com/show_bug.cgi?id=2057497#c9

But OK let's step through the full test scenario.

# rpm-ostree override replace http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/policycoreutils/2.9/19.el8/x86_64/policycoreutils-2.9-19.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/libsemanage/2.9/8.el8/x86_64/libsemanage-2.9-8.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/policycoreutils/2.9/19.el8/noarch/python3-policycoreutils-2.9-19.el8.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/libsemanage/2.9/8.el8/x86_64/python3-libsemanage-2.9-8.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/policycoreutils/2.9/19.el8/noarch/policycoreutils-python-utils-2.9-19.el8.noarch.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/ostree/2022.2/4.el8/x86_64/ostree-{,libs-,grub2-}2022.2-4.el8.x86_64.rpm
...
# rpm-ostree ex apply-live --allow-replacement
...
Upgraded:
  libsemanage 2.9-6.el8 -> 2.9-8.el8
  ostree 2022.1-2.el8 -> 2022.2-4.el8
  ostree-grub2 2022.1-2.el8 -> 2022.2-4.el8
  ostree-libs 2022.1-2.el8 -> 2022.2-4.el8
  policycoreutils 2.9-16.el8 -> 2.9-19.el8
  policycoreutils-python-utils 2.9-16.el8 -> 2.9-19.el8
  python3-libsemanage 2.9-6.el8 -> 2.9-8.el8
  python3-policycoreutils 2.9-16.el8 -> 2.9-19.el8
Successfully updated running filesystem tree; some services may need to be restarted.
# ostree admin config-diff | grep selinux
#

We now have the new ostree+policycoreutils, on a system with a "golden/pristine" selinux policy.

Let's perform a change that requires policy recompilation:

# setsebool -P container_manage_cgroup on
# ostree admin config-diff | grep selinux
M    selinux/targeted/active/commit_num
M    selinux/targeted/active/policy.kern
M    selinux/targeted/policy/policy.31
A    selinux/targeted/active/ports.local
A    selinux/targeted/semanage.trans.LOCK
A    selinux/targeted/semanage.read.LOCK
# 

Now, what we want to do is perform an upgrade that includes a new `selinux-policy-targeted` package; what we should have is the new policy *and* the boolean set.

I think actually the easiest way to test this crudely is to identify compiled policies by sha256sum.
In the *buggy* i.e. old behavior, what will happen is simple: We carry forward our modified policy.
Hence, we start with "gold" policy with hash A, we run setsebool, we get policy hash B.  When upgrading, we still have B.

In the fixed behavior, we should see a new compiled policy with a new hash C in /etc/selinux/targeted/policy/policy.31.

(This would be maximally convenient to test if we pushed e.g. the rhel8.6 image in the new base image format so it could just be rebased on)

Hmm, could it be that --rebuild-if-modules-changed doesn't trigger for `setsebool` or `semanage fcontext`?

It *does* seem to work for me if I insert a local custom module, e.g.: (following https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_selinux/writing-a-custom-selinux-policy_using-selinux )

# pwd
/root
# cat > walterscustom.cil < EOF
(allow install_t cupsd_var_run_t (sock_file (read)))
EOF
# semodule -i walterscustom.cil
# rpm-ostree rebase (insert 8.6 image here)
# systemctl stop ostree-finalize-staged

Then looking at the new deployment directory's /etc/selinux, I see a new policy checksum.

Soo...I think the feature may work, but actually be a bit incomplete.  I guess the name of the CLI option --rebuild-if-modules-changed makes this in retrospect obvious?

But I think we actually want to retrigger on the simple condition of having the binary policy.31 differ between /usr/etc/ and /etc...which is currently an ostree-specific invention, but perhaps that's OK as a short term hack.

How hard would it be to have `--rebuild-if-changed` instead?

When rebase (before reboot), if run `semodule -vvv --rebuild-if-modules-changed` manually, this will trigger the policy rebuild

- Exec setsebool, check `content of policy.31`
[root@cosa-devsh ~]# sha256sum /etc/selinux/targeted/policy/policy.31
47f200d67ee9cecb9b8be42412109a5deb02a6627c138b018943b5bc6e9c22a1  /etc/selinux/targeted/policy/policy.31

- After rebase and before reboot, exec `semodule -vvv --rebuild-if-modules-changed`
[root@cosa-devsh ~]# semodule -vvv --rebuild-if-modules-changed
Committing changes:
Ok: transaction number 10.
[root@cosa-devsh ~]# sha256sum /etc/selinux/targeted/policy/policy.31
8c923cdc09544f477375a6304973bc32e68f1275e9c41d613db4d7a337fc1d09  /etc/selinux/targeted/policy/policy.31
[root@cosa-devsh ~]# sudo systemctl reboot

- After reboot, check content of policy.31
[core@cosa-devsh ~]$ sha256sum /etc/selinux/targeted/policy/policy.31
7c8a0f80bd08cb65c0e1ea5f8a8e6d15d260dd15bb7da66d180eddd87a797d1f  /etc/selinux/targeted/policy/policy.31
[core@cosa-devsh ~]$ sudo semodule -vvv --rebuild-if-modules-changed
Committing changes:
Ok: transaction number 0.

So from my selfish POV..if policy upgrades work after modules have been installed that's good enough for me to go ahead with SPO. I understand that if booleans don't work the feature as a whole might not be considered complete, but I wonder if it would be OK to track that as another issue and solve in the next release?

Hmm, but for some reason, upgrading to a newer selinux-policy doesn't work for me:

$ rpm-ostree override replace https://jhrozek.fedorapeople.org/sepol-test/selinux-policy-3.14.3-81.el8.noarch.rpm https://jhrozek.fedorapeople.org/sepol-test/selinux-policy-targeted-3.14.3-81.el8.noarch.rpm                          
Downloading https://jhrozek.fedorapeople.org/sepol-test/selinux-policy-3.14.3-81.el8.noarch.rpm...done
Downloading https://jhrozek.fedorapeople.org/sepol-test/selinux-policy-targeted-3.14.3-81.el8.noarch.rpm...done
Checking out tree e0746a6... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Relabeling... done
Applying 10 overrides
Processing packages... done
error: Checkout selinux-policy-targeted-3.14.3-81.el8.noarch: Hardlinking a5/8b8b3f84fa2d588c41ae5fa6615dfe387b262737198f5b2a9c5f24b0b23045.file to commit_num: File exists

I already have all the ostree, semanage and policycoreutils deps installed (I hope):

rpm -qa \*policycoreutils\* \*libsemanage\* \*ostree\* | sort
libsemanage-2.9-7.el8.x86_64
ostree-2022.2-4.el8.x86_64
ostree-grub2-2022.2-4.el8.x86_64
ostree-libs-2022.2-4.el8.x86_64
policycoreutils-2.9-19.el8.x86_64
policycoreutils-python-utils-2.9-19.el8.noarch
python3-libsemanage-2.9-7.el8.x86_64
python3-policycoreutils-2.9-19.el8.noarch
rpm-ostree-2022.2-2.el8.x86_64
rpm-ostree-libs-2022.2-2.el8.x86_64

So based on all this, I think I'd say let's go ahead and mark VERIFIED, get this queued up for 8.7 (and tagged into 4.11 for OCP).

> error: Checkout selinux-policy-targeted-3.14.3-81.el8.noarch: Hardlinking a5/8b8b3f84fa2d588c41ae5fa6615dfe387b262737198f5b2a9c5f24b0b23045.file to commit_num: File exists

That's https://github.com/coreos/rpm-ostree/issues/3421 - not really related to this except if it was fixed it'd make testing this even easier.

(In reply to Colin Walters from comment #30)
> So based on all this, I think I'd say let's go ahead and mark VERIFIED, get
> this queued up for 8.7 (and tagged into 4.11 for OCP).
> 

+1

> > error: Checkout selinux-policy-targeted-3.14.3-81.el8.noarch: Hardlinking a5/8b8b3f84fa2d588c41ae5fa6615dfe387b262737198f5b2a9c5f24b0b23045.file to commit_num: File exists
> 
> That's https://github.com/coreos/rpm-ostree/issues/3421 - not really related
> to this except if it was fixed it'd make testing this even easier.

just checking - the bug only affects the manual override, correct? Not rebasing to a new image?

> just checking - the bug only affects the manual override, correct? Not rebasing to a new image?

Yeah.  Another point of clarification: this bug is really about upgrades in general - every rpm-ostree change is actually operating on a "new image", even when you just `rpm-ostree upgrade`.

To say this another way, what we're doing here with `rpm-ostree rebase` is triggering the same code path that production OpenShift nodes will go through when upgraded via the MCO.

Change status to VERIFIED based on results comment #33, thanks Micah and Michael!

(In reply to Colin Walters from comment #26)
> Hmm, could it be that --rebuild-if-modules-changed doesn't trigger for
> `setsebool` or `semanage fcontext`?
> 
> It *does* seem to work for me if I insert a local custom module, e.g.:
> (following
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/
> html/using_selinux/writing-a-custom-selinux-policy_using-selinux )
> 
> # pwd
> /root
> # cat > walterscustom.cil < EOF
> (allow install_t cupsd_var_run_t (sock_file (read)))
> EOF
> # semodule -i walterscustom.cil
> # rpm-ostree rebase (insert 8.6 image here)
> # systemctl stop ostree-finalize-staged
> 
> Then looking at the new deployment directory's /etc/selinux, I see a new
> policy checksum.
> 
> Soo...I think the feature may work, but actually be a bit incomplete.  I
> guess the name of the CLI option --rebuild-if-modules-changed makes this in
> retrospect obvious?
> 
> But I think we actually want to retrigger on the simple condition of having
> the binary policy.31 differ between /usr/etc/ and /etc...which is currently
> an ostree-specific invention, but perhaps that's OK as a short term hack.
> 
> How hard would it be to have `--rebuild-if-changed` instead?

Indeed, when the changes are only in booleans (or other semanage customizations), then the policy is not updated. This is actually an oversight in the intended logic of the --rebuild-if-changed flag, as I intended it to always refresh the non-module stuff. Thankfully, it should be easy to fix.

I've opened a separate BZ 2089802 for it (the change is needed only in the libsemanage component).

(Eh, there's no reason to make these comments private so I'm going to stop myself at least)

I think in practice, getting this into 4.11 needs to be prioritized behind shipping 8.6 in general in 4.11, which is... https://issues.redhat.com/browse/COS-1510 I guess?  I'm not sure if there's a coherent card for this, the work is scattered across teams.

(In reply to Colin Walters from comment #42)
> (Eh, there's no reason to make these comments private so I'm going to stop
> myself at least)
> 
> I think in practice, getting this into 4.11 needs to be prioritized behind
> shipping 8.6 in general in 4.11, which is...
> https://issues.redhat.com/browse/COS-1510 I guess?  I'm not sure if there's
> a coherent card for this, the work is scattered across teams.

Would it make sense to create a card in the same epic? Who would be the assignee?

We will aim to ship these fixes in 4.12 for sure, which will involve cherry picking the ostree (and rpm-ostree) updates which are queued for 8.7 to 4.12.

I think we also want https://bugzilla.redhat.com/show_bug.cgi?id=2089802 for consistency, so we'll need to cherry pick the updated libsemanage.

At the current time, anyone who wants to set a SELinux boolean can do via a systemd unit (e.g. injected via MachineConfig) that does e.g.:

```
[Unit]
Before=kubelet.service

[Service]
ExecStart=/usr/sbin/setsebool container_manage_cgroup on
```

But you should *not* use the -P option in this unit; it will trigger the bug here.

In this model, the boolean will be dynamically applied during boot.

(In reply to Colin Walters from comment #44)
> We will aim to ship these fixes in 4.12 for sure, which will involve cherry
> picking the ostree (and rpm-ostree) updates which are queued for 8.7 to 4.12.
> 

Does it still hold that we can still cherry-pick the changes to 8.6 so that they make their way to 4.11? If yes, where (which tracker) should I file the request? RHBZ/Jira/somewhere else..?

We would appreciate a 8.6 backport for this one. You can track the backport in a BZ as you usually do for packages and we will follow it.

Sorry, this is the ostree bug. Will ask for the backport in the right bug.

https://issues.redhat.com/browse/ART-4475 has this being shipped early in OCP 4.12 right now.

We still need to do a similar thing for the 8.7 policycoreutils build though.

An alternative indeed is to try to ship these in 8.6.z; but...that has wider impact outside of OCP (e.g. Edge and policycoreutils is also installed on most RHEL 8 instances).

I'd lean towards just tagging for OCP for now.

We have https://bugzilla.redhat.com/show_bug.cgi?id=2049186 & https://bugzilla.redhat.com/show_bug.cgi?id=2049189 already in 8.6. Do we need https://bugzilla.redhat.com/show_bug.cgi?id=2089802 in 8.6 before we tag this ostree build or can this be done independently?

(In reply to Timothée Ravier from comment #50)
> We have https://bugzilla.redhat.com/show_bug.cgi?id=2049186 &
> https://bugzilla.redhat.com/show_bug.cgi?id=2049189 already in 8.6. Do we
> need https://bugzilla.redhat.com/show_bug.cgi?id=2089802 in 8.6 before we
> tag this ostree build or can this be done independently?

No, at least not for SPO.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ostree bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7557