Bug 1957840
Summary: | kubelet service fail to load EnvironmentFile due to SELinux denial | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Juan Manuel Parrilla Madrid <jparrill> | ||||
Component: | container-selinux | Assignee: | Jindrich Novy <jnovy> | ||||
Status: | CLOSED ERRATA | QA Contact: | Edward Shen <weshen> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | urgent | ||||||
Version: | 8.4 | CC: | aos-bugs, dornelas, dwalsh, ercohen, itsoiref, jligon, jnovy, keyoung, lvrabec, mavazque, miabbott, mmalik, mrussell, nstielau, pablo.iranzo, plautrba, rfreiman, rrubins, shardy, ssekidde, tsweeney, walters, ypu, zpytela | ||||
Target Milestone: | beta | Keywords: | Reopened, Triaged, ZStream | ||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | container-selinux-2.165.1-2.el8 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | |||||||
: | 1960769 1969998 (view as bug list) | Environment: | |||||
Last Closed: | 2021-11-09 17:37:47 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1958966, 1960769 | ||||||
Attachments: |
|
Description
Juan Manuel Parrilla Madrid
2021-05-06 15:22:53 UTC
Note that crio requires the same configuration but gets it using another script that preforms`sed --in-place` on the crio.conf ``` sh-4.4# audit2allow -a #============= init_t ============== allow init_t kubernetes_file_t:file read; ``` @dwalsh Do you think this should be allowed by default? When installing with openshift-installer (kubelet can read the file) this is the selinux context: ls -lahZ /etc/kubernetes/kubelet-pause-image-override -rw-r--r--. 1 root root system_u:object_r:container_file_t:s0 146 May 6 17:51 /etc/kubernetes/kubelet-pause-image-override When installing with assisted-installer: [root@openshift-master-1 core]# ls -lahZ /etc/kubernetes/kubelet-pause-image-override -rw-r--r--. 1 root root system_u:object_r:kubernetes_file_t:s0 147 May 6 13:38 /etc/kubernetes/kubelet-pause-image-override This is with the same OCP release image (quay.io/openshift-release-dev/ocp-release:4.8.0-fc.0-x86_64) I think this should be allowed by default in policy. But there's an easy workaround: just store the environment override outside of `/etc/kubernetes`, say in `/etc/kubelet.env` or something that will be `etc_t`. Since kubelet itself isn't reading this file, it doesn't need to be in `/etc/kubernetes`. Started sno installation with assisted-service. changed Env params in service to: EnvironmentFile=/etc/kubernetes/kubelet-env EnvironmentFile=/etc/kubernetes/kubelet-pause-image-override reloaded daemon and restarted kubelet. Kubelet started as expected and i don't see any errors. Verified context of env file : -rw-r--r--. 1 root root system_u:object_r:container_file_t:s0 146 May 9 14:34 /etc/kubernetes/kubelet-pause-image-override Seems to be related to this: https://bugzilla.redhat.com/show_bug.cgi?id=1904693 In my testing on the bootstrap node I made env file /etc/kubernetes/kubelet-pause-image-override mandatory in kubelet service by removing '-' EnvironmentFile=/etc/kubernetes/kubelet-pause-image-override If you try to restart the kubelet service after that, it fails because of the reported issue. The original SELinux label on the /etc/kubernetes/kubelet-pause-image-override on the bootstrap node was, -rw-r--r--. 1 root root system_u:object_r:kubernetes_file_t:s0 147 May 10 08:12 /etc/kubernetes/kubelet-pause-image-override But when I changed the labels to system_u:object_r:container_file_t:s0 the kubelet service restart successfully. I am not sure why the installer is setting those labels incorrectly. Moving this back to RHCOS, should this be sent to the installer team? As a data point I see the same with rhcos-48.84.202104271417-0-qemu.x86_64.qcow2.gz [core@localhost ~]$ ls -lahZ /etc/kubernetes/kubelet-pause-image-override -rw-r--r--. 1 root root system_u:object_r:kubernetes_file_t:s0 146 May 10 10:15 /etc/kubernetes/kubelet-pause-image-override Doing a restorecon switches back to container_file_t, so it may be a label issue in the image? [core@localhost ~]$ sudo restorecon -R /etc [core@localhost ~]$ ls -lahZ /etc/kubernetes/kubelet-pause-image-override -rw-r--r--. 1 root root system_u:object_r:container_file_t:s0 146 May 10 10:15 /etc/kubernetes/kubelet-pause-image-override I did not see this with rhcos-48.83.202103221318-0-qemu.x86_64.qcow2.gz Another note, the file is created by this installer script https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/kubelet-pause-image.sh.template#L13 However I couldn't reproduce the mis-labled file by re-running that script, so I'm unclear why the file initially has the kubernetes_file_t label. Ok, testing with the rhcos-4.8.0-fc.3-x86_64-live.x86_64 ISO, it happens in the same way, but with a 4.7 does not happen. this is the policy oer the /etc/kubernetes folder: [root@openshift-master-1 core]# ls -alhZ /etc | grep kubernetes drwxr-xr-x. 8 root root system_u:object_r:kubernetes_file_t:s0 191 May 10 10:36 kubernetes So that explains that the placed files over/etc/kubernetes has this context. Summary: - Running the OCP Assistend Installer on an RHCOS node based on RHEL 8.3 creates a file `/etc/kubernetes/kubelet-pause-image-override` labeled `system_u:object_r:container_file_t:s0` - Running the OCP Assistend Installer on an RHCOS node based on RHEL 8.4 creates the same file with the label `system_u:object_r:kubernetes_file_t:s0` - When the `/etc/kubernetes/kubelet-pause-image-override` file is labeled `system_u:object_r:kubernetes_file_t:s0`, the `kubelet` cannot access the file with an SELinux denial: type=AVC msg=audit(1620311565.546:147): avc: denied { read } for pid=1 comm="systemd" name="kubelet-pause-image-override" dev="loop0" ino=34417753 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kubernetes_file_t:s0 tclass=file permissive=0 - `audit2allow` provides the following rule: ``` sh-4.4# audit2allow -a #============= init_t ============== allow init_t kubernetes_file_t:file read; ``` - We think this transition should be allowed, so we are re-routing the BZ to the SELinux team for additional triage. Please provide a list of policy modules which are active on your machine: # semodule -lfull Please tell us more about the kubernetes_file_t type: # seinfo -tkubernetes_file_t -x SELinux policy packages shipped in RHEL-8.x do not define kubernetes_file_t type, even if the container-selinux package is installed: # seinfo -tkubernetes_file_t -x Types: 0 # semodule -lfull | grep container 200 container pp # Created attachment 1781757 [details]
semodule -l full
``` sh-4.4# seinfo -tkubernetes_file_t -x Types: 1 type kubernetes_file_t, file_type, non_auth_file_type, non_security_file_type; ``` See the attachment with the semodules, too. kubernetes_file_t is a part of container-selinux and I think this problem needs to be addressed there: # rpm -q container-selinux container-selinux-2.158.0-1.module+el8.4.0+10607+f4da7515.noarch # seinfo -tkubernetes_file_t -x Types: 1 type kubernetes_file_t, file_type, non_auth_file_type, non_security_file_type; https://github.com/containers/container-selinux/releases/tag/v2.162.0 Now we need to get this packaged up for RHEL. This should be in the current version of container-selinux. Fix for this was released via RHSA-2021:2371: https://errata.devel.redhat.com/advisory/76279/builds Will be in v2.165.1. (I had a typo in the previous version). This should show up in RHEL 8.5 I would be fine with backporting the entire container-selinux to RHEL8.4 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4154 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |