2005018 – [8.4.0.z] kubelet service fail to load EnvironmentFile due to SELinux denial (Re-opened)

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2005018 - [8.4.0.z] kubelet service fail to load EnvironmentFile due to SELinux denial (Re-opened)

Summary: [8.4.0.z] kubelet service fail to load EnvironmentFile due to SELinux denial ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	container-selinux
Sub Component:
Version:	8.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	medium
Target Milestone:	beta
Target Release:	---
Assignee:	Jindrich Novy
QA Contact:	Edward Shen
Docs Contact:
URL:
Whiteboard:
Depends On:	1973418
Blocks:	1186913 1969998 1999245 2005053
TreeView+	depends on / blocked

Reported:	2021-09-16 14:55 UTC by Micah Abbott
Modified:	2021-11-09 20:01 UTC (History)
CC List:	12 users (show)
Fixed In Version:	container-selinux-2.167.0-1.el8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1973418
Clones:	2005053 (view as bug list)
Environment:
Last Closed:	2021-11-09 17:40:16 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-97484	0	None	None	None	2021-09-17 16:19:49 UTC
Red Hat Product Errata	RHSA-2021:4154	0	None	None	None	2021-11-09 17:40:54 UTC

Description Micah Abbott 2021-09-16 14:55:48 UTC

Cloned original BZ to facilitate an async release in support of OCP 4.9

I've chopped out a bunch of the original BZ content to focus just on the problem and reproducer.

OCP 4.9 already has `container-selinux-2.167.0-1.module+el8.5.0+12397+bf23b712` tagged for inclusion it it's GA release.


+++ This bug was initially created as a clone of Bug #1973418 +++

Description of problem:
Basically re-opening BZ1960769 (already closed) which does not seem to have fixed the problem in RHCOS.


Version-Release number of selected component (if applicable):
container-selinux-2.162.0-1.module+el8.4.0+11311+9da8acfb.noarch

How reproducible:
Always

Steps to Reproduce:
1. In RHCOS, create a environment file in /etc/kubernetes 
2. reference that EnvironmentFile in the kubelet.service file
3. systemctl daemon-reload
4. restart kubelet

Actual results:
avc denial reading the EnvironmentFile.

Expected results:
No denials, service restarts with no issue.

Additional info:
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-16-190035   True        False         3h46m   Cluster version is 4.8.0-0.nightly-2021-06-16-190035

$ oc get nodes
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-134-236.us-west-2.compute.internal   Ready    master   4h12m   v1.21.0-rc.0+120883f
ip-10-0-150-206.us-west-2.compute.internal   Ready    worker   4h7m    v1.21.0-rc.0+120883f
ip-10-0-164-27.us-west-2.compute.internal    Ready    master   4h12m   v1.21.0-rc.0+120883f
ip-10-0-183-87.us-west-2.compute.internal    Ready    worker   4h5m    v1.21.0-rc.0+120883f
ip-10-0-210-154.us-west-2.compute.internal   Ready    master   4h13m   v1.21.0-rc.0+120883f
ip-10-0-222-34.us-west-2.compute.internal    Ready    worker   4h6m    v1.21.0-rc.0+120883f

$ oc debug node/ip-10-0-183-87.us-west-2.compute.internal 
Starting pod/ip-10-0-183-87us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cd /etc/kubernetes/
sh-4.4# ls
ca.crt	    cni		kubelet-ca.crt	 kubelet.conf  static-pod-resources
cloud.conf  kubeconfig	kubelet-plugins  manifests
sh-4.4# echo TEST=TEST > test-env
sh-4.4# ls -laZ
total 40
drwxr-xr-x.  6 root root system_u:object_r:kubernetes_file_t:s0  193 Jun 17 17:59 .
drwxr-xr-x. 96 root root system_u:object_r:etc_t:s0             8192 Jun 17 17:25 ..
-rw-r--r--.  1 root root system_u:object_r:kubernetes_file_t:s0 1123 Jun 17 17:25 ca.crt
-rw-r--r--.  1 root root system_u:object_r:kubernetes_file_t:s0    0 Jun 17 17:25 cloud.conf
drwxr-xr-x.  3 root root system_u:object_r:kubernetes_file_t:s0   19 Jun 17 13:48 cni
-rw-r--r--.  1 root root system_u:object_r:kubernetes_file_t:s0 6050 Jun 17 13:46 kubeconfig
-rw-r--r--.  1 root root system_u:object_r:kubernetes_file_t:s0 5875 Jun 17 17:25 kubelet-ca.crt
drwxr-xr-x.  3 root root system_u:object_r:kubernetes_file_t:s0   20 Jun 17 13:48 kubelet-plugins
-rw-r--r--.  1 root root system_u:object_r:kubernetes_file_t:s0 1076 Jun 17 17:25 kubelet.conf
drwxr-xr-x.  2 root root system_u:object_r:kubernetes_file_t:s0    6 Jun 17 13:49 manifests
drwxr-xr-x.  3 root root system_u:object_r:kubernetes_file_t:s0   24 Jun 17 13:48 static-pod-resources
-rw-r--r--.  1 root root system_u:object_r:kubernetes_file_t:s0   10 Jun 17 17:59 test-env
sh-4.4# vi /etc/systemd/system/kubelet.service
sh-4.4# audit2allow -a
sh-4.4# cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service network-online.target
Requires=crio.service kubelet-auto-node-size.service
After=network-online.target crio.service kubelet-auto-node-size.service
After=ostree-finalize-staged.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env
EnvironmentFile=/etc/kubernetes/test-env
EnvironmentFile=/etc/node-sizing.env

ExecStart=/usr/bin/hyperkube \
    kubelet \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/worker,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \
      --minimum-container-ttl-duration=6m0s \
      --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
      --cloud-provider=aws \
       \
      --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fa0f2cad0e8d907a10bf91b2fe234659495a694235a9e2ef7015eb450ce9f1ba \
      --system-reserved=cpu=${SYSTEM_RESERVED_CPU},memory=${SYSTEM_RESERVED_MEMORY} \
      --v=${KUBELET_LOG_LEVEL}

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
sh-4.4# systemctl daemon-reload
sh-4.4# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-mco-default-madv.conf, 20-logging.conf
   Active: active (running) since Thu 2021-06-17 17:25:38 UTC; 35min ago
 Main PID: 1400 (kubelet)
    Tasks: 16 (limit: 48468)
   Memory: 201.0M
      CPU: 2min 27.198s
   CGroup: /system.slice/kubelet.service
           └─1400 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/ku>

Jun 17 18:01:04 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:04.109653    1400 scope.go:111] "RemoveContai>
Jun 17 18:01:04 ip-10-0-183-87 hyperkube[1400]: E0617 18:01:04.109979    1400 remote_runtime.go:334] "Con>
Jun 17 18:01:04 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:04.110010    1400 pod_container_deletor.go:52>
Jun 17 18:01:04 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:04.198000    1400 reconciler.go:196] "operati>
Jun 17 18:01:04 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:04.203632    1400 operation_generator.go:829]>
Jun 17 18:01:04 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:04.298840    1400 reconciler.go:319] "Volume >
Jun 17 18:01:05 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:05.110755    1400 kubelet.go:1960] "SyncLoop >
Jun 17 18:01:05 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:05.116113    1400 kubelet.go:1954] "SyncLoop >
Jun 17 18:01:05 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:05.116180    1400 kubelet.go:2153] "Failed to>
Jun 17 18:01:05 ip-10-0-183-87 hyperkube[1400]: I0617 18:01:05.116181    1400 reflector.go:225] Stopping >
sh-4.4# systemctl restart kubelet

Removing debug pod ...


== restarting the kubelet removes the debug node and I need to SSH back in ==

$ ./ssh.sh ip-10-0-183-87.us-west-2.compute.internal
Warning: Permanently added 'ip-10-0-183-87.us-west-2.compute.internal' (ECDSA) to the list of known hosts.

[root@ip-10-0-183-87 kubernetes]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-mco-default-madv.conf, 20-logging.conf
   Active: inactive (dead) (Result: resources) since Thu 2021-06-17 18:01:34 UTC; 3ms ago
  Process: 1400 ExecStart=/usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap>
 Main PID: 1400 (code=exited, status=0/SUCCESS)
      CPU: 0

Jun 17 18:01:24 ip-10-0-183-87 systemd[1]: kubelet.service: Failed to load environment files: Permissi>
Jun 17 18:01:24 ip-10-0-183-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: Permission>
Jun 17 18:01:24 ip-10-0-183-87 systemd[1]: kubelet.service: Failed with result 'resources'.
Jun 17 18:01:24 ip-10-0-183-87 systemd[1]: Failed to start Kubernetes Kubelet.
Jun 17 18:01:34 ip-10-0-183-87 systemd[1]: kubelet.service: Service RestartSec=10s expired, scheduling>
Jun 17 18:01:34 ip-10-0-183-87 systemd[1]: kubelet.service: Scheduled restart job, restart counter is >
Jun 17 18:01:34 ip-10-0-183-87 systemd[1]: Stopped Kubernetes Kubelet.
Jun 17 18:01:34 ip-10-0-183-87 systemd[1]: kubelet.service: Consumed 0 CPU time
lines 1-17/17 (END)

[root@ip-10-0-183-87 kubernetes]# audit2allow -a


#============= init_t ==============
allow init_t kubernetes_file_t:file read;
[root@ip-10-0-183-87 kubernetes]# grep avc /var/log/audit/audit.log | tail -1 - | audit2why
type=AVC msg=audit(1623953918.958:1790): avc:  denied  { read } for  pid=1 comm="systemd" name="test-env" dev="nvme0n1p4" ino=92295647 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:kubernetes_file_t:s0 tclass=file permissive=0

	Was caused by:
		Missing type enforcement (TE) allow rule.

		You can use audit2allow to generate a loadable module to allow this access.

[root@ip-10-0-183-87 kubernetes]# rpm -q container-selinux
container-selinux-2.162.0-1.module+el8.4.0+11311+9da8acfb.noarch
[root@ip-10-0-183-87 kubernetes]# rpm-ostree status
State: idle
Deployments:
● pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a9af00365ac9b38d479a6e33bb54fbc3f1150d7903873b9a6f1230a2a7577622
              CustomOrigin: Managed by machine-config-operator
                   Version: 48.84.202106141119-0 (2021-06-14T11:22:36Z)

  ostree://457db8ff03dda5b3ce1a8e242fd91ddbe6a82f838d1b0047c3d4aeaf6c53f572
                   Version: 48.84.202106091622-0 (2021-06-09T16:25:42Z)

--- Additional comment from Micah Abbott on 2021-06-17 19:37:14 UTC ---

Simple reproducer:

- create a simple file at /etc/kubernetes/test

```
# cat /etc/kubernetes/test
TEST=foobar
```

- create a simple systemd service with an EnvironmentFile

```
$ systemctl cat echo.service 
# /etc/systemd/system/echo.service
[Unit]
Description=An echo unit
[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/kubernetes/test
ExecStart=/usr/bin/echo ${PAUSE}
[Install]
WantedBy=multi-user.target
```

- systemd daemon-reload && systemctl start echo.service

```
$ sudo systemctl daemon-reload && sudo systemctl start echo.service
Job for echo.service failed because of unavailable resources or another system error.
See "systemctl status echo.service" and "journalctl -xe" for details.

$ systemctl status echo.service 
● echo.service - An echo unit
   Loaded: loaded (/etc/systemd/system/echo.service; enabled; vendor preset: enabled)
   Active: failed (Result: resources)

Jun 17 19:29:43 localhost systemd[1]: echo.service: Failed to load environment files: Permission denied
Jun 17 19:29:43 localhost systemd[1]: echo.service: Failed to run 'start' task: Permission denied
Jun 17 19:29:43 localhost systemd[1]: echo.service: Failed with result 'resources'.
Jun 17 19:29:43 localhost systemd[1]: Failed to start An echo unit.
```

```
$ sudo audit2allow -a


#============= init_t ==============
allow init_t kubernetes_file_t:file read;
```

Comment 11 errata-xmlrpc 2021-11-09 17:40:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4154

Note You need to log in before you can comment on or make changes to this bug.