Description of problem: Newly-installed OCP 4.8 cluster has AVC warnings on worker nodes. # ausearch -m avc ---- time->Thu Apr 1 13:19:02 2021 type=PROCTITLE msg=audit(1617283142.102:54): proctitle=2F7573722F6C6962657865632F706C6174666F726D2D707974686F6E002D4573002F7573722F7362696E2F74756E6564002D2D6E6F2D64627573 type=SYSCALL msg=audit(1617283142.102:54): arch=c000003e syscall=1 success=yes exit=4 a0=7 a1=7f7a24015680 a2=4 a3=0 items=0 ppid=2553 pid=3473 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="tuned" exe="/usr/libexec/platform-python3.6" subj=system_u:system_r:spc_t:s0 key=(null) type=AVC msg=audit(1617283142.102:54): avc: granted { setsecparam } for pid=3473 comm="tuned" scontext=system_u:system_r:spc_t:s0 tcontext=system_u:object_r:security_t:s0 tclass=security Version-Release number of selected component (if applicable): platform-python-3.6.8-31.el8.x86_64 selinux-policy-targeted-3.14.3-54.el8_3.2.noarch container-selinux-2.155.0-1.module+el8.3.1+9857+68fb1526.noarch selinux-policy-3.14.3-54.el8_3.2.noarch openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch Server Version: 4.8.0-0.nightly-2021-04-01-072432 Kubernetes Version: v1.20.0+29a606d How reproducible: always Additional info:
"audit2allow -a" mentioned that it needs "allow systemd_hostnamed_t initrc_t:dbus send_msg" in the policy.
Since RHCOS is just consuming the SELinux policy from RHEL, going to move this to RHEL/selinux-policy for additional triage
The above-mentioned SELinux denial contains the word "granted" which means that nothing was denied.
Normally, the tuned service runs under tuned_t context and SELinux policy contains these rules: allow tuned_t security_t:security setsecparam; auditallow tuned_t security_t:security setsecparam;
(In reply to Qian Cai from comment #1) > "audit2allow -a" mentioned that it needs "allow systemd_hostnamed_t initrc_t:dbus send_msg" in the policy. Also note this is a problem different to the one in the bz description. Can you find out which process is running in initrc_t context? Was it started properly as a service? ps -eo pid,ppid,command,context | grep -e CONTEXT -e initrc_t
# ps -eo pid,ppid,command,context | grep -e CONTEXT -e initrc_t PID PPID COMMAND CONTEXT 1298 1 /bin/bash /usr/sbin/gcp-rou system_u:system_r:initrc_t:s0 419607 1298 sleep 5 system_u:system_r:initrc_t:s0 419731 419375 grep -e CONTEXT -e initrc_t system_u:system_r:spc_t:s0 # cat /usr/sbin/gcp-routes.sh #!/bin/bash # Update iptables rules based on google cloud load balancer VIPS # # This is needed because the GCP L3 load balancer doesn't actually do DNAT; # the destination IP address is still the VIP. Normally, there is an agent that # adds the vip to the local routing table, tricking the kernel in to thinking # it's a local IP and allowing processes doing an accept(0.0.0.0) to receive # the packets. Clever. # # We don't do that. Instead, we DNAT with conntrack. This is so we don't break # existing connections when the vip is removed. This is useful for draining # connections - take ourselves out of the vip, but service existing conns. # # Additionally, clients can write a file to /run/gcp-routes/$IP.down to force # a VIP as down. This is useful for graceful shutdown / upgrade. # # ~cdc~ ...
(In reply to Milos Malik from comment #3) > The above-mentioned SELinux denial contains the word "granted" which means > that nothing was denied. Is there a way to silent those "granted" messages to not "pollute" the audit logs?
(In reply to Qian Cai from comment #7) > (In reply to Milos Malik from comment #3) > > The above-mentioned SELinux denial contains the word "granted" which means > > that nothing was denied. > > Is there a way to silent those "granted" messages to not "pollute" the audit > logs? The purpose of auditallow rules is to notify the administrator as this is of particular importance, I am not aware of a way of silencing the rules. There are only a few of such rules in the policy. > # ps -eo pid,ppid,command,context | grep -e CONTEXT -e initrc_t > PID PPID COMMAND CONTEXT > 1298 1 /bin/bash /usr/sbin/gcp-rou system_u:system_r:initrc_t:s0 > 419607 1298 sleep 5 system_u:system_r:initrc_t:s0 > 419731 419375 grep -e CONTEXT -e initrc_t system_u:system_r:spc_t:s0 How is the script started and which package it is a part of? If it needs some additional permissions like direct communication with hostnamed, it needs to be SELinux confined.
(In reply to Zdenek Pytela from comment #8) > (In reply to Qian Cai from comment #7) > > (In reply to Milos Malik from comment #3) > > > The above-mentioned SELinux denial contains the word "granted" which means > > > that nothing was denied. > > > > Is there a way to silent those "granted" messages to not "pollute" the audit > > logs? > The purpose of auditallow rules is to notify the administrator as this is of > particular importance, I am not aware of a way of silencing the rules. There > are only a few of such rules in the policy. Okay, so admin is now aware of this. What should admins do? If they reviewed those and acknowledged that those are the same things to do. Then, Isn't it annoying that they will see other over and over again which could render selinux less useful? > > > # ps -eo pid,ppid,command,context | grep -e CONTEXT -e initrc_t > > PID PPID COMMAND CONTEXT > > 1298 1 /bin/bash /usr/sbin/gcp-rou system_u:system_r:initrc_t:s0 > > 419607 1298 sleep 5 system_u:system_r:initrc_t:s0 > > 419731 419375 grep -e CONTEXT -e initrc_t system_u:system_r:spc_t:s0 > How is the script started and which package it is a part of? If it needs > some additional permissions like direct communication with hostnamed, it > needs to be SELinux confined. I think that is something that need input from OCP/RHCOS side. Micah, is that something you can find out more about gcp-routes.sh for default OCP installation on GCP.
(In reply to Qian Cai from comment #9) > I think that is something that need input from OCP/RHCOS side. Micah, is > that something you can find out more about gcp-routes.sh for default OCP > installation on GCP. The `gcp-routes.sh` script is something that we include in RHCOS at build time; it's used to manage iptables rules related to the GCP load balancer VIPs. It's started as a simple systemd service. See the service definition and script at the links below: https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/lib/systemd/system/gcp-routes.service https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/sbin/gcp-routes.sh
> > The purpose of auditallow rules is to notify the administrator as this is of > > particular importance, I am not aware of a way of silencing the rules. There > > are only a few of such rules in the policy. > > Okay, so admin is now aware of this. What should admins do? If they reviewed those and acknowledged that those are the same > things to do. Then, Isn't it annoying that they will see other over and over again which could render selinux less useful? These type of audit records is not frequent (there is total of 5 such rule) so it should not bother excessively. The sole purpose of an auditallow record is not to be overlooked. > The `gcp-routes.sh` script is something that we include in RHCOS at build time; it's used to manage iptables rules related to the GCP load balancer VIPs. > > It's started as a simple systemd service. See the service definition and script at the links below: > > https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/lib/systemd/system/gcp-routes.service > https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/sbin/gcp-routes.sh There is a substantial difference between ExecStart=/bin/bash /usr/sbin/gcp-routes.sh start and ExecStart=/usr/sbin/gcp-routes.sh start from selinux point of view: in the former case the service starts in initrc_t domain while in the latter one it is unconfined_service_t which is allowed most of possible permission. I suggest make the script executable and start directly, not preceded by bash. If still additional permissions were required, the script would need to be confined.
(In reply to Zdenek Pytela from comment #11) > These type of audit records is not frequent (there is total of 5 such rule) > so it should not bother excessively. The sole purpose of an auditallow > record is not to be overlooked. As mentioned, this looks like plague every single OCP installations. Since we noticed the issues has been there for a few OCP releases, I'll leave it to OCP support organizations to escalate further more once we hear more from customers. > > > The `gcp-routes.sh` script is something that we include in RHCOS at build time; it's used to manage iptables rules related to the GCP load balancer VIPs. > > > > It's started as a simple systemd service. See the service definition and script at the links below: > > > > https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/lib/systemd/system/gcp-routes.service > > https://github.com/openshift/os/blob/a06951cb0b9ec63d79087ca257a2135601e027dd/overlay.d/05rhcos/usr/sbin/gcp-routes.sh > There is a substantial difference between > ExecStart=/bin/bash /usr/sbin/gcp-routes.sh start > and > ExecStart=/usr/sbin/gcp-routes.sh start > from selinux point of view: in the former case the service starts in > initrc_t domain while in the latter one it is unconfined_service_t which is > allowed most of possible permission. I suggest make the script executable > and start directly, not preceded by bash. If still additional permissions > were required, the script would need to be confined. Okay, so looks like we should flip the bug to OCP CoreOS again, so they could fix the script?
(In reply to Qian Cai from comment #12) > Okay, so looks like we should flip the bug to OCP CoreOS again, so they > could fix the script? Yup, we'll take it. I have a patch to the service file that I've tested on an RHCOS node in GCP: ``` $ systemctl cat gcp-routes # /usr/lib/systemd/system/gcp-routes.service [Unit] Description=Update GCP routes for forwarded IPs. ConditionKernelCommandLine=|ignition.platform.id=gce ConditionKernelCommandLine=|ignition.platform.id=gcp After=network.target [Service] Type=simple ExecStart=/usr/sbin/gcp-routes.sh start ExecStopPost=/usr/sbin/gcp-routes.sh cleanup User=root RestartSec=30 Restart=always [Install] WantedBy=multi-user.target $ systemctl status gcp-routes ● gcp-routes.service - Update GCP routes for forwarded IPs. Loaded: loaded (/usr/lib/systemd/system/gcp-routes.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-04-16 18:23:17 UTC; 4min 7s ago Main PID: 1293 (gcp-routes.sh) Tasks: 2 (limit: 22948) Memory: 1.1M CGroup: /system.slice/gcp-routes.service ├─1293 /bin/bash /usr/sbin/gcp-routes.sh start └─1651 sleep 5 Apr 16 18:23:17 miabbott-rhcos-48.c.openshift-rhcos-devel.internal systemd[1]: Started Update GCP routes for forwarded IPs.. Apr 16 18:23:18 miabbott-rhcos-48.c.openshift-rhcos-devel.internal gcp-routes.sh[1293]: done applying vip rules Apr 16 18:27:18 miabbott-rhcos-48.c.openshift-rhcos-devel.internal gcp-routes.sh[1293]: done applying vip rules $ ps -eo pid,ppid,command,context | grep -e CONTEXT -e initrc_t PID PPID COMMAND CONTEXT $ ps -eo pid,ppid,command,context | grep -e CONTEXT -e unconfined_service_t PID PPID COMMAND CONTEXT 1293 1 /bin/bash /usr/sbin/gcp-rou system_u:system_r:unconfined_service_t:s0 1657 1293 sleep 5 system_u:system_r:unconfined_service_t:s0 $ sudo ausearch -m avc <no matches> ```
Please also make sure that “audit2allow -a” does not report anything.
This landed in RHCOS as part of https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1255/ First included in RHCOS 48.84.202104221420-0 Should be part of the nightly payloads now
miabbott There seems to be two issues discussed in this BZ--one described in the title and one described in the notes. The issue in the title ( systemd_hostnamed_t initrc_t:dbus send_msg) which is shown in `audit2allow -a` output is caused by gcp-routes running as initrc_t. I am still seeing this in the `audit2allow` output in 4.8.0-0.nightly-2021-04-30-201824. Looking at the context of gcp-routes, it is now running as `unconfined_service_t, but still showing up in `audit2allow -a` output. Is there something I need to do to reset this message? See output below. The second issue, as described in the comments `ausearch -m avc`, is caused by the openshift-cluster-node-tuning-operator. A separate issue needs to be filed for this. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-04-30-201824 True False 28m Cluster version is 4.8.0-0.nightly-2021-04-30-201824 $ oc get nodes NAME STATUS ROLES AGE VERSION ci-ln-h710m42-f76d1-tqdqv-master-0 Ready master 53m v1.21.0-rc.0+aa1dc1f ci-ln-h710m42-f76d1-tqdqv-master-1 Ready master 53m v1.21.0-rc.0+aa1dc1f ci-ln-h710m42-f76d1-tqdqv-master-2 Ready master 53m v1.21.0-rc.0+aa1dc1f ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm Ready worker 45m v1.21.0-rc.0+aa1dc1f ci-ln-h710m42-f76d1-tqdqv-worker-c-v4jh6 Ready worker 45m v1.21.0-rc.0+aa1dc1f ci-ln-h710m42-f76d1-tqdqv-worker-d-fbvtd Ready worker 45m v1.21.0-rc.0+aa1dc1f $ oc debug node/ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm Starting pod/ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# systemctl cat gcp-routes # /usr/lib/systemd/system/gcp-routes.service [Unit] Description=Update GCP routes for forwarded IPs. ConditionKernelCommandLine=|ignition.platform.id=gce ConditionKernelCommandLine=|ignition.platform.id=gcp After=network.target [Service] Type=simple ExecStart=/usr/sbin/gcp-routes.sh start ExecStopPost=/usr/sbin/gcp-routes.sh cleanup User=root RestartSec=30 Restart=always [Install] WantedBy=multi-user.target sh-4.4# systemctl status gcp-routes ● gcp-routes.service - Update GCP routes for forwarded IPs. Loaded: loaded (/usr/lib/systemd/system/gcp-routes.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2021-05-03 16:43:42 UTC; 46min ago Main PID: 1242 (gcp-routes.sh) Tasks: 2 (limit: 95515) Memory: 1.1M CPU: 1.342s CGroup: /system.slice/gcp-routes.service ├─ 1242 /bin/bash /usr/sbin/gcp-routes.sh start └─82775 sleep 5 May 03 17:15:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: done applying vip rules May 03 17:19:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: Processing route for NIC 0/42:01:0a:00:80:04 for 35.229.42.125 May 03 17:19:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: ensuring rule for 35.229.42.125 May 03 17:19:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: done applying vip rules May 03 17:23:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: Processing route for NIC 0/42:01:0a:00:80:04 for 35.229.42.125 May 03 17:23:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: ensuring rule for 35.229.42.125 May 03 17:23:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: done applying vip rules May 03 17:27:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: Processing route for NIC 0/42:01:0a:00:80:04 for 35.229.42.125 May 03 17:27:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: ensuring rule for 35.229.42.125 May 03 17:27:43 ci-ln-h710m42-f76d1-tqdqv-worker-b-lsjmm gcp-routes.sh[1242]: done applying vip rules sh-4.4# ps -eo pid,ppid,command,context | grep -e CONTEXT -e initrc_t PID PPID COMMAND CONTEXT 84707 82256 grep -e CONTEXT -e initrc_t system_u:system_r:spc_t:s0 sh-4.4# ps -eo pid,ppid,command,context | grep -e CONTEXT -e unconfined_service_t PID PPID COMMAND CONTEXT 1242 1 /bin/bash /usr/sbin/gcp-rou system_u:system_r:unconfined_service_t:s0 1472 1 kubelet --config=/etc/kuber system_u:system_r:unconfined_service_t:s0 84890 1242 sleep 5 system_u:system_r:unconfined_service_t:s0 84892 82256 grep -e CONTEXT -e unconfin system_u:system_r:spc_t:s0 sh-4.4# sudo ausearch -m avc ---- time->Mon May 3 16:44:38 2021 type=PROCTITLE msg=audit(1620060278.578:181): proctitle=2F7573722F6C6962657865632F706C6174666F726D2D707974686F6E002D4573002F7573722F7362696E2F74756E6564002D2D6E6F2D64627573 type=SYSCALL msg=audit(1620060278.578:181): arch=c000003e syscall=1 success=yes exit=4 a0=7 a1=7fa854015b80 a2=4 a3=0 items=0 ppid=2566 pid=3416 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="tuned" exe="/usr/libexec/platform-python3.6" subj=system_u:system_r:spc_t:s0 key=(null) type=AVC msg=audit(1620060278.578:181): avc: granted { setsecparam } for pid=3416 comm="tuned" scontext=system_u:system_r:spc_t:s0 tcontext=system_u:object_r:security_t:s0 tclass=security sh-4.4# sh-4.4# rpm-ostree status State: idle Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42d24e77a34e22a89dcd467e227bedab3dfb81b5bfec4ea9686f617680bcb9d9 CustomOrigin: Managed by machine-config-operator Version: 48.84.202104301525-0 (2021-04-30T15:29:10Z) ostree://328a44d7c259ca1e3ed31ae020f09d922f460be998657a92f684f6760443077b Version: 48.83.202103221318-0 (2021-03-22T13:22:02Z) sh-4.4# audit2allow -a #============= systemd_hostnamed_t ============== allow systemd_hostnamed_t initrc_t:dbus send_msg;
Since the fix attached to this BZ was for the gcp-routes.sh script, I'll retitle this BZ to scope the fix to just that script/problem. I'll create a clone of this BZ for the original problem related to the output in `audit2allow`
I've opened another BZ for the original problem at BZ#1956836
Closing as verified based on https://bugzilla.redhat.com/show_bug.cgi?id=1945646#c19 @ 4.8.0-0.nightly-2021-04-30-201824
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438