Bug 1727937 - Got "denied" in audit log on compute and ceph machines
Summary: Got "denied" in audit log on compute and ceph machines
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: 15.0 (Stein)
Assignee: Cédric Jeanneret
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-08 15:30 UTC by Sasha Smolyak
Modified: 2021-03-30 01:41 UTC (History)
7 users (show)

Fixed In Version: openstack-selinux-0.8.20-0.20190904140454.936ea4f.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:23:52 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers container-selinux issues 73 0 'None' closed AVCs on rhel-8 2021-02-04 00:19:24 UTC
Github redhat-openstack openstack-selinux pull 42 0 'None' closed Allow logrotate to access and write within container_file_t 2021-02-04 00:19:25 UTC
Launchpad 1836000 0 None None None 2019-07-10 06:19:46 UTC
OpenStack gerrit 669987 0 'None' MERGED Allow logrotate to access container_file_t files 2021-02-04 00:19:24 UTC
Red Hat Bugzilla 1728196 0 medium CLOSED Got 'denied' in audit.log for plymouth, modprobe and tmpwatch on compute node 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1728246 0 unspecified CLOSED Allow systemd-user-runtime-dir to list /run content 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:24:11 UTC

Internal Links: 1728246

Description Sasha Smolyak 2019-07-08 15:30:26 UTC
Description of problem:
When checking audit.log on compute node, got several "denied" lines:
1. Setup with 2 overclouds, without additional ceph, 1 controller, 1 compute on each overcloud, rebooted compute node:

[heat-admin@overcloud2-novacompute-0 ~]$ sudo cat /var/log/audit/audit.log | grep denied
type=AVC msg=audit(1562468761.649:33825): avc:  denied  { dac_override } for  pid=558318 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0
type=AVC msg=audit(1562468761.649:33826): avc:  denied  { dac_override } for  pid=558318 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0
type=AVC msg=audit(1562468761.650:33827): avc:  denied  { dac_override } for  pid=558318 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0
type=AVC msg=audit(1562468761.672:33828): avc:  denied  { read } for  pid=558347 comm="logrotate" name="qemu" dev="vda2" ino=1012139 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562468761.673:33829): avc:  denied  { read } for  pid=558347 comm="logrotate" name="openvswitch" dev="vda2" ino=1012142 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562557081.933:45121): avc:  denied  { dac_override } for  pid=758799 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0
type=AVC msg=audit(1562557081.933:45122): avc:  denied  { dac_override } for  pid=758799 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0
type=AVC msg=audit(1562557081.934:45123): avc:  denied  { dac_override } for  pid=758799 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0
type=AVC msg=audit(1562557081.934:45124): avc:  denied  { dac_override } for  pid=758799 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0
type=AVC msg=audit(1562557081.948:45125): avc:  denied  { read } for  pid=758807 comm="logrotate" name="qemu" dev="vda2" ino=1012139 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562557081.949:45126): avc:  denied  { read } for  pid=758807 comm="logrotate" name="openvswitch" dev="vda2" ino=1012142 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562587479.922:49211): avc:  denied  { connectto } for  pid=828775 comm="plymouth" path=002F6F72672F667265656465736B746F702F706C796D6F75746864 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:plymouthd_t:s0 tclass=unix_stream_socket permissive=0
type=AVC msg=audit(1562587479.939:49213): avc:  denied  { connectto } for  pid=828808 comm="plymouth" path=002F6F72672F667265656465736B746F702F706C796D6F75746864 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:plymouthd_t:s0 tclass=unix_stream_socket permissive=0
type=AVC msg=audit(1562587793.241:215): avc:  denied  { read } for  pid=8208 comm="systemd-user-ru" name="libpod" dev="tmpfs" ino=73106 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562587793.241:216): avc:  denied  { read } for  pid=8208 comm="systemd-user-ru" name="overlay-containers" dev="tmpfs" ino=73105 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562587793.241:217): avc:  denied  { read } for  pid=8208 comm="systemd-user-ru" name="overlay-layers" dev="tmpfs" ino=73104 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0


2. Setup 3 controllers, 2 computes, 3 cephs, without any reboots:
[heat-admin@compute-1 ~]$ sudo cat /var/log/audit/audit.log | grep denied
type=AVC msg=audit(1562508743.188:39): avc:  denied  { execute_no_trans } for  pid=1219 comm="modprobe" path="/usr/bin/bash" dev="vda2" ino=4215568 scontext=system_u:system_r:openvswitch_load_module_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1
type=AVC msg=audit(1562508745.961:81): avc:  denied  { sendto } for  pid=865 comm="chronyd" path="/run/chrony/chronyc.2707.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:cloud_init_t:s0 tclass=unix_dgram_socket permissive=1
type=AVC msg=audit(1562508756.013:83): avc:  denied  { sendto } for  pid=865 comm="chronyd" path="/run/chrony/chronyc.2719.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:cloud_init_t:s0 tclass=unix_dgram_socket permissive=1
type=AVC msg=audit(1562513521.954:5767): avc:  denied  { read } for  pid=54302 comm="logrotate" name="qemu" dev="vda2" ino=1012139 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562513521.955:5768): avc:  denied  { read } for  pid=54302 comm="logrotate" name="openvswitch" dev="vda2" ino=1012142 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562556122.068:12339): avc:  denied  { read } for  pid=178739 comm="logrotate" name="qemu" dev="vda2" ino=1012139 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562556122.069:12340): avc:  denied  { read } for  pid=178739 comm="logrotate" name="openvswitch" dev="vda2" ino=1012142 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0


[heat-admin@ceph-0 ~]$ sudo cat /var/log/audit/audit.log | grep denied
type=AVC msg=audit(1562508548.724:40): avc:  denied  { execute_no_trans } for  pid=1212 comm="modprobe" path="/usr/bin/bash" dev="vda2" ino=4215568 scontext=system_u:system_r:openvswitch_load_module_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1
type=AVC msg=audit(1562508551.990:82): avc:  denied  { sendto } for  pid=877 comm="chronyd" path="/run/chrony/chronyc.2725.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:cloud_init_t:s0 tclass=unix_dgram_socket permissive=1
type=AVC msg=audit(1562508552.402:83): avc:  denied  { sendto } for  pid=877 comm="chronyd" path="/run/chrony/chronyc.2763.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:cloud_init_t:s0 tclass=unix_dgram_socket permissive=1


Version-Release number of selected component (if applicable):
tmpwatch-2.11-14.el8.x86_64
logrotate-3.14.0-3.el8.x86_64
plymouth-scripts-0.9.3-12.el8.x86_64
plymouth-0.9.3-12.el8.x86_64
plymouth-core-libs-0.9.3-12.el8.x86_64
chrony-3.3-3.el8.x86_64
RHOS_TRUNK-15.0-RHEL-8-20190701.n.0

How reproducible:
100%

Steps to Reproduce:
1. Deploy
2. Enter compute or ceph node
3. sudo cat /var/log/audit/audit.log | grep denied

Actual results:
Get "denied" from various containers

Expected results:
No "denied" in log

Additional info:

Comment 1 Cédric Jeanneret 2019-07-09 05:47:28 UTC
So, I think we can already push some of the AVC to other projects:

type=AVC msg=audit(1562468761.649:33825): avc:  denied  { dac_override } for  pid=558318 comm="tmpwatch" capability=1  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0

That's for tmpwatch - although the issue itself will be sorted out by a selinux ruleset


type=AVC msg=audit(1562587479.922:49211): avc:  denied  { connectto } for  pid=828775 comm="plymouth" path=002F6F72672F667265656465736B746F702F706C796D6F75746864 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:plymouthd_t:s0 tclass=unix_stream_socket permissive=0

That's for plymouth - although, here again, the issue itself will probably be sorted out by a selinux ruleset. On might ask "why plymouth on a server" though...... "Graphical Boot Animation and Logger"


type=AVC msg=audit(1562587793.241:215): avc:  denied  { read } for  pid=8208 comm="systemd-user-ru" name="libpod" dev="tmpfs" ino=73106 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562587793.241:216): avc:  denied  { read } for  pid=8208 comm="systemd-user-ru" name="overlay-containers" dev="tmpfs" ino=73105 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1562587793.241:217): avc:  denied  { read } for  pid=8208 comm="systemd-user-ru" name="overlay-layers" dev="tmpfs" ino=73104 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0

That's probably for podman/libpod/runc, and, if legit accesses, will lead to a new rulset in container-selinux

=======

type=AVC msg=audit(1562508548.724:40): avc:  denied  { execute_no_trans } for  pid=1212 comm="modprobe" path="/usr/bin/bash" dev="vda2" ino=4215568 scontext=system_u:system_r:openvswitch_load_module_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1

There, no idea for now, need some investigations.


type=AVC msg=audit(1562508551.990:82): avc:  denied  { sendto } for  pid=877 comm="chronyd" path="/run/chrony/chronyc.2725.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:cloud_init_t:s0 tclass=unix_dgram_socket permissive=1

Probably due to some file created during bootstrap with cloudinit, needs some investigations


Finally, the logrotate will need an addition to openstack-selinux, I'll take care of that one now.


I'll push the SELinux issues related to podman on their github - care to push the others to the right components and link back the BZ here so that we can have a fine status?

Do you have an accessible env so that I can test the generated selinux rules in order to ensure it covers all?

Cheers,

C.

Comment 2 Cédric Jeanneret 2019-07-10 06:19:46 UTC
Just pushed a change that will get rid of the logrotate AVCs. Some details:
- it's the *host* logrotate, not the containerized one
- apparently we have a bunch of logs written with the container_file_t context outside of /var/log/containers
- using a simple SELinux boolean allows to correct that AVC, making the whole thing far, far easier to manage
- imho we shouldn't push logrotate in a container, since it's installed by default on the nodes.... But that's another question/issue ;)

Comment 3 Julie Pichon 2019-07-10 09:02:58 UTC
It looks like a couple of the AVCs moved to bug 1728196 may be relevant here after all:

- The modprobe denial references openvswitch / openvswitch_load_module_t

type=AVC msg=audit(1562508548.724:40): avc:  denied  { execute_no_trans } for  pid=1212 comm="modprobe" path="/usr/bin/bash" dev="vda2" ino=4215568 scontext=system_u:system_r:openvswitch_load_module_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1

- The tmpwatch denial references /var/log/containers (cf. https://bugzilla.redhat.com/show_bug.cgi?id=1728196#c3 )

type=PROCTITLE msg=audit(07/09/2019 05:07:01.625:5580) : proctitle=tmpwatch --nodirs -X /var/log/containers/*/*log -X /var/log/containers/*/*/*log -X /var/log/containers/*/*err 15 /var/log/contai 
type=SYSCALL msg=audit(07/09/2019 05:07:01.625:5580) : arch=x86_64 syscall=unlink success=no exit=EACCES(Permission denied) a0=0x55cd23fc7a6b a1=0x55cd238b0044 a2=0x55cd23fc6930 a3=0x0 items=0 ppid=118753 pid=118755 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=tmpwatch exe=/usr/sbin/tmpwatch subj=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(07/09/2019 05:07:01.625:5580) : avc:  denied  { dac_override } for  pid=118755 comm=tmpwatch capability=dac_override  scontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tcontext=system_u:system_r:tmpreaper_t:s0-s0:c0.c1023 tclass=capability permissive=0

Comment 4 Marius Cornea 2019-09-04 11:40:52 UTC
With latest compose it looks like we still have a denial for openvswitch:

[root@overcloud-novacompute-0 heat-admin]# grep denied /var/log/audit/audit.log
type=AVC msg=audit(1567480861.728:31696): avc:  denied  { write } for  pid=683284 comm="logrotate" name="openvswitch" dev="sda2" ino=2881762 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1567568341.529:44396): avc:  denied  { write } for  pid=989380 comm="logrotate" name="openvswitch" dev="sda2" ino=2881762 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0

[root@overcloud-novacompute-0 heat-admin]# ls -l /var/log/openvswitch/
total 28
-rw-r-----. 1 openvswitch hugetlbfs  2490 Aug 31 23:31 ovsdb-server.log
-rw-r-----. 1 openvswitch hugetlbfs 16947 Sep  4 02:39 ovs-vswitchd.log
-rw-r--r--. 1 root        root         90 Aug 31 22:14 readme.txt

Comment 5 Cédric Jeanneret 2019-09-04 12:41:21 UTC
So https://github.com/redhat-openstack/openstack-selinux/pull/42 should sort this issue.

Comment 6 Julie Pichon 2019-09-04 14:21:32 UTC
Adding build info that should fix the logrotate issues. If there are other types of denials causing problems, please file separate bugs for them. Thank you.

Comment 11 Marius Cornea 2019-09-11 11:38:35 UTC
[root@overcloud-novacompute-0 heat-admin]# rpm -q openstack-selinux
openstack-selinux-0.8.20-0.20190904140454.936ea4f.el8ost.noarch

[root@overcloud-novacompute-0 heat-admin]# grep denied /var/log/audit/audit.log
[root@overcloud-novacompute-0 heat-admin]#

Comment 13 errata-xmlrpc 2019-09-21 11:23:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.