Bug 1745142 - [OSP15] Large amount of AVC denials related to certmonger_t
Summary: [OSP15] Large amount of AVC denials related to certmonger_t
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Julie Pichon
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-23 17:32 UTC by Julie Pichon
Modified: 2019-12-09 14:41 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-12-09 14:41:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Permissive audit.log from the original bug (5.29 MB, text/plain)
2019-08-23 17:32 UTC, Julie Pichon
no flags Details

Description Julie Pichon 2019-08-23 17:32:01 UTC
Created attachment 1607449 [details]
Permissive audit.log from the original bug

+++ This bug was initially created as a copy of Bug #1743485 +++

The main "hanging" issue was handled with a revert in the original bug. This is a separate bug to deal with the remaining AVC denials. (Any reference/denial related to "allow certmonger_t etc_t:file write" can be ignored). Trimmed the comments to keep the relevant parts.


--- Additional comment from Michele Baldessari on 2019-08-20 08:50:23 UTC ---

Things maybe worth noting:
1) Restarting certmonger and dbus while puppet is running had no effect
2) There are a lot of selinux denials due unallowed commands like pkill.
[root@controller-0 ~]# grep denied /var/log/audit/audit.log  |grep pkill |wc -l
903
[root@controller-0 ~]# grep denied /var/log/audit/audit.log  |grep pkill |tail -n2
type=AVC msg=audit(1566287997.986:8897): avc:  denied  { getattr } for  pid=26996 comm="pkill" path="/proc/26970" dev="proc" ino=138544 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dir permissive=0
type=AVC msg=audit(1566287997.986:8898): avc:  denied  { getattr } for  pid=26996 comm="pkill" path="/proc/26985" dev="proc" ino=135164 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:cluster_t:s0 tclass=dir permissive=0

[root@controller-0 ~]# getcert list |grep post-save |sort|uniq
        post-save command: 
        post-save command: pkill -USR1 httpd
        post-save command: systemctl reload libvirtd
        post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload ctlplane
        post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload external
        post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload internal_api
        post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage
        post-save command: /usr/bin/certmonger-rabbitmq-refresh.sh
        post-save command: /usr/bin/certmonger-redis-refresh.sh

3) I also saw some selinux denials around certmonger:
[root@controller-0 ~]# grep denied /var/log/audit/audit.log  |grep certmonger |tail -n2
type=AVC msg=audit(1566288695.384:9015): avc:  denied  { write } for  pid=9771 comm="certmonger" name="ca.crt" dev="vda2" ino=7156021 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=0
type=AVC msg=audit(1566289692.281:9075): avc:  denied  { write } for  pid=10436 comm="certmonger" name="ca.crt" dev="vda2" ino=7156021 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=0

4) The systemctl reload libvirtd post-save command does not work now because libvirt runs inside the nova_libvirt container and libvirtd is not running/disabled

5) I did not manage to observe any special errors in the freeipa-0 VM  

I will try a deploy with selinux disabled and see if that helps here

--- Additional comment from Michele Baldessari on 2019-08-20 10:49:09 UTC ---

Indeed disabling selinux worked for me and I got a correctly deployed overcloud:
Ansible passed.
Overcloud configuration completed.
Overcloud Endpoint: https://overcloud.redhat.local:13000
Overcloud Horizon Dashboard URL: https://overcloud.redhat.local:443/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed

On the nodes all certs are in good states:
[root@controller-0 ~]# getcert list |grep status |uniq
        status: MONITORING

--- Additional comment from Michele Baldessari on 2019-08-20 10:57:53 UTC ---

Moving to openstack-selinux package. The ask here is two rules to fix the following two classes of denials:
A) pkill
type=AVC msg=audit(1566287997.986:8897): avc:  denied  { getattr } for  pid=26996 comm="pkill" path="/proc/26970" dev="proc" ino=138544 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=dir permissive=0
type=AVC msg=audit(1566287997.986:8898): avc:  denied  { getattr } for  pid=26996 comm="pkill" path="/proc/26985" dev="proc" ino=135164 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:cluster_t:s0 tclass=dir permissive=0

This one is needed because some certmonger post-save scripts call pkill (https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/apache/apache-baremetal-puppet.j2.yaml#L118)

[...]

--- Additional comment from Julie Pichon on 2019-08-20 14:36:32 UTC ---

I am seeing a lot of pkill commands on many different targets. That reminds me of bug 1703242 and the design issues discussed upstream in https://specs.openstack.org/openstack/tripleo-specs/specs/train/certificate-management.html#problem-description, though that other bug was around renewing the certificates as opposed to a new deployment.

The THT file you link to (https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/apache/apache-baremetal-puppet.j2.yaml#L118) seems different than the file mentioned on that bug (certmonger-haproxy-refresh.sh) although you do have it in comment 3. How did you narrow down to these two denials in A) for pkill?

There are a few other denials related to certmonger trying to access hiera files like /etc/puppet/hiera.yaml and /etc/puppet/hieradata/controller_extraconfig.json. Do you reckon these are unlikely to be causing this particular issue at this point?

--- Additional comment from Julie Pichon on 2019-08-20 15:24:20 UTC ---

For better visibility on how many contexts are touched by certmonger in this audit file:

#============= certmonger_t ==============
allow certmonger_t NetworkManager_t:dir { getattr search };
allow certmonger_t NetworkManager_t:file { open read };
allow certmonger_t auditd_t:dir { getattr search };
allow certmonger_t auditd_t:file { open read };
allow certmonger_t chronyd_t:dir { getattr search };
allow certmonger_t chronyd_t:file { open read };
allow certmonger_t cluster_t:dir { getattr search };
allow certmonger_t cluster_t:file { open read };
allow certmonger_t container_runtime_exec_t:file { execute execute_no_trans getattr open read };
allow certmonger_t container_share_t:blk_file { create unlink };
allow certmonger_t container_share_t:dir { add_name getattr ioctl mounton open read remove_name setattr write };
allow certmonger_t container_share_t:file { getattr lock open read write };
allow certmonger_t container_var_lib_t:dir getattr;
allow certmonger_t container_var_lib_t:file { getattr lock open read write };
allow certmonger_t container_var_run_t:dir setattr;
allow certmonger_t container_var_run_t:file { getattr lock open read write };
allow certmonger_t crond_t:dir { getattr search };
allow certmonger_t crond_t:file { open read };
allow certmonger_t fs_t:filesystem unmount;
allow certmonger_t getty_t:dir { getattr search };
allow certmonger_t getty_t:file { open read };
allow certmonger_t gssproxy_t:dir { getattr search };
allow certmonger_t gssproxy_t:file { open read };
allow certmonger_t iptables_exec_t:file { execute execute_no_trans getattr open read };
allow certmonger_t irqbalance_t:dir { getattr search };
allow certmonger_t irqbalance_t:file { open read };
allow certmonger_t kernel_t:dir { getattr search };
allow certmonger_t kernel_t:file { open read };
allow certmonger_t oddjob_t:dir { getattr search };
allow certmonger_t oddjob_t:file { open read };
allow certmonger_t openvswitch_t:dir { getattr search };
allow certmonger_t openvswitch_t:file { open read };
allow certmonger_t policykit_t:dir { getattr search };
allow certmonger_t policykit_t:file { open read };
allow certmonger_t puppet_etc_t:file { getattr ioctl open read };
allow certmonger_t rhsmcertd_t:dir { getattr search };
allow certmonger_t rhsmcertd_t:file { open read };
allow certmonger_t rngd_t:dir { getattr search };
allow certmonger_t rngd_t:file { open read };
allow certmonger_t rpcbind_t:dir { getattr search };
allow certmonger_t rpcbind_t:file { open read };
allow certmonger_t self:capability { mknod sys_resource };
allow certmonger_t self:netlink_netfilter_socket { bind create getattr };
allow certmonger_t self:process setrlimit;
allow certmonger_t sshd_t:dir { getattr search };
allow certmonger_t sshd_t:file { open read };
allow certmonger_t sssd_t:dir { getattr search };
allow certmonger_t sssd_t:file { open read };
allow certmonger_t sysctl_net_t:dir search;
allow certmonger_t sysctl_net_t:file { open read };
allow certmonger_t syslogd_t:dir { getattr search };
allow certmonger_t syslogd_t:file { open read };
allow certmonger_t system_dbusd_t:dir { getattr search };
allow certmonger_t system_dbusd_t:file { open read };
allow certmonger_t systemd_logind_t:dir { getattr search };
allow certmonger_t systemd_logind_t:file { open read };
allow certmonger_t systemd_machined_t:dir { getattr search };
allow certmonger_t systemd_machined_t:file { open read };
allow certmonger_t tuned_t:dir { getattr search };
allow certmonger_t tuned_t:file { open read };
allow certmonger_t udev_t:dir { getattr search };
allow certmonger_t udev_t:file { open read };
allow certmonger_t unconfined_t:dir { getattr search };
allow certmonger_t unconfined_t:file { open read };
allow certmonger_t virt_qemu_ga_t:dir { getattr search };
allow certmonger_t virt_qemu_ga_t:file { open read };
allow certmonger_t virtd_t:dir { getattr search };
allow certmonger_t virtd_t:file { open read };
allow certmonger_t virtd_unit_file_t:service reload;

On the plus side, the majority appear to be read-only permissions...?

--- Additional comment from Michele Baldessari on 2019-08-20 15:32:49 UTC ---

(In reply to Julie Pichon from comment #8)
> I am seeing a lot of pkill commands on many different targets. That reminds
> me of bug 1703242 and the design issues discussed upstream in
> https://specs.openstack.org/openstack/tripleo-specs/specs/train/certificate-
> management.html#problem-description, though that other bug was around
> renewing the certificates as opposed to a new deployment.
> 
> The THT file you link to
> (https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/
> apache/apache-baremetal-puppet.j2.yaml#L118) seems different than the file
> mentioned on that bug (certmonger-haproxy-refresh.sh) although you do have
> it in comment 3. How did you narrow down to these two denials in A) for
> pkill?

Oh so I blindly took the last two pkill denials under the (maybe wrong?) assumption
that we had a generic 'allow certmonger to run pkill' kind of issue.

In terms of code we have pkill (or kill in the case of 3) in the following places:
1) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/apache/apache-baremetal-puppet.j2.yaml#L118
2) https://github.com/openstack/puppet-tripleo/blob/master/files/certmonger-redis-refresh.sh#L14
3) https://github.com/openstack/puppet-tripleo/blob/master/files/certmonger-haproxy-refresh.sh#L46 (this one uses kill and not pkill)

> There are a few other denials related to certmonger trying to access hiera
> files like /etc/puppet/hiera.yaml and
> /etc/puppet/hieradata/controller_extraconfig.json. Do you reckon these are
> unlikely to be causing this particular issue at this point?

I think I missed those tbh. They do look relevant although am not entirely sure.
Want to throw a patch my way for fixing B) and we see if the deploy works and then
we investigate A) and the hiera denials a bit more in detail?

--- Additional comment from Julie Pichon on 2019-08-20 16:12:33 UTC ---

Useful comments, thank you - dumping some additional information after looking a bit more at the non-read-only denials:

container_runtime_exec_t (e.g. execute, execute_no_trans):
- /usr/bin/certmonger-rabbitmq-refresh.sh
- /usr/bin/certmonger-haproxy-refresh.sh reload ctlplane
- podman exec  cp /var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/haproxy/overcloud-haproxy-ctlplane.pem /etc/pki/tls/certs/
- podman exec  cp /var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/haproxy/overcloud-haproxy-ctlplane.pem /etc/pki/tls/certs/

I suspect we will need these? Though not entirely sure why the podman exec is running with certmonger_t as source context.

container_share_t starts to looks more suspicious, even though it starts from the same two scripts:
- /usr/bin/certmonger-rabbitmq-refresh.sh
- /usr/bin/certmonger-haproxy-refresh.sh reload ctlplane

Lots of mentions of the container storage overlays, not sure that we want certmonger to be able to mess with these:
type=AVC msg=audit(20/08/19 11:22:32.425:7865) : avc:  denied  { ioctl } for  pid=26655 comm=podman path=/var/lib/containers/storage/overlay dev="vda2" ino=92275029 ioctlcmd=0x581f scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_share_t:s0 tclass=dir permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.425:7866) : avc:  denied  { unlink } for  pid=26655 comm=podman name=backingFsBlockDev dev="vda2" ino=92275030 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_share_t:s0 tclass=blk_file permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.425:7866) : avc:  denied  { remove_name } for  pid=26655 comm=podman name=backingFsBlockDev dev="vda2" ino=92275030 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_share_t:s0 tclass=dir permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.425:7866) : avc:  denied  { write } for  pid=26655 comm=podman name=overlay dev="vda2" ino=92275029 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_share_t:s0 tclass=dir permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.425:7868) : avc:  denied  { read write } for  pid=26655 comm=podman name=layers.lock dev="vda2" ino=46138520 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_share_t:s0 tclass=file permissive=1 

Similarly with container_var_lib_t:
type=AVC msg=audit(20/08/19 11:22:32.423:7856) : avc:  denied  { lock } for  pid=26655 comm=podman path=/var/lib/containers/storage/libpod/bolt_state.db dev="vda2" ino=85984286 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.423:7855) : avc:  denied  { read write } for  pid=26655 comm=podman name=bolt_state.db dev="vda2" ino=85984286 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=1 

And again for container_var_run_t:
type=AVC msg=audit(20/08/19 11:22:32.432:7876) : avc:  denied  { write } for  pid=26655 comm=podman name=alive.lck dev="tmpfs" ino=90484 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_var_run_t:s0 tclass=file permissive=1 
type=AVC msg=audit(20/08/19 11:22:39.634:7971) : avc:  denied  { lock } for  pid=26953 comm=podman path=/run/libpod/alive.lck dev="tmpfs" ino=90484 scontext=system_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_var_run_t:s0 tclass=file permissive=1 

fs_t:filesystem unmount AVCs seems to originate from running "podman ps --format={{.Names}}"?

iptables_exec_t (execute, execute_no_trans)
- /usr/sbin/iptables --version coming from "comm=podman path=/usr/sbin/xtables-nft-multi"?

Again something related to podman... The netlink_netfilter_socket AVCs are all related to iptables as well.

self:capability, setrlimit, again, the script and some podman commands that look suspicious:
 /usr/bin/certmonger-rabbitmq-refresh.sh 
type=AVC msg=audit(20/08/19 11:22:32.422:7853) : avc:  denied  { setrlimit } for  pid=26655 comm=podman scontext=system_u:system_r:certmonge
r_t:s0 tcontext=system_u:system_r:certmonger_t:s0 tclass=process permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.422:7853) : avc:  denied  { sys_resource } for  pid=26655 comm=podman capability=sys_resource  scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:certmonger_t:s0 tclass=capability permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.425:7867) : avc:  denied  { create } for  pid=26655 comm=podman name=backingFsBlockDev scontext=system_
u:system_r:certmonger_t:s0 tcontext=system_u:object_r:container_share_t:s0 tclass=blk_file permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.425:7867) : avc:  denied  { add_name } for  pid=26655 comm=podman name=backingFsBlockDev scontext=syste
m_u:system_r:certmonger_t:s0 tcontext=unconfined_u:object_r:container_share_t:s0 tclass=dir permissive=1 
type=AVC msg=audit(20/08/19 11:22:32.425:7867) : avc:  denied  { mknod } for  pid=26655 comm=podman capability=mknod  scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:certmonger_t:s0 tclass=capability permissive=1 

"podman exec  cp /var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/rabbitmq.crt /etc/pki/tls/certs/rabbitmq.crt" leads to "type=AVC msg=audit(20/08/19 11:22:32.585:7879) : avc:  denied  { sys_resource } for  pid=26669 comm=podman capability=sys_resource  scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:certmonger_t:s0 tclass=capability permissive=1"

There is only the one virtd denial, not sure what it's related to although certmonger_t is the source context again:
type=USER_AVC msg=audit(20/08/19 11:22:38.440:7949) : pid=1 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg='avc:  denied  { reload } for auid=unset uid=root gid=root path=/usr/lib/systemd/system/libvirtd.service cmdline="" scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:virtd_unit_file_t:s0 tclass=service permissive=1  exe=/usr/lib/systemd/systemd sauid=root hostname=? addr=? terminal=?'

--- Additional comment from Julie Pichon on 2019-08-21 16:40:33 UTC ---

There were thoughts that maybe it's hiera trying to parse everything causing all the read-only denials, but from what I can tell the hiera part should be resolved with "allow certmonger_t puppet_etc_t:file { getattr ioctl open read }" as the files appear to be labelled as expected. Matching proctitle to the read-only AVCs, this is what I see:

proctitle=pkill -o -HUP stunnel (getattr/open/read/search denials)
- auditd_t
- chronyd_t
- cluster_t
- crond_t
- getty_t
- gssproxy_t
- irqbalance_t
- kernel_t
- NetworkManager_t
- oddjob_t
- openvswitch_t
- policykit_t
- rhsmcertd_t
- rngd_t
- rpcbind_t
- sshd_t
- sssd_t
- syslogd_t
- system_dbusd_t
- systemd_logind_t
- systemd_machined_t
- tuned_t
- udev_t
- unconfined_t:s0-s0:c0.c1023
- virt_qemu_ga_t
- virtd_t

proctitle=pkill -USR1 httpd (open/search/getattr/read denials)
- chronyd_t
- gssproxy_t
- irqbalance_t
- kernel_t
- policykit_t
- rhsmcertd_t
- system_dbusd_t

I didn't see any other kill commands.

The "podman ps" command at https://github.com/openstack/puppet-tripleo/blob/master/files/certmonger-haproxy-refresh.sh#L36 seems responsible for many of the complicated denials in comment 13.

Comment 1 Julie Pichon 2019-08-23 17:54:51 UTC
I'm leaning toward the thought we don't need to worry about all the read denials as it's SELinux doing its job. Running "pkill" apparently means reading through every process to get their info, to get at the right one e.g.:

avc:  denied  { getattr } for  pid=26751 comm=pkill path=/proc/2 dev="proc" ino=13044 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=dir permissive=1 
type=AVC msg=audit(20/08/19 11:22:33.512:7882) : avc:  denied  { open } for  pid=26751 comm=pkill path=/proc/2/stat dev="proc" ino=13045 sco
ntext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=file permissive=1 

and so on. There are dontaudit rules we might consider using if we wanted to hide these (maybe something like domain_dontaudit_read_all_domains_state, like "ps" uses?), though we'd want to make sure it doesn't hide access issues to labels that are needed. Perhaps if someone could run ps with -Z to confirm the label of the httpd and stunnel processes on that machine, since there are no obviously related denials?

I'm looking at domain transitions and other interfaces related to containers to see how we might handle the other ones.

Comment 2 Julie Pichon 2019-12-09 14:41:10 UTC
The pkill denials will likely remain as per the reasoning in comment 1. For the rest I believe these should be resolved thanks to bug 1777368 and bug 1777263, which should be pulled automatically into the next openstack-selinux rebase for OSP15. I'm going to close this bug for now based on this, though anyone feel free to reopen if you still see relevant AVCs after this.


Note You need to log in before you can comment on or make changes to this bug.