Bug 1703242
Summary: | certmonger post save command fails with avc denied | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Jeremy Agee <jagee> | ||||
Component: | openstack-selinux | Assignee: | Julie Pichon <jpichon> | ||||
Status: | CLOSED NEXTRELEASE | QA Contact: | nlevinki <nlevinki> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 14.0 (Rocky) | CC: | cjeanner, ggrasza, lhh, lvrabec, zcaplovi | ||||
Target Milestone: | --- | Keywords: | Triaged, ZStream | ||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2020-07-16 13:55:36 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1609025 | ||||||
Attachments: |
|
Description
Jeremy Agee
2019-04-25 21:07:44 UTC
Thank you for the report. Could you attach a full copy of the audit.log from the permissive run, so that we can look in details at the AVCs? Some of these don't look related to OpenStack. Thank you. openstack-selinux and selinux-policy versions would be helpful as well. Created attachment 1559999 [details]
audit log from after the post run command in permissive mode
Attached the audit log. here is the version info. [stack@undercloud-0 ~]$ rpm -q openstack-selinux openstack-selinux-0.8.18-1.el7ost.noarch [stack@undercloud-0 ~]$ rpm -q selinux-policy selinux-policy-3.13.1-229.el7_6.12.noarch This is what the script thats running with the "reload external" passed options. [stack@undercloud-0 ~]$ sudo cat /usr/bin/certmonger-haproxy-refresh.sh #!/bin/bash # This script is meant to reload HAProxy when certmonger triggers a certificate # renewal. It'll concatenate the needed certificates for the PEM file that # HAProxy reads. die() { echo "$*" 1>&2 ; exit 1; } [[ $# -eq 2 ]] || die "Invalid number of arguments" [[ $1 == @(reload|restart) ]] || die "First argument must be one of 'reload' or 'restart'." ACTION=$1 NETWORK=$2 certmonger_ca=$(hiera -c /etc/puppet/hiera.yaml certmonger_ca) container_cli=$(hiera -c /etc/puppet/hiera.yaml container_cli docker) service_certificate="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::certificate_dir)/overcloud-haproxy-$NETWORK.crt" service_key="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::key_dir)/overcloud-haproxy-$NETWORK.key" ca_path="" if [ "$certmonger_ca" == "local" ]; then ca_path="/etc/pki/ca-trust/source/anchors/cm-local-ca.pem" elif [ "$certmonger_ca" == "IPA" ]; then ca_path="/etc/ipa/ca.crt" fi if [ "$NETWORK" != "external" ]; then service_pem="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::certificate_dir)/overcloud-haproxy-$NETWORK.pem" else service_pem="$(hiera -c /etc/puppet/hiera.yaml tripleo::haproxy::service_certificate)" fi cat "$service_certificate" "$ca_path" "$service_key" > "$service_pem" haproxy_container_name=$($container_cli ps --format="{{.Names}}" | grep haproxy) if [ "$ACTION" == "reload" ]; then # Copy the new cert from the mount-point to the real path $container_cli exec "$haproxy_container_name" cp "/var/lib/kolla/config_files/src-tls$service_pem" "$service_pem" # Set appropriate permissions $container_cli exec "$haproxy_container_name" chown haproxy:haproxy "$service_pem" # Trigger a reload for HAProxy to read the new certificates pkill -f -HUP haproxy-systemd-wrapper elif [ "$ACTION" == "restart" ]; then # Copying the certificate and permissions will be handled by kolla's start # script. $container_cli restart "$haproxy_container_name" fi Thank you for the audit.log and including the script details! This is very helpful. Where does the script come from, by the way? Is it something we ship in OpenStack? There are 140 unique AVC denials in the logs. I thought it strange that 121 of them are related to "pkill" and certmonger trying to kill everything, which led me to this upstream spec for Train/OSP16, https://specs.openstack.org/openstack/tripleo-specs/specs/train/certificate-management.html : "The main issue now is the use of "pkill", especially for httpd services. Since Certmonger has no knowledge of what container has an httpd service running, it uses a wide fly swatter in the hope all related services will effectively be reloaded with the new certificate." If we only look at the non-pkill AVC denials, the following rules (as well as enabling logrotate_read_inside_containers, which seems generally useful) are enough to resolve all denials but 3: allow certmonger_t puppet_etc_t:dir search; allow certmonger_t puppet_etc_t:file { getattr ioctl open read }; allow certmonger_t container_runtime_exec_t:file { execute execute_no_trans getattr ioctl open read }; allow certmonger_t container_runtime_t:dir { getattr search }; allow certmonger_t container_runtime_t:file { open read }; This generally seems in line with what the script does. The 3 other denials seem somewhat unrelated... type=AVC msg=audit(1588173023.389:4931): avc: denied { search } for pid=145752 comm="docker-current" name="net" dev="proc" ino=8799 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:sysctl_net_t:s0 tclass=dir permissive=1 bz1703242:11:type=AVC msg=audit(1588173023.389:4931): avc: denied { read } for pid=145752 comm="docker-current" name="somaxconn" dev="proc" ino=63146 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:sysctl_net_t:s0 tclass=file permissive=1 bz1703242:12:type=AVC msg=audit(1588173023.389:4931): avc: denied { open } for pid=145752 comm="docker-current" path="/proc/sys/net/core/somaxconn" dev="proc" ino=63146 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:sysctl_net_t:s0 tclass=file permissive=1 I do see there is an explicit pkill call in the script, "pkill -f -HUP haproxy-systemd-wrapper" that I can't match explicitly to one of the pkill AVC denials in the logs. I'm not sure if the rules above are enough to handle that case as well. Jeremy, would you be able to run "ps -efZ | grep haproxy" so that we can see the full label for the haproxy-systemd-wrapper process? Thank you. I'd like to understand better how certmonger is meant to be handled in 14 until the Train upstream spec above is implemented... Cédric, I see you wrote up that spec, would you have any pointers? Hello Julie, I indeed wrote that spec, but the whole certmonger and related TLS configurations are more from DFG:Security - I therefore add a needinfo() for Grzegorz (xek). pkill is used because we have shared certificates, and certmonger doesn't know what services uses said certificate when renewing them - therefore, the pkill was added in order to force-reload upon renewal. I'm not really sure we will be able to backport the spec content to 14 nor 15 though, so I'm pretty positive we will need something in-between, being some SELinux rules or equivalent. I don't have a "nice" solution for that, unfortunately :(. The rules you listed should indeed be OK, and shouldn't create big issues regarding system security. But if I understand you correctly, they aren't meant for the pkill calls directly, probably more for some process listing and the like, right? Cheers, C. Thanks Cédric! I am hoping once we know the label for the haproxy-systemd-wrapper process this will be enough to narrow down the pkill rule we need, but it's also possible I misunderstand how this is all supposed to work. At the moment I doubt the rules I mentioned will be enough. Thanks for bringing in expertise from DFG:Security as well! I believe most of the AVCs denials preventing the cert renewal should be resolved thanks to bug 1777368 and bug 1777263. We will want to rebase the OSP14 openstack-selinux package to include these patches. The read-only AVCs denials related to pkill will likely remain but that is sort of expected behaviour (running "pkill" involves reading through every process to get their info in order to get at the right one, so if the process is unrelated the caller (certmonger here) won't have permission to do anything with them.) the "reload external" passed options are used on the controller: [heat-admin@overcloud-controller-0 ~]$ ps -efZ | grep haproxy-systemd-wrapper unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 heat-ad+ 125682 124084 0 09:37 pts/1 00:00:00 grep --color=auto haproxy-systemd-wrapper system_u:system_r:spc_t:s0 root 255232 255111 0 2019 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg I think this is resolved based on the bugs linked to in comment 8. The blocked bug is also closed, and links to another few bugs from that one show a number of related certmonger changes happened in other components (puppet-tripleo, THT, etc). Based on all this I am closing this bz, but feel free to reopen against a newer version if there are still related issues. Thank you. |