Bug 1738134 - OSP15 | sensu | containers health check reports "Failed to connect to bus" instead of reporting health status of containers.
Summary: OSP15 | sensu | containers health check reports "Failed to connect to bus" in...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
medium
urgent
Target Milestone: rc
: 15.0 (Stein)
Assignee: Julie Pichon
QA Contact: Nataf Sharabi
URL:
Whiteboard:
Depends On:
Blocks: 1728226
TreeView+ depends on / blocked
 
Reported: 2019-08-06 13:13 UTC by Martin Magr
Modified: 2020-01-12 11:49 UTC (History)
12 users (show)

Fixed In Version: openstack-selinux-0.8.19-0.20190813150447.72046d3.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1728226
Environment:
Last Closed: 2019-09-21 11:24:21 UTC
Target Upstream Version:


Attachments (Terms of Use)
audit.log (7.70 MB, text/plain)
2019-08-06 13:18 UTC, Martin Magr
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:24:41 UTC

Description Martin Magr 2019-08-06 13:13:35 UTC
+++ This bug was initially created as a clone of Bug #1728226 +++

<snip>

One thing we will need to fix is AVC denials in SELinux:

[root@controller-0 ~]# podman run --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 7138c84aec57 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer"
Failed to connect to bus: Permission denied
[root@controller-0 ~]# ausearch -m avc
<snip>
----
time->Tue Aug  6 12:20:42 2019
type=PROCTITLE msg=audit(1565094042.901:130455): proctitle=73797374656D63746C006C6973742D74696D657273002D2D6E6F2D7061676572002D2D6E6F2D6C6567656E6400747269706C656F2A6865616C7468636865636B2E74696D6572
type=SYSCALL msg=audit(1565094042.901:130455): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=55974d280730 a2=1d a3=7fffa95761d0 items=0 ppid=812841 pid=812858 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=60 comm="systemctl" exe="/usr/bin/systemctl" subj=system_u:system_r:container_t:s0:c400,c976 key=(null)
type=AVC msg=audit(1565094042.901:130455): avc:  denied  { connectto } for  pid=812858 comm="systemctl" path="/run/dbus/system_bus_socket" scontext=system_u:system_r:container_t:s0:c400,c976 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0

tl;dr: We gonna need to get rid of AVC above to enable reporting of container health checks to Sensu server side.

Comment 1 Martin Magr 2019-08-06 13:18:18 UTC
Created attachment 1601016 [details]
audit.log

Comment 2 Martin Magr 2019-08-06 13:45:25 UTC
type=AVC msg=audit(1565098666.014:137948): avc:  denied  { connectto } for  pid=219615 comm="systemctl" path="/run/systemd/private" scontext=system_u:system_r:container_t:s0:c104,c864 tcontext=system_u:system_r:init_t:s0 tclass=unix_stream_socket permissive=1
type=SYSCALL msg=audit(1565098666.014:137948): arch=c000003e syscall=42 success=yes exit=0 a0=3 a1=55efbfd4cb70 a2=16 a3=7fff69376be0 items=0 ppid=219606 pid=219615 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=60 comm="systemctl" exe="/usr/bin/systemctl" subj=system_u:system_r:container_t:s0:c104,c864 key=(null)ARCH=x86_64 SYSCALL=connect AUID="heat-admin" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"


Audit log contains a lot of AVC denials from health check runs. Above seems to me as the log you are looking for.

Comment 3 Martin Magr 2019-08-06 13:57:51 UTC
Ok, to summarize that, I ran podman command from description and right after it finished I ran: 
[root@controller-0 ~]# egrep -i "system(ctl|d)" /var/log/audit/audit.log | grep -i avc

considering the yum install prior to systemctl execution I think following is relevant as all (except the one before the last) is from pid=304402:
type=AVC msg=audit(1565099541.252:139337): avc:  denied  { write } for  pid=304402 comm="yum" path="/usr/lib/systemd/boot/efi/linuxx64.efi.stub;5d498615" dev="vda2" ino=228359 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1565099541.259:139340): avc:  denied  { write } for  pid=304402 comm="yum" name="system" dev="vda2" ino=4539392 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1565099541.259:139340): avc:  denied  { add_name } for  pid=304402 comm="yum" name="cryptsetup-pre.target;5d498615" scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1565099541.259:139340): avc:  denied  { create } for  pid=304402 comm="yum" name="cryptsetup-pre.target;5d498615" scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=file permissive=1
type=AVC msg=audit(1565099541.259:139340): avc:  denied  { write open } for  pid=304402 comm="yum" path="/usr/lib/systemd/system/cryptsetup-pre.target;5d498615" dev="vda2" ino=5931842 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=file permissive=1
type=AVC msg=audit(1565099541.260:139341): avc:  denied  { setattr } for  pid=304402 comm="yum" name="cryptsetup-pre.target;5d498615" dev="vda2" ino=5931842 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=file permissive=1
type=AVC msg=audit(1565099541.260:139342): avc:  denied  { remove_name } for  pid=304402 comm="yum" name="cryptsetup-pre.target;5d498615" dev="vda2" ino=5931842 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1565099541.260:139342): avc:  denied  { rename } for  pid=304402 comm="yum" name="cryptsetup-pre.target;5d498615" dev="vda2" ino=5931842 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=file permissive=1
type=AVC msg=audit(1565099541.260:139342): avc:  denied  { unlink } for  pid=304402 comm="yum" name="cryptsetup-pre.target" dev="vda2" ino=5931843 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=file permissive=1
type=AVC msg=audit(1565099541.261:139343): avc:  denied  { create } for  pid=304402 comm="yum" name="systemd-remount-fs.service;5d498615" scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=lnk_file permissive=1
type=AVC msg=audit(1565099541.261:139344): avc:  denied  { setattr } for  pid=304402 comm="yum" name="systemd-remount-fs.service;5d498615" dev="vda2" ino=228408 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=lnk_file permissive=1
type=AVC msg=audit(1565099541.261:139345): avc:  denied  { rename } for  pid=304402 comm="yum" name="systemd-remount-fs.service;5d498615" dev="vda2" ino=228408 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=lnk_file permissive=1
type=AVC msg=audit(1565099541.261:139345): avc:  denied  { unlink } for  pid=304402 comm="yum" name="systemd-remount-fs.service" dev="vda2" ino=228409 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=lnk_file permissive=1
type=AVC msg=audit(1565099541.265:139346): avc:  denied  { setattr } for  pid=304402 comm="yum" name="systemd-udev-trigger.service.d" dev="vda2" ino=6465377 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1565099541.685:139359): avc:  denied  { connectto } for  pid=305401 comm="systemctl" path="/run/systemd/private" scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:system_r:init_t:s0 tclass=unix_stream_socket permissive=1
type=AVC msg=audit(1565099541.686:139360): avc:  denied  { execute_no_trans } for  pid=305402 comm="sh" path="/usr/lib/systemd/systemd-random-seed" dev="vda2" ino=1233861 scontext=system_u:system_r:container_t:s0:c243,c493 tcontext=system_u:object_r:lib_t:s0 tclass=file permissive=1

You can ignore yum AVCs are this is caused by installing new systemd-udev in container than is in the host and container has /usr/lib/systemd shared.

Comment 4 Julie Pichon 2019-08-07 14:32:27 UTC
Thank you for providing access to the machine with all the logs.

Very strangely, the AVC denial with system_dbusd_t from comment 0 disappears as soon as SELinux was set to permissive. Did you do anything else at the same time permissive mode was set? Maybe that yum install? Normally there shouldn't be a difference... The init_t denial in comment 3 however shows up all the time whether in permissive mode or not, when calling that healtcheck timer command.

Before we add a rule for init_t, I wonder: did you try to mount the systemd volume with :z enabled already? Something like --volume=/usr/lib/systemd:/usr/lib/systemd:z. It's documented at https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label and I've seen it resolve issues with "permission denied" in host/container communications before. Though there are serious warnings that go with using the label and I'm not sure if it would work as well on a /usr/lib directory.

Comment 5 Martin Magr 2019-08-08 12:27:48 UTC
No the only thing I was doing was the podman command and setenforce 0/1.

Comment 6 Martin Magr 2019-08-08 12:29:13 UTC
Oh yes, I tried --volume=/usr/lib/systemd:/usr/lib/systemd:ro,z --volume=/usr/lib/systemd:/usr/lib/systemd:rw,z , but podman is complaining that especially that path is forbidden to relabel.

Comment 7 Julie Pichon 2019-08-08 12:54:23 UTC
Ok, thank you for the answer!

Proposed rules update at https://github.com/redhat-openstack/openstack-selinux/pull/35

Comment 17 Martin Magr 2019-08-28 11:24:46 UTC
Selinux problem of this issue has been solved:

[root@controller-0 ~]# podman run --systemd --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run/dbus/system_bus_socket://run/dbus/system_bus_socket:rw,z --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 2cc104aac809 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer"
Failed to connect to bus: No data available

Comment 19 errata-xmlrpc 2019-09-21 11:24:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.