Description of problem: Changing the haproxy configuration(e.g. timeouts adjustments or master/infra nodes scale out) after the initial deployment results in the haproxy container stuck in a restart loop due to selinux denials: [root@openshift-master-0 ~]# docker ps | grep haproxy ed4f116c597c 192.168.24.1:8787/rhosp14/openstack-haproxy:2018-12-07.1 "kolla_start" 11 minutes ago Restarting (1) 4 minutes ago haproxy [root@openshift-master-0 ~]# docker logs -f haproxy Running command: '/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg' + echo 'Running command: '\''/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'\''' + exec /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg <7>haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ds [ALERT] 343/232417 (11) : Starting frontend GLOBAL: error when trying to preserve previous UNIX socket [/var/lib/haproxy/stats] <5>haproxy-systemd-wrapper: exit, haproxy RC=1 + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc/haproxy/haproxy.cfg INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/haproxy/haproxy.cfg to /etc/haproxy/haproxy.cfg INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/lib/haproxy ERROR:__main__:Failed to change ownership of /var/lib/haproxy to 42454:42454 Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 345, in set_perms os.chown(path, uid, gid) OSError: [Errno 13] Permission denied: '/var/lib/haproxy' ++ cat /run_command + CMD='/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start + echo 'Running command: '\''/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'\''' Running command: '/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg' + exec /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg <7>haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ds [ALERT] 343/232742 (11) : Starting frontend GLOBAL: error when trying to preserve previous UNIX socket [/var/lib/haproxy/stats] <5>haproxy-systemd-wrapper: exit, haproxy RC=1 [root@openshift-master-0 ~]# grep denied /var/log/audit/audit.log type=AVC msg=audit(1544502049.751:11633): avc: denied { setattr } for pid=52906 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502049.752:11634): avc: denied { read } for pid=52906 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502049.763:11637): avc: denied { write } for pid=52916 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502050.186:11646): avc: denied { setattr } for pid=53040 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502050.186:11647): avc: denied { read } for pid=53040 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502050.201:11650): avc: denied { write } for pid=53042 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502050.700:11655): avc: denied { setattr } for pid=53115 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502050.700:11656): avc: denied { read } for pid=53115 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502050.714:11659): avc: denied { write } for pid=53117 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502051.406:11668): avc: denied { setattr } for pid=53256 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502051.407:11669): avc: denied { read } for pid=53256 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502051.419:11672): avc: denied { write } for pid=53259 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502052.505:11677): avc: denied { setattr } for pid=53381 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502052.506:11678): avc: denied { read } for pid=53381 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502052.520:11681): avc: denied { write } for pid=53385 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502054.394:11686): avc: denied { setattr } for pid=53532 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502054.394:11687): avc: denied { read } for pid=53532 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502054.409:11690): avc: denied { write } for pid=53549 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502057.902:11695): avc: denied { setattr } for pid=53943 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502057.902:11696): avc: denied { read } for pid=53943 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502057.915:11699): avc: denied { write } for pid=53945 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502064.597:11730): avc: denied { setattr } for pid=54234 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502064.598:11731): avc: denied { read } for pid=54234 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502064.614:11734): avc: denied { write } for pid=54236 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502077.733:11754): avc: denied { setattr } for pid=54420 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502077.734:11755): avc: denied { read } for pid=54420 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502077.755:11758): avc: denied { write } for pid=54422 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502103.638:11835): avc: denied { setattr } for pid=55496 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502103.638:11836): avc: denied { read } for pid=55496 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502103.651:11839): avc: denied { write } for pid=55498 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502155.167:12007): avc: denied { setattr } for pid=59795 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502155.167:12008): avc: denied { read } for pid=59795 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502155.182:12011): avc: denied { write } for pid=59803 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502257.860:12166): avc: denied { setattr } for pid=63828 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502257.861:12167): avc: denied { read } for pid=63828 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502257.875:12170): avc: denied { write } for pid=63830 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502462.972:13210): avc: denied { setattr } for pid=71398 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502462.973:13211): avc: denied { read } for pid=71398 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544502462.992:13218): avc: denied { write } for pid=71406 comm="haproxy" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0 Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-9.0.1-0.20181013060901.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OCP overcloud 2. Adjust the haproxy config by adding the following inside an environment file: ExtraConfig: tripleo::haproxy::haproxy_defaults_override: retries: 3 maxconn: 20000 tripleo::haproxy::haproxy_default_timeout: [ 'http-request 10s', 'queue 1m', 'connect 10s', 'client 500s', 'server 500s', 'check 10s', 'http-keep-alive 10s' ] 3. Re-run overcloud deploy Actual results: The haproxy container on master nodes get stuck in a restart loop because of: ERROR:__main__:Failed to change ownership of /var/lib/haproxy to 42454:42454 Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 345, in set_perms os.chown(path, uid, gid) OSError: [Errno 13] Permission denied: '/var/lib/haproxy' Expected results: haproxy container start without issues. Additional info:
I investigated a little bit... It seems we change the context of /var/lib/haproxy during the update to haproxy_var_lib_t again. In other words, during the initial deployment we set the right context to the folder here: https://github.com/openstack/tripleo-heat-templates/blob/bf48c36bc4e78f19ed4c488b8174b03ccbf5a4d7/docker/services/haproxy.yaml#L289 (svirt_sandbox_file_t is an alias of container_file_t) And for this reason we can write the config. But then, when we perform an update, the context changes to haproxy_var_lib_t and a process with container_t can't do anything there. Now I'm looking what causes that context switching.
Okay, it seems I've figured it out: we need to add relabling here: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/haproxy.yaml#L239 should be /var/lib/haproxy:/var/lib/haproxy:rw,z
No doc text required.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045