Description of problem: Currently docker service is configured without --selinux-enabled in /etc/sysconfig/docker based on what it appears to be a limitation of RHEL 7.2: https://review.openstack.org/#/c/551985/4/manifests/profile/base/docker.pp Starting with RHEL 7.4 OverlayFS supports SELinux security labels, and you can enable SELinux support for containers by specifying --selinux-enabled in /etc/sysconfig/docker: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.6_release_notes/#new_features_file_systems Since OSP14 is delievered on RHEL 7.6 we should configure --selinux-enabled as well. Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-9.0.1-0.20181013060899.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP14 2. On overcloud nodes check /etc/sysconfig/docker Actual results: --selinux-enabled is not present Expected results: --selinux-enabled should be present. Additional info:
I applied https://review.openstack.org/#/c/623541/ to get --selinux-enabled during deployment but docker-puppet* containers fail to start due to selinux denials: root@controller-1 heat-admin]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1ac540d35bb5 192.168.24.1:8787/rhosp14/openstack-horizon:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-horizon d0983e70b752 192.168.24.1:8787/rhosp14/openstack-neutron-server:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-neutron 9e4d0cb8ed72 192.168.24.1:8787/rhosp14/openstack-ceilometer-central:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-ceilometer bbfadd108db1 192.168.24.1:8787/rhosp14/openstack-rabbitmq:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-rabbitmq 6ad67ef9f9fe 192.168.24.1:8787/rhosp14/openstack-haproxy:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-haproxy 427b0c9083fe 192.168.24.1:8787/rhosp14/openstack-heat-api-cfn:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-heat_api_cfn adb78babf622 192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-swift 14fa7978866c 192.168.24.1:8787/rhosp14/openstack-keystone:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-keystone e89c446cbf00 192.168.24.1:8787/rhosp14/openstack-cinder-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-cinder 64347367c712 192.168.24.1:8787/rhosp14/openstack-cron:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-crond 4e2cc781df8b 192.168.24.1:8787/rhosp14/openstack-glance-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-glance_api b691439b7411 192.168.24.1:8787/rhosp14/openstack-nova-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-nova_metadata caa45ecef46f 192.168.24.1:8787/rhosp14/openstack-panko-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-panko 7d3ce9503b58 192.168.24.1:8787/rhosp14/openstack-heat-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-heat 0bb529e943fe 192.168.24.1:8787/rhosp14/openstack-memcached:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-memcached b697bf3c992a 192.168.24.1:8787/rhosp14/openstack-heat-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-heat_api a95b67a62838 192.168.24.1:8787/rhosp14/openstack-iscsid:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-iscsid a4afbee7a5f0 192.168.24.1:8787/rhosp14/openstack-nova-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-nova 176a7dbff767 192.168.24.1:8787/rhosp14/openstack-gnocchi-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-gnocchi 3ff6ec25bbd9 192.168.24.1:8787/rhosp14/openstack-redis:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-redis 55c5bb55214c 192.168.24.1:8787/rhosp14/openstack-aodh-api:2018-12-05.2 "/var/lib/docker-p..." 11 minutes ago Exited (1) 11 minutes ago docker-puppet-aodh 0cf6e0dff859 192.168.24.1:8787/rhosp14/openstack-nova-placement-api:2018-12-05.2 "/var/lib/docker-p..." 12 minutes ago Exited (1) 11 minutes ago docker-puppet-nova_placement 61ec3593183b 192.168.24.1:8787/rhosp14/openstack-mariadb:2018-12-05.2 "/var/lib/docker-p..." 12 minutes ago Exited (1) 11 minutes ago docker-puppet-mysql 617313d5820c 192.168.24.1:8787/rhosp14/openstack-mariadb:2018-12-05.2 "/var/lib/docker-p..." 12 minutes ago Exited (1) 12 minutes ago docker-puppet-clustercheck 65b79cbb62f6 192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2018-12-05.2 "/var/lib/docker-p..." 12 minutes ago Exited (1) 12 minutes ago docker-puppet-swift_ringbuilder [root@controller-1 heat-admin]# docker logs -f docker-puppet-aodh + mkdir -p /etc/puppet + cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + rm -Rf /etc/puppet/ssl + echo '{"step": 6}' + TAGS= + '[' -n file,file_line,concat,augeas,cron,aodh_api_paste_ini,aodh_config,aodh_config,aodh_config,aodh_config ']' + TAGS='--tags file,file_line,concat,augeas,cron,aodh_api_paste_ini,aodh_config,aodh_config,aodh_config,aodh_config' + CHECK_MODE= + '[' -d /tmp/puppet-check-mode ']' + origin_of_time=/var/lib/config-data/aodh.origin_of_time + touch /var/lib/config-data/aodh.origin_of_time + sync + set +e + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + FACTER_hostname=controller-1 + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,aodh_api_paste_ini,aodh_config,aodh_config,aodh_config,aodh_config /etc/config.pp Error: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Permission denied - /usr/share/openstack-puppet/modules Error: Could not prepare for execution: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Permission denied - /usr/share/openstack-puppet/modules Permission denied - /usr/share/openstack-puppet/modules + rc=1 + set -e + '[' 1 -ne 2 -a 1 -ne 0 ']' + exit 1 [root@controller-1 heat-admin]# grep denied /var/log/audit/audit.log type=AVC msg=audit(1544204598.899:6881): avc: denied { read } for pid=35051 comm="puppet" name="65b79cbb62f6d75bed9280e575672324decdc68c6abe3f2f67f4a2e49040ea40" dev="tmpfs" ino=147921 scontext=system_u:system_r:container_t:s0:c67,c845 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204599.104:6883): avc: denied { read } for pid=35074 comm="puppet" name="617313d5820c4b3159b362b9338b8b1307f1c20473180e059aa02abb7e455046" dev="tmpfs" ino=149757 scontext=system_u:system_r:container_t:s0:c273,c525 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204600.797:6903): avc: denied { read } for pid=35282 comm="puppet" name="61ec3593183bed36b75b761956a3915c4b4424f4a47c7265256afd176109f312" dev="tmpfs" ino=154709 scontext=system_u:system_r:container_t:s0:c91,c94 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204601.732:6912): avc: denied { read } for pid=35357 comm="puppet" name="0cf6e0dff8596130bbcd8f1a1fe54431b57abfd3972d2a74798603b3734f7f33" dev="tmpfs" ino=155961 scontext=system_u:system_r:container_t:s0:c195,c215 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204603.746:6930): avc: denied { read } for pid=35620 comm="puppet" name="55c5bb55214c8271f6f91c5d5c02c64c6c3517305cbdf6449904e79e7b48bb3d" dev="tmpfs" ino=158550 scontext=system_u:system_r:container_t:s0:c304,c897 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204608.672:6972): avc: denied { read } for pid=35900 comm="puppet" name="3ff6ec25bbd9fc341dcf8e36226f8b66ce4fee1fe873dd26aeaec39a5e68674e" dev="tmpfs" ino=161820 scontext=system_u:system_r:container_t:s0:c658,c843 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204608.746:6973): avc: denied { read } for pid=35923 comm="puppet" name="176a7dbff7670027fb544447199f7f1f6393b006c5d189324a2117d09cf69a14" dev="tmpfs" ino=161845 scontext=system_u:system_r:container_t:s0:c545,c759 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204609.842:6990): avc: denied { read } for pid=36014 comm="puppet" name="a4afbee7a5f0df4d742874f9db2ade04b1a5ca663fb3a32064e8897b3d16752c" dev="tmpfs" ino=162874 scontext=system_u:system_r:container_t:s0:c334,c991 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204611.540:7003): avc: denied { read } for pid=36310 comm="puppet" name="a95b67a628381055dc973d5a4262a67e198224a3e3aa822a5bff2634c4e18762" dev="tmpfs" ino=164013 scontext=system_u:system_r:container_t:s0:c416,c884 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204614.081:7020): avc: denied { read } for pid=36480 comm="puppet" name="b697bf3c992a450f0fc66d29971a2f30ad7e724dd488a06094d7f92b3163973f" dev="tmpfs" ino=167388 scontext=system_u:system_r:container_t:s0:c474,c821 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204614.464:7029): avc: denied { read } for pid=36543 comm="puppet" name="0bb529e943fe327bcf239b3ef4020b0a33226410f8842c0e74410682e51d9014" dev="tmpfs" ino=167454 scontext=system_u:system_r:container_t:s0:c497,c998 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204615.830:7038): avc: denied { read } for pid=36710 comm="puppet" name="7d3ce9503b58f34acccd2a495913967e03c206bce84db222bbb1a588b1369f54" dev="tmpfs" ino=165570 scontext=system_u:system_r:container_t:s0:c76,c451 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204621.818:7053): avc: denied { read } for pid=36937 comm="puppet" name="caa45ecef46fea28d14f2e3dbf669e3844971cf27493283e191e9d3884ae4a71" dev="tmpfs" ino=163610 scontext=system_u:system_r:container_t:s0:c609,c807 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204623.885:7066): avc: denied { read } for pid=37079 comm="puppet" name="b691439b74112356724bf82d171efa214b897c8487215b0703ed38f8b31c1f3b" dev="tmpfs" ino=168099 scontext=system_u:system_r:container_t:s0:c420,c509 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204624.676:7075): avc: denied { read } for pid=37141 comm="puppet" name="4e2cc781df8b921dd5fc4c754a04be42dd4a46383159e631e4b38db895136810" dev="tmpfs" ino=170924 scontext=system_u:system_r:container_t:s0:c631,c790 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204626.869:7088): avc: denied { read } for pid=37386 comm="puppet" name="64347367c7120920aafb061e7694633a6f10548cc324203d79f925f915faa533" dev="tmpfs" ino=179246 scontext=system_u:system_r:container_t:s0:c461,c580 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204631.159:7105): avc: denied { read } for pid=37554 comm="puppet" name="e89c446cbf007c549abf476247d3da60037c8aab95b08b7ec922902da68c4158" dev="tmpfs" ino=177749 scontext=system_u:system_r:container_t:s0:c85,c721 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204632.460:7114): avc: denied { read } for pid=37678 comm="puppet" name="14fa7978866c804835e14b0af90d061de777e48b2169e14d354a0911ea80d95f" dev="tmpfs" ino=175779 scontext=system_u:system_r:container_t:s0:c671,c760 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204633.071:7123): avc: denied { read } for pid=37766 comm="puppet" name="adb78babf62276457de483adec841d70ce7a608be6cced06b8738de3b0efa152" dev="tmpfs" ino=177140 scontext=system_u:system_r:container_t:s0:c686,c878 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204636.667:7140): avc: denied { read } for pid=38053 comm="puppet" name="427b0c9083fec66f4eb329eb3f59db5f16b0fbf24bccf701cae540aa7be9c13e" dev="tmpfs" ino=173635 scontext=system_u:system_r:container_t:s0:c710,c967 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204637.067:7141): avc: denied { read } for pid=38118 comm="puppet" name="6ad67ef9f9fe654afe0c7dca339bdf65411c74fb4a536675a213d6a870406b0c" dev="tmpfs" ino=178877 scontext=system_u:system_r:container_t:s0:c331,c737 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204641.297:7158): avc: denied { read } for pid=38316 comm="puppet" name="bbfadd108db1122e8d91a36ef7bc5f24c388c26c81a4be2f84f323b22e92fb3d" dev="tmpfs" ino=185996 scontext=system_u:system_r:container_t:s0:c307,c884 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204641.613:7166): avc: denied { read } for pid=38379 comm="puppet" name="9e4d0cb8ed72707167b958f7657bbf0ac799eef7a7c0992fa386298bcddedc6e" dev="tmpfs" ino=191505 scontext=system_u:system_r:container_t:s0:c374,c696 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204650.895:7172): avc: denied { read } for pid=38580 comm="puppet" name="d0983e70b752f4640c156e5a9b6e3b04a70a93407b3e5453397981ee244a9211" dev="tmpfs" ino=193833 scontext=system_u:system_r:container_t:s0:c67,c424 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544204659.884:7185): avc: denied { read } for pid=38745 comm="puppet" name="1ac540d35bb57337ad9fa16122a335326ae40373d6c669537d2eb9033a5fcd39" dev="tmpfs" ino=191251 scontext=system_u:system_r:container_t:s0:c587,c986 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0
Hello Marius, Was it a new deploy, or an upgrade/update? In the first case, I would think it should work as I did a lot of work with the podman integration - it enforces selinux separation. In the second case, I think we might need to add upgrade_tasks in order to ensure the rights and selinux context for a couple of files, especially since a lot of directories/files were created by docker, as we were used to (ab)use the fact docker creates on-host missing trees... And they did get weird/wrong selinux label by that time. Care to give a bit more context? :) Cheers, C.
Hm, after some thoughts, we might need to do some backports, as most of my work was done in Master... I'll try to get a list of the relevant commits today. It will probably hurt a bit.
There is a bug in docker-puppet.py in the way how we mount volumes in the containers. I've fixed it recently: https://review.openstack.org/#/c/623649/ and now everything works fine. I need probably 30 minutes to update the commit message there, and we can merge it.
We might need to backport the following changes in Rocky (first is newest, last is oldest): Id1f65369f86425e083dfa538bde732a4f246511f https://review.openstack.org/#/c/619565/ I47756939fd933788b85bbd141d3c3bd870445efa https://review.openstack.org/#/c/618714/ I6f8dc49ff556215a25e8bb23ae8da63ce607d70c https://review.openstack.org/#/c/615956/ Icde6c61a0b26741946d079b2b00475de34722bea https://review.openstack.org/#/c/611801/ I44881508b01407e8fdd754cc5872babbc70d422e https://review.openstack.org/#/c/611110/ I4bfa2e1d3fe6c968c4d4a2ee1c2d4fb00a1667a1 https://review.openstack.org/#/c/607557/ I70ff5acd7913f9c5f5ead2d9dee83bab49f1f949 https://review.openstack.org/#/c/607147/ I3d63d1df7496d3b8a5883b07e9d40aa21153c086 https://review.openstack.org/#/c/605452/ I71e638bedde3836e05cffab53ad80bfd35313a31 https://review.openstack.org/#/c/605446/ Ia2cd08b9b7950ebca4d75938ae4329641c2d6f7c https://review.openstack.org/#/c/605450/ !! and its dependency: Ic9076a0a1a8e1360495dcf0eb766118ec63dc362 https://review.openstack.org/#/c/607130/ (tripleo-common) Ia00219337737dca87f745af5519effc04ce0a620 https://review.openstack.org/#/c/600535/ I284126db5dcf9dc31ee5ee640b2684643ef3a066 https://review.openstack.org/#/c/600534/ (the depend-on is a revert for a patch that was NOT included in Rocky) If70da9804d8a26fff594f7282f64318fd6b79e2c https://review.openstack.org/#/c/600533/ I504b52a2bb3c89e75ac3402f259c317889c054e6 https://review.openstack.org/#/c/605039/ Ic1bede203e8199a296944273cb334027dab940fe https://review.openstack.org/#/c/600532/ Question: do we really want that? It's a pretty huge list, and I'm pretty sure we can get some merge issues - also, I did take *my* patches, but I think others also added some stuff that impact SELinux... So we might need some more :/.
@Mike: did you test the change in Master with podman? Although that specific change might work as well, not 100% sure though. Please, don't break podman ;).
(In reply to Cédric Jeanneret from comment #4) > Hello Marius, > > Was it a new deploy, or an upgrade/update? This is a new deployment. > In the first case, I would think it should work as I did a lot of work with > the podman integration - it enforces selinux separation. > In the second case, I think we might need to add upgrade_tasks in order to > ensure the rights and selinux context for a couple of files, especially > since a lot of directories/files were created by docker, as we were used to > (ab)use the fact docker creates on-host missing trees... And they did get > weird/wrong selinux label by that time. The environment was an OSP14 so probably the podman integration doesn't apply here. > Care to give a bit more context? :) > > Cheers, > > C.
Coming back. With those two patches: - https://review.openstack.org/#/c/623649/ - https://review.openstack.org/#/c/623541/ I could get an undercloud + overcloud deployed On the undercloud, I get this from the `docker info` command output: Security Options: seccomp Profile: /etc/docker/seccomp.json selinux From one of the overcloud nodes: Security Options: seccomp WARNING: You're not using the default seccomp profile Profile: /etc/docker/seccomp.json selinux So I think we're good with those two patches - of course, more work will be needed, especially to know *what* will be done in case of an update (i.e. if pushed to z-stream instead of ga). I suspect it will require a dockerd restart, meaning a full container restart, thus possible downtime. To be investigated.
Current status: We had to revert upstream patches because the version of openstack-selinux wasn't recent enough to enable SElinux in the containers on the overcloud. openstack-selinux-0.8.15-1.el7ost.noarch works, openstack-selinux-0.8.17-0.20190116193749.faef39f.el7.noarch does not. We need to check what version we have in upstream rocky and also in OSP14. Then revert the reverts and then we need to submit the downstream backports.
We now have openstack-selinux-0.8.18-0.20190312025909.c0dd7b6.el7.noarch - so need to check if that version works fine.
I proposed the upstream reverts, let's see how it pass CI and local tests. Will update BZ later.