Bug 1657078 - Director deployed OCP 3.11: docker-puppet.py for keepalived and haproxy fails when re-running overcloud deploy after initial deployment
Summary: Director deployed OCP 3.11: docker-puppet.py for keepalived and haproxy fails...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 14.0 (Rocky)
Assignee: Mike Fedosin
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-07 03:13 UTC by Marius Cornea
Modified: 2019-01-11 11:55 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060905.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-11 11:55:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
audit.log (2.39 MB, text/plain)
2018-12-07 03:15 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 623649 0 None None None 2018-12-10 07:46:40 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:55:28 UTC

Description Marius Cornea 2018-12-07 03:13:36 UTC
Description of problem:
Director deployed OCP 3.11: docker-puppet.py for keepalived and haproxy fails when re-running overcloud deploy after initial deployment:

We can see that the failure is caused by:

        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,haproxy_config /etc/config.pp", 
        "Error: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Permission denied - /usr/share/openstack-puppet/modules", 
        "Error: Could not prepare for execution: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Permission denied - /usr/share/openstack-puppet/modules", 
        "Permission denied - /usr/share/openstack-puppet/modules", 


We can also notice the following SElinux denials which appear to be related:

[root@openshift-master-0 heat-admin]# grep denied /var/log/audit/audit.log 
type=AVC msg=audit(1544149222.195:19193): avc:  denied  { read } for  pid=64980 comm="puppet" name="79c0a0162fb2c129196dc9f9fde2b44b3195b73a80485d05bd48a4772cca5e27" dev="tmpfs" ino=1356607 scontext=system_u:system_r:container_t:s0:c22,c364 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1544149222.246:19196): avc:  denied  { read } for  pid=64986 comm="puppet" name="521e31f211a50e0e665da6c251613a7abaa3a65f34b7800aa6a90b760cb82804" dev="tmpfs" ino=1358896 scontext=system_u:system_r:container_t:s0:c737,c943 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0


Version-Release number of selected component (if applicable):
selinux-policy-targeted-3.13.1-229.el7_6.6.noarch
libselinux-python-2.5-14.1.el7.x86_64
libselinux-ruby-2.5-14.1.el7.x86_64
selinux-policy-3.13.1-229.el7_6.6.noarch
container-selinux-2.74-1.el7.noarch
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
openstack-selinux-0.8.15-1.el7ost.noarch
libselinux-2.5-14.1.el7.x86_64
libselinux-utils-2.5-14.1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Deploy OCP overcloud with 3 masters + 3 infra + 3 worker nodes
2. Re-run overcloud deploy

Actual results:
docker-puppet.py for keepalived and haproxy fail due to selinux

Expected results:
docker-puppet.py for keepalived and haproxy succeeds

Additional info:
Attaching audit logs from the master nodes.

Comment 1 Marius Cornea 2018-12-07 03:15:34 UTC
Created attachment 1512347 [details]
audit.log

Comment 2 Marius Cornea 2018-12-07 03:16:44 UTC
This is most probably introduced by the removal of openshift_docker_selinux_enabled: false as part of the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1648493

Comment 3 Mike Fedosin 2018-12-08 17:19:09 UTC
I found the cause of the issue.
In the docker-puppet.py script we try to relable /usr/share/openstack-puppet/modules by adding ":z" suffix in the end.
https://github.com/openstack/tripleo-heat-templates/blob/stable/rocky/docker/docker-puppet.py#L332

Unfortunatelly this operation is forbidden in docker with enabled SELinux. Docker's error message is:
Error response from daemon: error setting label on mount source '/usr/share/openstack-puppet/modules': SELinux relabeling of /usr/share/openstack-puppet/modules is not allowed: "Relabeling content in /usr is not allowed.".
It leads to the fact that during the configuration the jobs fails with "Permission denied - /usr/share/openstack-puppet/modules"

From my perspective there is no need to relable that folder since it's read-only.
After removing ":s" there I was able to deploy the overcloud with enabled SELinux.

Note: currently SELinux is disabled in tripleo by default: https://bugzilla.redhat.com/show_bug.cgi?id=1657321
If we enable it overcloud deployment fails in the same place. Here's the patch that enables SELinux: https://review.openstack.org/#/c/623541
Here's the fix that makes everything work: https://review.openstack.org/#/c/623649

Comment 14 Martin André 2019-01-10 10:11:30 UTC
No doc text required.

Comment 15 errata-xmlrpc 2019-01-11 11:55:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.