Bug 1657078
| Summary: | Director deployed OCP 3.11: docker-puppet.py for keepalived and haproxy fails when re-running overcloud deploy after initial deployment | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | Mike Fedosin <mfedosin> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Marius Cornea <mcornea> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 14.0 (Rocky) | CC: | athomas, bperkins, dbecker, gfidente, ltomasbo, m.andre, mburns, mfedosin, morazi, racedoro | ||||
| Target Milestone: | rc | Keywords: | Triaged | ||||
| Target Release: | 14.0 (Rocky) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-heat-templates-9.0.1-0.20181013060905.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-01-11 11:55:18 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1512347 [details]
audit.log
This is most probably introduced by the removal of openshift_docker_selinux_enabled: false as part of the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1648493 I found the cause of the issue. In the docker-puppet.py script we try to relable /usr/share/openstack-puppet/modules by adding ":z" suffix in the end. https://github.com/openstack/tripleo-heat-templates/blob/stable/rocky/docker/docker-puppet.py#L332 Unfortunatelly this operation is forbidden in docker with enabled SELinux. Docker's error message is: Error response from daemon: error setting label on mount source '/usr/share/openstack-puppet/modules': SELinux relabeling of /usr/share/openstack-puppet/modules is not allowed: "Relabeling content in /usr is not allowed.". It leads to the fact that during the configuration the jobs fails with "Permission denied - /usr/share/openstack-puppet/modules" From my perspective there is no need to relable that folder since it's read-only. After removing ":s" there I was able to deploy the overcloud with enabled SELinux. Note: currently SELinux is disabled in tripleo by default: https://bugzilla.redhat.com/show_bug.cgi?id=1657321 If we enable it overcloud deployment fails in the same place. Here's the patch that enables SELinux: https://review.openstack.org/#/c/623541 Here's the fix that makes everything work: https://review.openstack.org/#/c/623649 No doc text required. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045 |
Description of problem: Director deployed OCP 3.11: docker-puppet.py for keepalived and haproxy fails when re-running overcloud deploy after initial deployment: We can see that the failure is caused by: "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,haproxy_config /etc/config.pp", "Error: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Permission denied - /usr/share/openstack-puppet/modules", "Error: Could not prepare for execution: Could not create resources for managing Puppet's files and directories in sections [:main, :agent, :ssl]: Permission denied - /usr/share/openstack-puppet/modules", "Permission denied - /usr/share/openstack-puppet/modules", We can also notice the following SElinux denials which appear to be related: [root@openshift-master-0 heat-admin]# grep denied /var/log/audit/audit.log type=AVC msg=audit(1544149222.195:19193): avc: denied { read } for pid=64980 comm="puppet" name="79c0a0162fb2c129196dc9f9fde2b44b3195b73a80485d05bd48a4772cca5e27" dev="tmpfs" ino=1356607 scontext=system_u:system_r:container_t:s0:c22,c364 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1544149222.246:19196): avc: denied { read } for pid=64986 comm="puppet" name="521e31f211a50e0e665da6c251613a7abaa3a65f34b7800aa6a90b760cb82804" dev="tmpfs" ino=1358896 scontext=system_u:system_r:container_t:s0:c737,c943 tcontext=system_u:object_r:container_var_run_t:s0 tclass=dir permissive=0 Version-Release number of selected component (if applicable): selinux-policy-targeted-3.13.1-229.el7_6.6.noarch libselinux-python-2.5-14.1.el7.x86_64 libselinux-ruby-2.5-14.1.el7.x86_64 selinux-policy-3.13.1-229.el7_6.6.noarch container-selinux-2.74-1.el7.noarch openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch openstack-selinux-0.8.15-1.el7ost.noarch libselinux-2.5-14.1.el7.x86_64 libselinux-utils-2.5-14.1.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Deploy OCP overcloud with 3 masters + 3 infra + 3 worker nodes 2. Re-run overcloud deploy Actual results: docker-puppet.py for keepalived and haproxy fail due to selinux Expected results: docker-puppet.py for keepalived and haproxy succeeds Additional info: Attaching audit logs from the master nodes.