Bug 1517903
| Summary: | after reboot TLS everywhere cluster swift_proxy_tls_proxy and glance_api_tls_proxy are Restarting | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> |
| Component: | openstack-containers | Assignee: | Martin André <m.andre> |
| Status: | CLOSED ERRATA | QA Contact: | Artem Hrechanychenko <ahrechan> |
| Severity: | urgent | Docs Contact: | Andrew Burden <aburden> |
| Priority: | urgent | ||
| Version: | 12.0 (Pike) | CC: | dprince, jschluet, m.andre, mariel, mburns, rhallise, sasha, sgordon, slinaber, tvignaud, ushkalim |
| Target Milestone: | ga | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-common-7.6.3-7.el7ost openstack-ec2-api-docker-12.0-20171201.1 openstack-glance-api-docker-12.0-20171201.1 openstack-neutron-server-docker-12.0-20171201.1 openstack-swift-proxy-server-docker-12.0-20171201.1 openstack-mistral-api-docker-1 | Doc Type: | Bug Fix |
| Doc Text: |
Previously, if containers were shut down unexpectedly, Apache still left runtime files in the containers, which causes the containers to stay in a Restarting state after the host comes back up. If you use TLS everywhere, this means that the Glance and Swift services were unreachable after the host rebooted.
This fix adds runtime cleanup in the container images startup scripts. Glance and Swift services are now functioning normally after the host reboots when deployed with TLS everywhere.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 22:23:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1506020 | ||
It is possible to patch the container images locally using the following commands (here for glance-api image):
(undercloud) [stack@undercloud-0 ~]$ cat glance_api_tls_workaround/Dockerfile
FROM 192.168.24.1:8787/rhosp12/openstack-glance-api:12.0-20171122.1
USER root
RUN sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf \
&& sed -i -r 's,^(Listen 443),#\1,' /etc/httpd/conf.d/ssl.conf \
&& echo "if [[ "\$\(whoami\)" == 'root' ]]; then rm -rf /var/run/httpd/* /run/httpd/* /tmp/httpd*; fi" >> /usr/local/bin/kolla_glance_extend_start
USER glance
(undercloud) [stack@undercloud-0 ~]$ docker build --rm -t 192.168.24.1:8787/rhosp12/openstack-glance-api:workaround ~/glance_api_tls_workaround/
(undercloud) [stack@undercloud-0 ~]$ docker push 192.168.24.1:8787/rhosp12/openstack-glance-api:workaround
(In reply to Martin André from comment #4) > It is possible to patch the container images locally using the following > commands (here for glance-api image): > > (undercloud) [stack@undercloud-0 ~]$ cat > glance_api_tls_workaround/Dockerfile > FROM 192.168.24.1:8787/rhosp12/openstack-glance-api:12.0-20171122.1 > USER root > RUN sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf \ > && sed -i -r 's,^(Listen 443),#\1,' /etc/httpd/conf.d/ssl.conf \ > && echo "if [[ "\$\(whoami\)" == 'root' ]]; then rm -rf /var/run/httpd/* > /run/httpd/* /tmp/httpd*; fi" >> /usr/local/bin/kolla_glance_extend_start > USER glance > > (undercloud) [stack@undercloud-0 ~]$ docker build --rm -t > 192.168.24.1:8787/rhosp12/openstack-glance-api:workaround > ~/glance_api_tls_workaround/ > > (undercloud) [stack@undercloud-0 ~]$ docker push > 192.168.24.1:8787/rhosp12/openstack-glance-api:workaround What about swift_tlx_proxy? (In reply to Artem Hrechanychenko from comment #5) > What about swift_tlx_proxy? This would be needed for all the container images where we install apache via our overrides file. See the upstream patch at https://review.openstack.org/#/c/523404/ for all the containers that need to be rebuilt. I've only posted here the procedure I used locally to patch the glance-api image as an example. FailedQA openstack-tripleo-common-7.6.3-7.el7ost.noarch b4423d0cfa5f 192.168.24.1:8787/rhosp12/openstack-glance-api:12.0-20171129.1 "kolla_start" 37 minutes ago Restarting (0) 2 minutes ago glance_api_tls_proxy 06c15e46b39e 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server:12.0-20171129.1 "kolla_start" 37 minutes ago Restarting (0) 2 minutes ago swift_proxy_tls_proxy INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc/httpd/conf.d/25-swift-proxy-api-proxy.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/25-swift-proxy-api-proxy.conf to /etc/httpd/conf.d/25-swift-proxy-api-proxy.conf INFO:__main__:Deleting /etc/httpd/conf.d/ssl.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/ssl.conf to /etc/httpd/conf.d/ssl.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/access_compat.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/access_compat.load to /etc/httpd/conf.modules.d/access_compat.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/actions.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/actions.load to /etc/httpd/conf.modules.d/actions.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/alias.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/alias.conf to /etc/httpd/conf.modules.d/alias.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/alias.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/alias.load to /etc/httpd/conf.modules.d/alias.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/auth_basic.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/auth_basic.load to /etc/httpd/conf.modules.d/auth_basic.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/auth_digest.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/auth_digest.load to /etc/httpd/conf.modules.d/auth_digest.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authn_anon.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authn_anon.load to /etc/httpd/conf.modules.d/authn_anon.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authn_core.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authn_core.load to /etc/httpd/conf.modules.d/authn_core.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authn_dbm.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authn_dbm.load to /etc/httpd/conf.modules.d/authn_dbm.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authn_file.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authn_file.load to /etc/httpd/conf.modules.d/authn_file.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authz_core.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authz_core.load to /etc/httpd/conf.modules.d/authz_core.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authz_dbm.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authz_dbm.load to /etc/httpd/conf.modules.d/authz_dbm.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authz_groupfile.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authz_groupfile.load to /etc/httpd/conf.modules.d/authz_groupfile.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authz_host.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authz_host.load to /etc/httpd/conf.modules.d/authz_host.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authz_owner.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authz_owner.load to /etc/httpd/conf.modules.d/authz_owner.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/authz_user.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/authz_user.load to /etc/httpd/conf.modules.d/authz_user.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/autoindex.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/autoindex.conf to /etc/httpd/conf.modules.d/autoindex.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/autoindex.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/autoindex.load to /etc/httpd/conf.modules.d/autoindex.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/cache.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/cache.load to /etc/httpd/conf.modules.d/cache.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/cgi.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/cgi.load to /etc/httpd/conf.modules.d/cgi.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/dav.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/dav.load to /etc/httpd/conf.modules.d/dav.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/dav_fs.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/dav_fs.conf to /etc/httpd/conf.modules.d/dav_fs.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/dav_fs.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/dav_fs.load to /etc/httpd/conf.modules.d/dav_fs.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/deflate.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/deflate.conf to /etc/httpd/conf.modules.d/deflate.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/deflate.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/deflate.load to /etc/httpd/conf.modules.d/deflate.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/dir.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/dir.conf to /etc/httpd/conf.modules.d/dir.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/dir.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/dir.load to /etc/httpd/conf.modules.d/dir.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/env.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/env.load to /etc/httpd/conf.modules.d/env.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/expires.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/expires.load to /etc/httpd/conf.modules.d/expires.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/ext_filter.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/ext_filter.load to /etc/httpd/conf.modules.d/ext_filter.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/filter.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/filter.load to /etc/httpd/conf.modules.d/filter.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/headers.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/headers.load to /etc/httpd/conf.modules.d/headers.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/include.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/include.load to /etc/httpd/conf.modules.d/include.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/log_config.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/log_config.load to /etc/httpd/conf.modules.d/log_config.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/logio.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/logio.load to /etc/httpd/conf.modules.d/logio.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/mime.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/mime.conf to /etc/httpd/conf.modules.d/mime.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/mime.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/mime.load to /etc/httpd/conf.modules.d/mime.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/mime_magic.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/mime_magic.conf to /etc/httpd/conf.modules.d/mime_magic.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/mime_magic.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/mime_magic.load to /etc/httpd/conf.modules.d/mime_magic.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/negotiation.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/negotiation.conf to /etc/httpd/conf.modules.d/negotiation.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/negotiation.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/negotiation.load to /etc/httpd/conf.modules.d/negotiation.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/prefork.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/prefork.conf to /etc/httpd/conf.modules.d/prefork.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/prefork.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/prefork.load to /etc/httpd/conf.modules.d/prefork.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/proxy.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/proxy.conf to /etc/httpd/conf.modules.d/proxy.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/proxy.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/proxy.load to /etc/httpd/conf.modules.d/proxy.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/proxy_http.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/proxy_http.load to /etc/httpd/conf.modules.d/proxy_http.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/rewrite.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/rewrite.load to /etc/httpd/conf.modules.d/rewrite.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/setenvif.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/setenvif.conf to /etc/httpd/conf.modules.d/setenvif.conf INFO:__main__:Deleting /etc/httpd/conf.modules.d/setenvif.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/setenvif.load to /etc/httpd/conf.modules.d/setenvif.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/socache_shmcb.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/socache_shmcb.load to /etc/httpd/conf.modules.d/socache_shmcb.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/speling.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/speling.load to /etc/httpd/conf.modules.d/speling.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/ssl.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/ssl.load to /etc/httpd/conf.modules.d/ssl.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/substitute.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/substitute.load to /etc/httpd/conf.modules.d/substitute.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/suexec.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/suexec.load to /etc/httpd/conf.modules.d/suexec.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/systemd.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/systemd.load to /etc/httpd/conf.modules.d/systemd.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/unixd.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/unixd.load to /etc/httpd/conf.modules.d/unixd.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/usertrack.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/usertrack.load to /etc/httpd/conf.modules.d/usertrack.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/version.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/version.load to /etc/httpd/conf.modules.d/version.load INFO:__main__:Deleting /etc/httpd/conf.modules.d/vhost_alias.load INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d/vhost_alias.load to /etc/httpd/conf.modules.d/vhost_alias.load INFO:__main__:Deleting /etc/httpd/conf/httpd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf/httpd.conf to /etc/httpd/conf/httpd.conf INFO:__main__:Deleting /etc/httpd/conf/ports.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf/ports.conf to /etc/httpd/conf/ports.conf INFO:__main__:Deleting /etc/swift/account-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account-server.conf to /etc/swift/account-server.conf INFO:__main__:Deleting /etc/swift/container-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container-server.conf to /etc/swift/container-server.conf INFO:__main__:Deleting /etc/swift/object-expirer.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-expirer.conf to /etc/swift/object-expirer.conf INFO:__main__:Deleting /etc/swift/object-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object-server.conf to /etc/swift/object-server.conf INFO:__main__:Deleting /etc/swift/proxy-server.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/proxy-server.conf to /etc/swift/proxy-server.conf INFO:__main__:Deleting /etc/swift/swift.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf INFO:__main__:Deleting /etc/swift/account.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.ring.gz to /etc/swift/account.ring.gz INFO:__main__:Deleting /etc/swift/container.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.ring.gz to /etc/swift/container.ring.gz INFO:__main__:Deleting /etc/swift/object.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.ring.gz to /etc/swift/object.ring.gz INFO:__main__:Deleting /etc/swift/account.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/account.builder to /etc/swift/account.builder INFO:__main__:Deleting /etc/swift/container.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/container.builder to /etc/swift/container.builder INFO:__main__:Deleting /etc/swift/object.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/object.builder to /etc/swift/object.builder INFO:__main__:Deleting /etc/swift/backups/1512066633.account.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066633.account.builder to /etc/swift/backups/1512066633.account.builder INFO:__main__:Deleting /etc/swift/backups/1512066633.container.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066633.container.builder to /etc/swift/backups/1512066633.container.builder INFO:__main__:Deleting /etc/swift/backups/1512066633.object.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066633.object.builder to /etc/swift/backups/1512066633.object.builder INFO:__main__:Deleting /etc/swift/backups/1512066638.account.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066638.account.builder to /etc/swift/backups/1512066638.account.builder INFO:__main__:Deleting /etc/swift/backups/1512066638.account.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066638.account.ring.gz to /etc/swift/backups/1512066638.account.ring.gz INFO:__main__:Deleting /etc/swift/backups/1512066638.object.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066638.object.builder to /etc/swift/backups/1512066638.object.builder INFO:__main__:Deleting /etc/swift/backups/1512066638.object.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066638.object.ring.gz to /etc/swift/backups/1512066638.object.ring.gz INFO:__main__:Deleting /etc/swift/backups/1512066639.container.builder INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066639.container.builder to /etc/swift/backups/1512066639.container.builder INFO:__main__:Deleting /etc/swift/backups/1512066639.container.ring.gz INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/backups/1512066639.container.ring.gz to /etc/swift/backups/1512066639.container.ring.gz INFO:__main__:Deleting /etc/rsyncd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf INFO:__main__:Writing out command to execute Running command: '/usr/sbin/httpd -DFOREGROUND' [Thu Nov 30 19:27:58.198264 2017] [so:warn] [pid 1] AH01574: module access_compat_module is already loaded, skipping [Thu Nov 30 19:27:58.198758 2017] [so:warn] [pid 1] AH01574: module actions_module is already loaded, skipping [Thu Nov 30 19:27:58.198767 2017] [so:warn] [pid 1] AH01574: module alias_module is already loaded, skipping [Thu Nov 30 19:27:58.198874 2017] [so:warn] [pid 1] AH01574: module auth_basic_module is already loaded, skipping [Thu Nov 30 19:27:58.198882 2017] [so:warn] [pid 1] AH01574: module auth_digest_module is already loaded, skipping [Thu Nov 30 19:27:58.198888 2017] [so:warn] [pid 1] AH01574: module authn_anon_module is already loaded, skipping [Thu Nov 30 19:27:58.198894 2017] [so:warn] [pid 1] AH01574: module authn_core_module is already loaded, skipping [Thu Nov 30 19:27:58.198986 2017] [so:warn] [pid 1] AH01574: module authn_dbm_module is already loaded, skipping [Thu Nov 30 19:27:58.198998 2017] [so:warn] [pid 1] AH01574: module authn_file_module is already loaded, skipping [Thu Nov 30 19:27:58.199108 2017] [so:warn] [pid 1] AH01574: module authz_core_module is already loaded, skipping [Thu Nov 30 19:27:58.199229 2017] [so:warn] [pid 1] AH01574: module authz_dbm_module is already loaded, skipping [Thu Nov 30 19:27:58.199243 2017] [so:warn] [pid 1] AH01574: module authz_groupfile_module is already loaded, skipping [Thu Nov 30 19:27:58.199254 2017] [so:warn] [pid 1] AH01574: module authz_host_module is already loaded, skipping [Thu Nov 30 19:27:58.199260 2017] [so:warn] [pid 1] AH01574: module authz_owner_module is already loaded, skipping [Thu Nov 30 19:27:58.199267 2017] [so:warn] [pid 1] AH01574: module authz_user_module is already loaded, skipping [Thu Nov 30 19:27:58.199273 2017] [so:warn] [pid 1] AH01574: module autoindex_module is already loaded, skipping [Thu Nov 30 19:27:58.199281 2017] [so:warn] [pid 1] AH01574: module cache_module is already loaded, skipping [Thu Nov 30 19:27:58.199612 2017] [so:warn] [pid 1] AH01574: module deflate_module is already loaded, skipping [Thu Nov 30 19:27:58.199629 2017] [so:warn] [pid 1] AH01574: module dir_module is already loaded, skipping [Thu Nov 30 19:27:58.199825 2017] [so:warn] [pid 1] AH01574: module env_module is already loaded, skipping [Thu Nov 30 19:27:58.199838 2017] [so:warn] [pid 1] AH01574: module expires_module is already loaded, skipping [Thu Nov 30 19:27:58.199847 2017] [so:warn] [pid 1] AH01574: module ext_filter_module is already loaded, skipping [Thu Nov 30 19:27:58.199854 2017] [so:warn] [pid 1] AH01574: module filter_module is already loaded, skipping [Thu Nov 30 19:27:58.199860 2017] [so:warn] [pid 1] AH01574: module headers_module is already loaded, skipping [Thu Nov 30 19:27:58.199869 2017] [so:warn] [pid 1] AH01574: module include_module is already loaded, skipping [Thu Nov 30 19:27:58.199989 2017] [so:warn] [pid 1] AH01574: module log_config_module is already loaded, skipping [Thu Nov 30 19:27:58.200000 2017] [so:warn] [pid 1] AH01574: module logio_module is already loaded, skipping [Thu Nov 30 19:27:58.200009 2017] [so:warn] [pid 1] AH01574: module mime_magic_module is already loaded, skipping [Thu Nov 30 19:27:58.200018 2017] [so:warn] [pid 1] AH01574: module mime_module is already loaded, skipping [Thu Nov 30 19:27:58.200032 2017] [so:warn] [pid 1] AH01574: module negotiation_module is already loaded, skipping [Thu Nov 30 19:27:58.200239 2017] [so:warn] [pid 1] AH01574: module rewrite_module is already loaded, skipping [Thu Nov 30 19:27:58.200252 2017] [so:warn] [pid 1] AH01574: module setenvif_module is already loaded, skipping [Thu Nov 30 19:27:58.200645 2017] [so:warn] [pid 1] AH01574: module socache_shmcb_module is already loaded, skipping [Thu Nov 30 19:27:58.200763 2017] [so:warn] [pid 1] AH01574: module substitute_module is already loaded, skipping [Thu Nov 30 19:27:58.200779 2017] [so:warn] [pid 1] AH01574: module suexec_module is already loaded, skipping [Thu Nov 30 19:27:58.200881 2017] [so:warn] [pid 1] AH01574: module unixd_module is already loaded, skipping [Thu Nov 30 19:27:58.200980 2017] [so:warn] [pid 1] AH01574: module version_module is already loaded, skipping [Thu Nov 30 19:27:58.200993 2017] [so:warn] [pid 1] AH01574: module vhost_alias_module is already loaded, skipping [Thu Nov 30 19:27:58.201031 2017] [so:warn] [pid 1] AH01574: module dav_module is already loaded, skipping [Thu Nov 30 19:27:58.201041 2017] [so:warn] [pid 1] AH01574: module dav_fs_module is already loaded, skipping [Thu Nov 30 19:27:58.201936 2017] [so:warn] [pid 1] AH01574: module mpm_prefork_module is already loaded, skipping [Thu Nov 30 19:27:58.201968 2017] [so:warn] [pid 1] AH01574: module proxy_module is already loaded, skipping [Thu Nov 30 19:27:58.203300 2017] [so:warn] [pid 1] AH01574: module proxy_http_module is already loaded, skipping [Thu Nov 30 19:27:58.203612 2017] [so:warn] [pid 1] AH01574: module ssl_module is already loaded, skipping [Thu Nov 30 19:27:58.203646 2017] [so:warn] [pid 1] AH01574: module systemd_module is already loaded, skipping [Thu Nov 30 19:27:58.203716 2017] [so:warn] [pid 1] AH01574: module cgi_module is already loaded, skipping [Thu Nov 30 19:27:58.207916 2017] [alias:warn] [pid 1] AH00671: The Alias directive in /etc/httpd/conf.d/autoindex.conf at line 21 will probably never match because it overlaps an earlier Alias. httpd (pid 1) already running The fix is missing from the images tagged with 12.0-20171129.1:
(undercloud) [stack@undercloud-0 ~]$ docker run -ti --rm docker-registry.engineering.redhat.com/rhosp12/openstack-swift-proxy-server:12.0-20171129.1 cat /usr/local/bin/kolla_extend_start
(undercloud) [stack@undercloud-0 ~]$ docker run -ti --rm docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api:12.0-20171129.1 cat /usr/local/bin/kolla_glance_extend_start
#!/bin/bash
# Bootstrap and exit if KOLLA_BOOTSTRAP variable is set. This catches all cases
# of the KOLLA_BOOTSTRAP variable being set, including empty.
if [[ "${!KOLLA_BOOTSTRAP[@]}" ]]; then
glance-manage db_sync
glance-manage db_load_metadefs
exit 0
fi
(undercloud) [stack@undercloud-0 ~]$
Both images are missing the following line in their startup script:
if [[ "$(whoami)" == 'root' ]]; then rm -rf /var/run/httpd/* /run/httpd/* /tmp/httpd*; fi
openstack-ec2-api-docker
openstack-glance-api-docker
openstack-neutron-server-docker
openstack-swift-proxy-server-docker
4 Dockerfiles were updated
openstack-ec2-api
+RUN sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf \
+ && sed -i -r 's,^(Listen 443),#\1,' /etc/httpd/conf.d/ssl.conf \
+ && echo "if [[ "\$\(whoami\)" == 'root' ]]; then rm -rf /var/run/httpd/* /run/httpd/* /tmp/httpd*; fi" >> /usr/local/bin/kolla_extend_start
openstack-glance-api-docker
RUN mkdir -p /openstack && \
ln -s /usr/share/openstack-tripleo-common/healthcheck/glance-api /openstack/healthcheck && \
- chmod -R a+rx /openstack
+ chmod -R a+rx /openstack && \
+ sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf && \
+ sed -i -r 's,^(Listen 443),#\1,' /etc/httpd/conf.d/ssl.conf && \
+ echo "if [[ "\$\(whoami\)" == 'root' ]]; then rm -rf /var/run/httpd/* /run/httpd/* /tmp/httpd*; fi" >> /usr/local/bin/kolla_glance_extend_start
HEALTHCHECK CMD /openstack/healthcheck
openstack-neutron-server-docker
+RUN sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf \
+ && sed -i -r 's,^(Listen 443),#\1,' /etc/httpd/conf.d/ssl.conf \
+ && echo "if [[ "\$\(whoami\)" == 'root' ]]; then rm -rf /var/run/httpd/* /run/httpd/* /tmp/httpd*; fi" >> /usr/local/bin/kolla_neutron_extend_start
+
openstack-swift-proxy-docker
RUN mkdir -p /openstack && \
ln -s /usr/share/openstack-tripleo-common/healthcheck/swift-proxy /openstack/healthcheck && \
- chmod -R a+rx /openstack
+ chmod -R a+rx /openstack && \
+ sed -i -r 's,^(Listen 80),#\1,' /etc/httpd/conf/httpd.conf && \
+ sed -i -r 's,^(Listen 443),#\1,' /etc/httpd/conf.d/ssl.conf && \
+ echo "if [[ "\$\(whoami\)" == 'root' ]]; then rm -rf /var/run/httpd/* /run/httpd/* /tmp/httpd*; fi" >> /usr/local/bin/kolla_extend_start
HEALTHCHECK CMD /openstack/healthcheck
Also openstack-mistral-api-docker had similar set of changes VERIFIED openstack-tripleo-common-7.6.3-8.el7ost.noarch docker images from 12.0-20171201.1 https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/OT2_container_netiso_osp12_HA_TLS_everywhere/116/console [heat-admin@overcloud-controller-0 ~]$ sudo docker ps |grep tls d68ac20c0957 192.168.24.1:8787/rhosp12/openstack-glance-api:12.0-20171201.1 "kolla_start" 31 minutes ago Up 9 minutes (healthy) glance_api_tls_proxy e3e4731bcd86 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server:12.0-20171201.1 "kolla_start" 31 minutes ago Up 9 minutes (healthy) swift_proxy_tls_proxy 6281e0681a71 192.168.24.1:8787/rhosp12/openstack-redis:pcmklatest "kolla_start" 39 minutes ago Up 9 minutes redis_tls_proxy (undercloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.16 "sudo docker ps |grep tls " 8ca2e29e11c2 192.168.24.1:8787/rhosp12/openstack-glance-api:12.0-20171201.1 "kolla_start" 34 minutes ago Up 11 minutes (healthy) glance_api_tls_proxy 173c59b36b67 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server:12.0-20171201.1 "kolla_start" 34 minutes ago Up 11 minutes (healthy) swift_proxy_tls_proxy 4ddc6da9d081 192.168.24.1:8787/rhosp12/openstack-redis:pcmklatest "kolla_start" 46 minutes ago Up 11 minutes redis_tls_proxy 3d247f4a6c8b 192.168.24.1:8787/rhosp12/openstack-glance-api:12.0-20171201.1 "kolla_start" 34 minutes ago Up 12 minutes (healthy) glance_api_tls_proxy ef2b218d2f16 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server:12.0-20171201.1 "kolla_start" 35 minutes ago Up 12 minutes (healthy) swift_proxy_tls_proxy cf3173da40e1 192.168.24.1:8787/rhosp12/openstack-redis:pcmklatest "kolla_start" 46 minutes ago Up 12 minutes redis_tls_proxy Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |
Description of problem: after reboot TLS everywhere cluster swift_proxy_tls_proxy and glance_api_tls_proxy are Restarting and we cannot launch instance 16:21:41 + swift_upload 16:21:41 + echo 'Checking if the swift container already exists.' 16:21:41 + swift list 16:21:41 + grep -q c1 16:22:14 ('Connection aborted.', BadStatusLine("''",)) 16:22:14 There was an error running swift list. Exiting.... e602b7f2798c 192.168.24.1:8787/rhosp12/openstack-glance-api:12.0-20171122.1 "kolla_start" 39 minutes ago Restarting (0) 3 minutes ago glance_api_tls_proxy c96e6587d52b 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server:12.0-20171122.1 "kolla_start" 40 minutes ago Restarting (0) 3 minutes ago swift_proxy_tls_proxy http://pastebin.test.redhat.com/535414 12.0-20171122.1 How reproducible: Always puppet-openstacklib-11.3.0-2.el7ost.noarch openstack-swift-account-2.15.1-3.el7ost.noarch openstack-swift-proxy-2.15.1-3.el7ost.noarch openstack-ironic-api-9.1.2-3.el7ost.noarch openstack-tripleo-common-7.6.3-4.el7ost.noarch openstack-mistral-common-5.2.0-1.el7ost.noarch openstack-tripleo-validations-7.4.2-1.el7ost.noarch openstack-nova-api-16.0.2-2.el7ost.noarch openstack-nova-conductor-16.0.2-2.el7ost.noarch openstack-neutron-openvswitch-11.0.1-5.el7ost.noarch python-openstackclient-lang-3.12.0-1.el7ost.noarch openstack-heat-engine-9.0.1-3.el7ost.noarch openstack-mistral-executor-5.2.0-1.el7ost.noarch openstack-nova-common-16.0.2-2.el7ost.noarch openstack-heat-common-9.0.1-3.el7ost.noarch openstack-keystone-12.0.0-3.el7ost.noarch puppet-openstack_extras-11.3.0-2.el7ost.noarch openstack-tripleo-common-containers-7.6.3-4.el7ost.noarch openstack-tripleo-heat-templates-7.0.3-13.el7ost.noarch openstack-mistral-engine-5.2.0-1.el7ost.noarch openstack-zaqar-5.0.0-3.el7ost.noarch openstack-swift-container-2.15.1-3.el7ost.noarch openstack-heat-api-9.0.1-3.el7ost.noarch openstack-ironic-common-9.1.2-3.el7ost.noarch openstack-mistral-api-5.2.0-1.el7ost.noarch openstack-tripleo-puppet-elements-7.0.1-1.el7ost.noarch openstack-nova-scheduler-16.0.2-2.el7ost.noarch openstack-glance-15.0.0-3.el7ost.noarch openstack-puppet-modules-11.0.0-1.el7ost.noarch openstack-neutron-ml2-11.0.1-5.el7ost.noarch openstack-heat-api-cfn-9.0.1-3.el7ost.noarch openstack-tripleo-ui-7.4.3-4.el7ost.noarch openstack-nova-compute-16.0.2-2.el7ost.noarch openstack-neutron-11.0.1-5.el7ost.noarch openstack-ironic-conductor-9.1.2-3.el7ost.noarch python-openstackclient-3.12.0-1.el7ost.noarch openstack-tempest-17.1.0-1.el7ost.noarch openstack-selinux-0.8.11-1.el7ost.noarch openstack-nova-placement-api-16.0.2-2.el7ost.noarch openstack-swift-object-2.15.1-3.el7ost.noarch openstack-neutron-common-11.0.1-5.el7ost.noarch python-openstacksdk-0.9.17-1.el7ost.noarch openstack-ironic-inspector-6.0.0-3.el7ost.noarch openstack-tripleo-image-elements-7.0.1-1.el7ost.noarch Steps to Reproduce: Deploy TLS everywhere overcloud - http://tripleo.org/install/advanced_deployment/ssl.html#tls-everywhere-for-the-overcloud Restart OC check container status Actual results: swift_proxy_tls_proxy and glance_api_tls_proxy are Restarting