Bug 1935621
| Summary: | Injecting certificate with "podman cp" can break cluster monitoring and operation | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Damien Ciabrini <dciabrin> | |
| Component: | puppet-tripleo | Assignee: | Damien Ciabrini <dciabrin> | |
| Status: | CLOSED ERRATA | QA Contact: | David Rosenfeld <drosenfe> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 16.2 (Train) | CC: | aschultz, dabarzil, jjoyce, jmarcian, jschluet, lmiccini, ltoscano, satmakur, slinaber, tvignaud | |
| Target Milestone: | beta | Keywords: | Triaged | |
| Target Release: | 16.2 (Train on RHEL 8.4) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | puppet-tripleo-11.6.2-2.20210428172107.5c76ddc.el8ost.2 openstack-tripleo-heat-templates-11.5.1-2.20210506094805.28ae385.el8ost.1 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1969461 (view as bug list) | Environment: | ||
| Last Closed: | 2021-09-15 07:12:31 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1969461 | |||
|
Description
Damien Ciabrini
2021-03-05 09:55:37 UTC
Instruction for testing:
On a controller node, with haproxy container running.
1. check one current certificate for haproxy (e.g. internal API)
# md5sum /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
bf41675624ef5b748335bf4668d6f7c3 /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
2. generate a new certificate with certmonger, and wait a couple of seconds for the new cert to get injected in haproxy
# getcert resubmit -i haproxy-internal_api-cert
Resubmitting "haproxy-internal_api-cert" to "IPA".
3. verify that a new certificate has been retrieved by certmonger on the host
# md5sum /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
fa01cdc093402a172c16a2baa363ba99 /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
3. verify that the cert got injected in the container as expected
# podman exec $(podman ps -q --filter name=haproxy) md5sum /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
fa01cdc093402a172c16a2baa363ba99 /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
4. verify that haproxy reloaded its config after the certificate injection
# journalctl -t haproxy
Aug 11 10:30:39 controller-0.redhat.local haproxy[536933]: Proxy cinder started.
Aug 11 10:30:39 controller-0.redhat.local haproxy[536933]: Proxy glance_api started.
Aug 11 10:30:39 controller-0.redhat.local haproxy[536933]: Proxy haproxy.stats started.
[...]
5. double check that the cert injection uses "tar" instead of "podman cp" to fix the bz
# /usr/bin/certmonger-haproxy-refresh.sh reload internal_api
[...]
+ cat /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.crt /etc/ipa/ca.crt /etc/pki/tls/private/haproxy/overcloud-haproxy-internal_api.key
++ podman ps '--format={{.Names}}'
++ grep -w -E 'haproxy(-bundle-.*-[0-9]+)?'
+ haproxy_container_name=haproxy-bundle-podman-0
+ '[' reload == reload ']'
+ echo haproxy-bundle-podman-0
+ grep -q '^haproxy-bundle'
+ tar -c /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
+ podman exec -i haproxy-bundle-podman-0 tar -C / -xv
Verified: Verified in version: [heat-admin@controller-0 ~]$ rpm -qa|grep puppet-tripleo puppet-tripleo-11.6.2-2.20210603175724.f642d04.el8ost.1.noarch Steps were performed: [root@controller-0 ~]# date Sun Aug 22 20:06:30 UTC 2021 // Check md5sum of the current haproxy internal api certificate: [root@controller-0 ~]# md5sum /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem 3e9187b760be94888e44e9e0f9f6eb4c /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem // Generate a new certificate by certmonger: [root@controller-0 ~]# getcert resubmit -i haproxy-internal_api-cert Resubmitting "haproxy-internal_api-cert" to "IPA". // Wait a few seconds and verify that the previous action caused to haproxy configuration reload: [root@controller-0 ~]# journalctl -t haproxy ... Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy cinder started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy glance_api started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy haproxy.stats started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy heat_api started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy heat_cfn started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy horizon started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy keystone_admin started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy keystone_public started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy mysql started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy neutron started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy nova_metadata started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy nova_novncproxy started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy nova_osapi started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy placement started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy redis started. Aug 22 20:07:05 controller-0.redhat.local haproxy[536933]: Proxy swift_proxy_server started // verify that a new certificate has been retrieved by certmonger on the host: [root@controller-0 ~]# md5sum /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem a7ef9913940ffe3bc69dd4d4bb63ef2a /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem // Verify that the new certificate was injected to the haproxy container as expected: [root@controller-0 ~]# podman exec $(podman ps -q --filter name=haproxy) md5sum /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem a7ef9913940ffe3bc69dd4d4bb63ef2a /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem .pem // Ensure that certificate injection has been done by using "tar" and not "podman cp" command: [root@controller-0 ~]# date Sun Aug 22 20:18:02 UTC 2021 [root@controller-0 ~]# /usr/bin/certmonger-haproxy-refresh.sh reload internal_api ... ++ grep -w -E 'haproxy(-bundle-.*-[0-9]+)?' + haproxy_container_name=haproxy-bundle-podman-0 + '[' reload == reload ']' + echo haproxy-bundle-podman-0 + grep -q '^haproxy-bundle' + tar -c /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem + podman exec -i haproxy-bundle-podman-0 tar -C / -xv tar: Removing leading `/' from member names etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem ... // See (not nesaccery) that haproxy reloaded its configuration : [root@controller-0 ~]# journalctl -t haproxy ... Aug 22 20:18:16 controller-0.redhat.local haproxy[536933]: Proxy cinder started. Aug 22 20:18:16 controller-0.redhat.local haproxy[536933]: Proxy glance_api started. Aug 22 20:18:16 controller-0.redhat.local haproxy[536933]: Proxy haproxy.stats started. Aug 22 20:18:16 controller-0.redhat.local haproxy[536933]: Proxy heat_api started. Aug 22 20:18:16 controller-0.redhat.local haproxy[536933]: Proxy heat_cfn started. ... Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483 *** Bug 1847320 has been marked as a duplicate of this bug. *** |