Bug 1765839
| Summary: | openstack undercloud install don't update the Certificate | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Luigi Tamagnone <ltamagno> |
| Component: | openstack-tripleo | Assignee: | Roger Heslop <rheslop> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jeremy Agee <jagee> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 13.0 (Queens) | CC: | alee, dwilde, ggrasza, jslagle, mburns, moguimar, rmascena |
| Target Milestone: | z13 | Keywords: | Documentation, Triaged, ZStream |
| Target Release: | 13.0 (Queens) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-07-28 17:11:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Luigi Tamagnone
2019-10-26 15:16:27 UTC
I'm not sure about this one, but I think we could do with input from DFG:Security. Let me know if this is wrong! I've done some more research into this issue and I believe I have isolated the root cause, my notes from my investigation are here[1] for the curious but the TLDR is that I was mistaken in my original assumption that haproxy was not being restarted, the issue lies in the bind mounted certificate file that haproxy is using. What I found is that the PEM file is being bind mounted into the haproxy container and due to the nature of bind mounts this is essentially acting as a hard link, liking the inode of the original file to the target inside of the container. Even though the original file is correctly replaced with the new certificate the container is still maintaining a link to the old inode:
(undercloud) [stack@undercloud ~]$ sudo stat /etc/pki/tls/private/overcloud_endpoint.pem
File: ‘/etc/pki/tls/private/overcloud_endpoint.pem’
Size: 3381 Blocks: 8 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 71375612 Links: 1
Access: (0440/-r--r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:cert_t:s0
Access: 2020-04-16 18:03:57.123790389 +0000
Modify: 2020-04-16 18:03:57.011785706 +0000
Change: 2020-04-16 18:03:57.329799001 +0000
Birth: -
(undercloud) [stack@undercloud ~]$ sudo podman exec -it $(sudo podman ps | awk '/haproxy/ { print $1}') stat /var/lib/kolla/config_files/src-tls//etc/pki/tls/private/overcloud_endpoint.pem
File: ‘/var/lib/kolla/config_files/src-tls//etc/pki/tls/private/overcloud_endpoint.pem’
Size: 7236 Blocks: 16 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 46261016 Links: 0
Access: (0440/-r--r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-04-16 18:03:54.787692730 +0000
Modify: 2020-04-15 21:31:43.557632361 +0000
Change: 2020-04-16 18:03:57.123790389 +0000
Birth: -
From what I can tell the only way to update that bind mounted link is to restart the container (which is why we don't see this on master as it fully restarts the haproxy container). I can see two possible fixes for this, the first is to change the postsave_cmd in haproxy-public-tls-certmonger.yaml[2] from the reload action to the restart action, or to bind mount a directory for the certificates rather than the file itself. The /etc/pki/tls/private directory can be problematic for bind mounting as there can be other keys in that directory that could be exposed should haproxy be compromised. There is an /etc/pki/tls/private/haproxy directory that is created that we could possibly bind mount and store the certificate in.
[1]: https://hackmd.io/U56lZfNMRZmbO3P3eCMouw
[2]: https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/haproxy/haproxy-public-tls-certmonger.yaml#L80
Changed running text to include restarting of HAproxy | Changes published: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/director_installation_and_usage/appe-ssltls_certificate_configuration |