Description of problem: (related bug https://bugzilla.redhat.com/show_bug.cgi?id=1645536) Octavia was overwriting certificate authority when updating a stack, breaking management of existing haproxy load balancers (amphorae). The fix to rhbz 1645536 (see related) was to not regenerate certificates on update. However, this means octavia cannot be added to existing clouds. Version-Release number of selected component (if applicable): All current director based Octavia deployments using self-signed autogenerated certificates. 13, 14, 15, and current 16. How reproducible: 100% Steps to Reproduce: Try to add Octavia to an existing deployment.
A workaround is to change "CREATE" to "UPDATE" in https://github.com/openstack/tripleo-heat-templates/blob/6112a21aa6d5136a55d9db180205ef5e1d5816bb/deployment/octavia/octavia-deployment-config.j2.yaml#L202 in the undercloud node. Once Octavia is enabled, please make sure to revert this change.
Hi Carlos Goncalves, I still found this issue on OSP16 today /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem missing on overcloud controller nodes: ls -la /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem ls: cannot access /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem: No such file or directory 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server [-] Exception during message handling: octavia.common.exceptions.CertificateGenerationException: Could not sign the certificate request: Failed to load CA Certificate /etc/octavia/certs/ca_01.pem. 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/certificates/generator/local.py", line 49, in _validate_cert 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server ca_cert = open(CONF.certificates.ca_certificate, 'rb').read() 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server FileNotFoundError: [Errno 2] No such file or directory: '/etc/octavia/certs/ca_01.pem' 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server > A workaround is to change "CREATE" to "UPDATE" in https://github.com/openstack/tripleo-heat-templates/blob/6112a21aa6d5136a55d9db180205ef5e1d5816bb/deployment/octavia/octavia-deployment-config.j2.yaml#L202 in the undercloud node. Once Octavia is enabled, please make sure to revert this change. After executing this workaround, It's looks Octavia running well, but I hit this error https://bugzilla.redhat.com/show_bug.cgi?id=1804578 when used PING health monitor for my LB Thanks
This issue is still present in OSP 16 today. Engineering is working on it. We're glad to learn you managed to resolve the issue with the workaround, great! I don't follow what the issue you're experiencing with PING type health monitors is. The BZ you pointed is a BZ against documentation to document that PING is discouraged and not supported in OSP. We also added a red warning box to the upstream documentation page calling out that PING "only check if the member is reachable and responds to ICMP echo requests. It will not detect if your application running on that instance is healthy or not." -- https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html#other-health-monitors
Thank you for the workaround, hopefully the bug is fixed soon. sorry for the misunderstanding, I'll update or search for correct BZ about my issue about Octavia PING health monitor test fails
(In reply to Saputro Aryulianto from comment #2) > Hi Carlos Goncalves, > > I still found this issue on OSP16 today > > /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem > missing on overcloud controller nodes: > > ls -la > /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem > ls: cannot access > /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem: > No such file or directory > > 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server [-] Exception > during message handling: > octavia.common.exceptions.CertificateGenerationException: Could not sign the > certificate request: Failed to load CA Certificate > /etc/octavia/certs/ca_01.pem. > 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server Traceback (most > recent call last): > 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server File > "/usr/lib/python3.6/site-packages/octavia/certificates/generator/local.py", > line 49, in _validate_cert > 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server ca_cert = > open(CONF.certificates.ca_certificate, 'rb').read() > 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server > FileNotFoundError: [Errno 2] No such file or directory: > '/etc/octavia/certs/ca_01.pem' > 2020-04-21 08:37:55.388 23 ERROR oslo_messaging.rpc.server > > > A workaround is to change "CREATE" to "UPDATE" in https://github.com/openstack/tripleo-heat-templates/blob/6112a21aa6d5136a55d9db180205ef5e1d5816bb/deployment/octavia/octavia-deployment-config.j2.yaml#L202 in the undercloud node. Once Octavia is enabled, please make sure to revert this change. > > After executing this workaround, It's looks Octavia running well, but I hit > this error https://bugzilla.redhat.com/show_bug.cgi?id=1804578 when used > PING health monitor for my LB > > Thanks What log are these entries from ?
These verification steps seem to pertain to a different BZ. Could you please confirm and share verification steps associated to this BZ? Thanks.
Yes you're right. These are the right verification steps: Version verified: 16.1 -p RHOS-16.1-RHEL-8-20201005.n.0 1) Adding octavia yaml to overcloud_deploy.sh and re-running overcloud_deploy.sh (update) - Successful without errors: Ansible passed. Overcloud configuration completed. Overcloud Endpoint: http://10.0.0.119:5000 Overcloud Horizon Dashboard URL: http://10.0.0.119:80/dashboard Overcloud rc file: /home/stack/overcloudrc Overcloud Deployed without error Checked that Octavia is stated as a service and containers are up: openstack service list | grep octavia | 31b153c079cd4b8e945c2243870b8fc9 | octavia | load-balancer | [root@controller-2 ~]# podman ps | grep octavia c124fb3185ca undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-octavia-worker:16.1_20201003.1 kolla_start 9 hours ago Up 9 hours ago octavia_worker 18b083875e68 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-octavia-housekeeping:16.1_20201003.1 kolla_start 9 hours ago Up 9 hours ago octavia_housekeeping 28a0ae50b125 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-octavia-health-manager:16.1_20201003.1 kolla_start 9 hours ago Up 9 hours ago octavia_health_manager 09505d508487 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-octavia-api:16.1_20201003.1 kolla_start 9 hours ago Up 9 hours ago octavia_driver_agent 376bf4018991 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-octavia-api:16.1_20201003.1 kolla_start 9 hours ago Up 9 hours ago octavia_api
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4284