Panko user and panko endpoints are missing. Last deployment that panko user and panko endpoints were exist was OSP16 build RHOS_TRUNK-16.0-RHEL-8-20200226.n.1 e.g. Feb 26th. After that both OSP16 and OSP 16.1 are missing panko endpoints and panko users. Panko containers are exist and running. Here is the output from OSP 16 RHOS_TRUNK-16.0-RHEL-8-20200226.n.1 ------------------------------------------------------------------- (undercloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep panko | 5606df3cb3de452dbda674c24636e754 | regionOne | panko | event | True | admin | http://192.168.24.3:8977 | | 617ffd4ef1bd4dc9bd6bc15a25bb47f7 | regionOne | panko | event | True | public | https://192.168.24.2:13977 | | b8bb82e54661429094b467c58e35420b | regionOne | panko | event | True | internal | http://192.168.24.3:8977 | (undercloud) [stack@undercloud-0 ~]$ openstack user list | grep panko | 1726506fb78e4215a7a329d26dae20c5 | panko | (undercloud) [stack@undercloud-0 ~]$ podman ps | grep panko 80cfe44f735f undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:20200226.1 kolla_start 57 minutes ago Up 57 minutes ago panko_api b426d3554304 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:20200226.1 kolla_start 57 minutes ago Up 57 minutes ago panko_api_cron (undercloud) [stack@undercloud-0 ~]$ RHOS-16.1-RHEL-8-20200428.n.0 ------------------------------- (undercloud) [stack@undercloud-0 ~]$ openstack user list | grep panko (undercloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep panko (undercloud) [stack@undercloud-0 ~]$ sudo podman ps | grep panko 045839c0fd66 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:16.1_20200428.1 kolla_start 18 hours ago Up 18 hours ago panko_api 04d1f17088b6 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:16.1_20200428.1 kolla_start 18 hours ago Up 18 hours ago panko_api_cron
I have compared panko rpms between OSP16 puddles that panko user and endpoint were exist and later puddles that it don't exist and RPMs appeared to be the same.
The only thing I can see when diffing the prepared build trees between these puddles for puppet-tripleo openstack-tripleo-heat-templates and openstack-tripleo-common is in ceilometer-write-qdr.yaml, CeilometerEnablePanko: false was added to parameter_defaults, but I wouldn't think that would cause what you're seeing. Beyond that, though, I don't see any changes in the patches included or the tarballs of the source trees that has much that's different that's related to panko.
There is a slight difference in puppet-tripleo in manifests/profile/base/keystone.pp: >>>>>> if $::hostname == downcase($bootstrap_node) { <snip> $manage_endpoint = true <snip> } else { <snip> $manage_endpoint = false <snip> } ====== if $::hostname == downcase($bootstrap_node) and $keystone_resources_managed { <snip> $manage_endpoint = true <snip> } else { <snip> $manage_endpoint = false <snip> } <<<<<< With later block: if $step == 3 and $manage_endpoint { include ::keystone::endpoint <snip> if hiera('panko_api_enabled', false) { include ::panko::keystone::auth } <snip> The $keystone_resources_managed is documented as: # [*keystone_resources_managed*] # (Optional) Enable the management of Keystone resources with Puppet. # Can be disabled if Ansible manages these resources instead of Puppet. # The resources are: endpoints, roles, services, projects, users and their # assignment. # Defaults to hiera('keystone_resources_managed', true) # From [1] we can see that keystone resources are not managed by puppet any more, so we can blame person transferring resource management from puppet to ansible for forgetting about panko :). I think I figured out how to add this information to tripleo-ansible, will submit patch shortly. [1] https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/keystone/keystone-container-puppet.yaml#L382
Patch works: Overcloud configuration completed. Waiting for messages on queue 'tripleo' with no timeout. Overcloud Endpoint: https://10.0.0.101:13000 Overcloud Horizon Dashboard URL: https://10.0.0.101:443/dashboard Overcloud rc file: /home/stack/overcloudrc Overcloud Deployed sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 48472)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 45178), raddr=('192.168.24.2', 13004)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 37690), raddr=('192.168.24.2', 13989)> (undercloud) [stack@undercloud-0 ~]$ source overcloudrc (overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep panko | 5c25d2211b1f4f208b1dfc510cb4e393 | regionOne | panko | event | True | admin | http://172.17.1.117:8977 | | 736380d8aaae4668856a82369d85e943 | regionOne | panko | event | True | public | https://10.0.0.101:13977 | | cee51a1e3f3d47299e020233d1e37d80 | regionOne | panko | event | True | internal | http://172.17.1.117:8977 | (overcloud) [stack@undercloud-0 ~]$ openstack user list | grep panko | 505fa34f416145ab88ff61a732a2b3db | panko | Submitting DS backport
verified by automation tests.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148