Bug 1829696 - Panko user and panko endpoint are missing on OSP 16/OSP16.1 deployments (undercloud and overcloud)
Summary: Panko user and panko endpoint are missing on OSP 16/OSP16.1 deployments (unde...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.0 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 16.1 (Train on RHEL 8.2)
Assignee: Martin Magr
QA Contact: Leonid Natapov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-30 06:36 UTC by Leonid Natapov
Modified: 2020-07-29 07:52 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.3.2-0.20200616081526.396affd.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-29 07:52:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 737356 0 None MERGED [TRAIN-ONLY] Add keystone_resources for Panko 2020-12-09 21:21:12 UTC
Red Hat Product Errata RHBA-2020:3148 0 None None None 2020-07-29 07:52:42 UTC

Description Leonid Natapov 2020-04-30 06:36:08 UTC
Panko user and panko endpoints are missing.

Last deployment that panko user and panko endpoints were exist was OSP16 build RHOS_TRUNK-16.0-RHEL-8-20200226.n.1 e.g. Feb 26th.


After that both OSP16 and OSP 16.1 are missing panko endpoints and panko users.

Panko containers are exist and running.

Here is the output from OSP 16 RHOS_TRUNK-16.0-RHEL-8-20200226.n.1 
-------------------------------------------------------------------
(undercloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep panko
| 5606df3cb3de452dbda674c24636e754 | regionOne | panko            | event                   | True    | admin     | http://192.168.24.3:8977                         |
| 617ffd4ef1bd4dc9bd6bc15a25bb47f7 | regionOne | panko            | event                   | True    | public    | https://192.168.24.2:13977                       |
| b8bb82e54661429094b467c58e35420b | regionOne | panko            | event                   | True    | internal  | http://192.168.24.3:8977                         |
(undercloud) [stack@undercloud-0 ~]$ openstack user list | grep panko
| 1726506fb78e4215a7a329d26dae20c5 | panko                                                 |
(undercloud) [stack@undercloud-0 ~]$ podman ps | grep panko
80cfe44f735f  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:20200226.1                  kolla_start           57 minutes ago     Up 57 minutes ago            panko_api
b426d3554304  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:20200226.1                  kolla_start           57 minutes ago     Up 57 minutes ago            panko_api_cron
(undercloud) [stack@undercloud-0 ~]$




RHOS-16.1-RHEL-8-20200428.n.0
-------------------------------
(undercloud) [stack@undercloud-0 ~]$ openstack user list | grep panko
(undercloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep panko
(undercloud) [stack@undercloud-0 ~]$ sudo podman ps | grep panko
045839c0fd66  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:16.1_20200428.1                  kolla_start           18 hours ago  Up 18 hours ago         panko_api
04d1f17088b6  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-panko-api:16.1_20200428.1                  kolla_start           18 hours ago  Up 18 hours ago         panko_api_cron

Comment 4 Leonid Natapov 2020-05-11 06:06:03 UTC
I have compared panko rpms between OSP16 puddles that panko user and endpoint were exist and later puddles that it don't exist and RPMs appeared to be the same.

Comment 5 Ryan McCabe 2020-06-01 19:18:17 UTC
The only thing I can see when diffing the prepared build trees between these puddles for puppet-tripleo openstack-tripleo-heat-templates and openstack-tripleo-common is in ceilometer-write-qdr.yaml,

CeilometerEnablePanko: false

was added to parameter_defaults, but I wouldn't think that would cause what you're seeing. Beyond that, though, I don't see any changes in the patches included or the tarballs of the source trees that has much that's different that's related to panko.

Comment 6 Martin Magr 2020-06-22 18:40:08 UTC
There is a slight difference in puppet-tripleo in manifests/profile/base/keystone.pp:

>>>>>>
  if $::hostname == downcase($bootstrap_node) {
    <snip>
    $manage_endpoint = true
    <snip>
  } else {
    <snip>
    $manage_endpoint = false
    <snip>
  }
======
  if $::hostname == downcase($bootstrap_node) and $keystone_resources_managed {
    <snip>
    $manage_endpoint = true
    <snip>
  } else {
    <snip>
    $manage_endpoint = false
    <snip>
  }
<<<<<<

With later block:
  if $step == 3 and $manage_endpoint {
    include ::keystone::endpoint
  <snip>
    if hiera('panko_api_enabled', false) {
      include ::panko::keystone::auth
    }
  <snip>

The $keystone_resources_managed is documented as:
# [*keystone_resources_managed*]
#   (Optional) Enable the management of Keystone resources with Puppet.
#   Can be disabled if Ansible manages these resources instead of Puppet.
#   The resources are: endpoints, roles, services, projects, users and their
#   assignment.
#   Defaults to hiera('keystone_resources_managed', true)
#

From [1] we can see that keystone resources are not managed by puppet any more, so we can blame person transferring resource management from puppet to ansible for forgetting about panko :). I think I figured out how to add this information to tripleo-ansible, will submit patch shortly.


[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/keystone/keystone-container-puppet.yaml#L382

Comment 7 Martin Magr 2020-06-22 19:52:35 UTC
Patch works:

Overcloud configuration completed.
Waiting for messages on queue 'tripleo' with no timeout.
Overcloud Endpoint: https://10.0.0.101:13000
Overcloud Horizon Dashboard URL: https://10.0.0.101:443/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 48472)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 45178), raddr=('192.168.24.2', 13004)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 37690), raddr=('192.168.24.2', 13989)>
(undercloud) [stack@undercloud-0 ~]$ source overcloudrc 
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep panko
| 5c25d2211b1f4f208b1dfc510cb4e393 | regionOne | panko        | event          | True    | admin     | http://172.17.1.117:8977                       |
| 736380d8aaae4668856a82369d85e943 | regionOne | panko        | event          | True    | public    | https://10.0.0.101:13977                       |
| cee51a1e3f3d47299e020233d1e37d80 | regionOne | panko        | event          | True    | internal  | http://172.17.1.117:8977                       |
(overcloud) [stack@undercloud-0 ~]$ openstack user list | grep panko
| 505fa34f416145ab88ff61a732a2b3db | panko                   |

Submitting DS backport

Comment 17 Leonid Natapov 2020-06-29 07:23:47 UTC
verified by automation tests.

Comment 21 errata-xmlrpc 2020-07-29 07:52:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148


Note You need to log in before you can comment on or make changes to this bug.