Description of problem: After upgrading from OSP8 to OSP9 the new services available in OSP9 are not present in keystone catalog (user, service and endpoint) - aodh - gnocchi - sahara Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-2.0.0-33.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP8 2. Upgrade to OSP9 Actual results (after the upgrade to OSP9): $ keystone user-list +----------------------------------+------------+---------+--------------------------+ | id | name | enabled | email | +----------------------------------+------------+---------+--------------------------+ | 0b65776def3244838da71484636de0c8 | admin | True | admin | | 685c20df801344e3ab31a50df75465f1 | ceilometer | True | email=nobody | | 09d89c4362cf4e1692bbb7ebd1e20727 | cinder | True | email=nobody | | bd3506e2d41b43b09c0ef8b18ac85414 | cinderv2 | True | email=nobody | | 538eecd4a43745a4bd204894d867603c | glance | True | email=nobody | | d4282df3ed944638b2864e1cc6336dfa | heat | True | email=nobody | | fda90e9479894a609311950ced3b59d9 | neutron | True | email=nobody | | a7c03d1c3453418c9cdffd7b24060474 | nova | True | email=nobody | | 929b6782980e4c91b219cf4cea258509 | swift | True | email=nobody | +----------------------------------+------------+---------+--------------------------+ $ keystone service-list +----------------------------------+------------+---------------+------------------------------+ | id | name | type | description | +----------------------------------+------------+---------------+------------------------------+ | 8cfb5e8a1f9146c6b414bac23d0e985e | ceilometer | metering | Ceilometer Service | | 556d2dd838b94d88b9faa1e277a401e1 | cinder | volume | Cinder Volume Service | | 5ecc529dd8904666b0b736fbb7b4b564 | cinderv2 | volumev2 | Cinder Volume Service v2 | | c82527e6f23b496fb66e67bd8aa52e88 | glance | image | Glance Image Service | | f80bd235ad704af0b4dee88aa82e3508 | heat | orchestration | Heat Service | | 195ef51764354567b3f42a621aba9309 | keystone | identity | OpenStack Identity Service | | 4b60a8136bfd4dacb4ebc94530ecd80c | neutron | network | Neutron Service | | a6410a8c694b406880425772f77cdc3c | nova | compute | Nova Compute Service | | 40287f914ad6482289b7892aaf6f600a | swift | object-store | Swift Object Storage Service | +----------------------------------+------------+---------------+------------------------------+ $ keystone endpoint-list +----------------------------------+-----------+--------------------------------------------+----------------------------------------------+-------------------------------------------+----------------------------------+ | id | region | publicurl | internalurl | adminurl | service_id | +----------------------------------+-----------+--------------------------------------------+----------------------------------------------+-------------------------------------------+----------------------------------+ | 06546c8e6ba94390bbb0f68f59572dd8 | regionOne | http://10.0.0.4:9292/ | http://172.16.1.4:9292/ | http://172.16.1.4:9292/ | c82527e6f23b496fb66e67bd8aa52e88 | | 1d2897d84125440e86235d0fe7829b9d | regionOne | http://10.0.0.4:8776/v2/%(tenant_id)s | http://172.16.2.4:8776/v2/%(tenant_id)s | http://172.16.2.4:8776/v2/%(tenant_id)s | 5ecc529dd8904666b0b736fbb7b4b564 | | 346dae77bb3d40b891dae2b29d100fa5 | regionOne | http://10.0.0.4:5000/v2.0 | http://172.16.2.4:5000/v2.0 | http://192.0.2.6:35357/v2.0 | 195ef51764354567b3f42a621aba9309 | | 4e576e15209e4084aa9001ec80597b2b | regionOne | http://10.0.0.4:8777/ | http://172.16.2.4:8777/ | http://172.16.2.4:8777/ | 8cfb5e8a1f9146c6b414bac23d0e985e | | 6e84a0f93d5847c5a7742db4e6d7ebf6 | regionOne | http://10.0.0.4:8080/v1/AUTH_%(tenant_id)s | http://172.16.1.4:8080/v1/AUTH_%(tenant_id)s | http://172.16.1.4:8080/v1 | 40287f914ad6482289b7892aaf6f600a | | b0868c59c9f64b7997704c03cd681a9a | regionOne | http://10.0.0.4:9696/ | http://172.16.2.4:9696/ | http://172.16.2.4:9696/ | 4b60a8136bfd4dacb4ebc94530ecd80c | | b10c128ea6ab459fa4306942a3a42f7c | regionOne | http://10.0.0.4:8776/v1/%(tenant_id)s | http://172.16.2.4:8776/v1/%(tenant_id)s | http://172.16.2.4:8776/v1/%(tenant_id)s | 556d2dd838b94d88b9faa1e277a401e1 | | ccaed908e4884cb99e615c181e3b0704 | regionOne | http://10.0.0.4:8774/v2.1/$(tenant_id)s | http://172.16.2.4:8774/v2.1/$(tenant_id)s | http://172.16.2.4:8774/v2.1/$(tenant_id)s | a6410a8c694b406880425772f77cdc3c | | dd4d16e3e89344788267cffc3202b120 | regionOne | http://10.0.0.4:8004/v1/%(tenant_id)s | http://172.16.2.4:8004/v1/%(tenant_id)s | http://172.16.2.4:8004/v1/%(tenant_id)s | f80bd235ad704af0b4dee88aa82e3508 | +----------------------------------+-----------+--------------------------------------------+----------------------------------------------+-------------------------------------------+----------------------------------+ Expected results (classic OSP9 deployment): $ keystone user-list +----------------------------------+------------+---------+--------------------------+ | id | name | enabled | email | +----------------------------------+------------+---------+--------------------------+ | e3c017fa23b1450298325d881dada396 | admin | True | admin | | c5c0cfa06ed64beb97580cf6c3ccb45d | aodh | True | email=nobody | | 718743a9dda94ae1b48d24ee29915647 | ceilometer | True | email=nobody | | d64dc42aa81048e5b303a0914232b878 | cinder | True | email=nobody | | 29ebd0ffb3d347a483f082b2ffbd5fec | cinderv2 | True | email=nobody | | 2b8cf7e9a35849e7aba6fb1f202da02f | glance | True | email=nobody | | 8f72152c489340d2b41d9b2ae66c636d | gnocchi | True | email=nobody | | 78e5b23531c34cd1afbce405056df78d | heat | True | email=nobody | | 679a53117aa0422295204b24cb0b1665 | neutron | True | email=nobody | | 3ab0348e65674cc1998ff50bc48786e1 | nova | True | email=nobody | | d3978ba47ca94159b3e2b2c4cfb7d06f | sahara | True | email=nobody | | dd36de650c244f46a925448952c67e56 | swift | True | email=nobody | +----------------------------------+------------+---------+--------------------------+ $ keystone service-list +----------------------------------+------------+-----------------+------------------------------+ | id | name | type | description | +----------------------------------+------------+-----------------+------------------------------+ | 49d3fbfad3594e7d95919d7ee4cc1e17 | aodh | alarming | OpenStack Alarming Service | | 7b68e12cd7f14098a4ae827f2b1bac15 | ceilometer | metering | Ceilometer Service | | 2f9df21309964922bac0e549d69dea0c | cinder | volume | Cinder Volume Service | | 0651f9cb9a0845a9ae08d3c04a98944a | cinderv2 | volumev2 | Cinder Volume Service v2 | | 9cf87f4c685c4b6db819eb04fcf8ba82 | glance | image | Glance Image Service | | d07e7b2f2cd848758ccca36b8a4d8656 | gnocchi | metric | OpenStack Metric Service | | c107f0b424c94918b7942f3211d584a9 | heat | orchestration | Heat Service | | e8eff6de777f4ff98e55d32ee986ab3d | keystone | identity | OpenStack Identity Service | | bd349e71ccac4a898b74e3d0188a6d06 | neutron | network | Neutron Service | | a4b09dc87715488694861a914f84f832 | nova | compute | Nova Compute Service | | 5b653b1fe68e40b9a43d4789051885f1 | sahara | data-processing | Sahara Service | | 245f29807485429584a62dffd677bfa9 | swift | object-store | Swift Object Storage Service | +----------------------------------+------------+-----------------+------------------------------+ $ keystone endpoint-list +----------------------------------+-----------+--------------------------------------------+----------------------------------------------+-------------------------------------------+----------------------------------+ | id | region | publicurl | internalurl | adminurl | service_id | +----------------------------------+-----------+--------------------------------------------+----------------------------------------------+-------------------------------------------+----------------------------------+ | 016ec86831d241c39c85d0019089c2fd | regionOne | http://10.0.0.4:9292/ | http://172.16.1.4:9292/ | http://172.16.1.4:9292/ | 9cf87f4c685c4b6db819eb04fcf8ba82 | | 083a95edd86c43a295010249b1ce4b02 | regionOne | http://10.0.0.4:8776/v2/%(tenant_id)s | http://172.16.2.4:8776/v2/%(tenant_id)s | http://172.16.2.4:8776/v2/%(tenant_id)s | 0651f9cb9a0845a9ae08d3c04a98944a | | 2e968b20147648ee98ed23fd8e3b30e3 | regionOne | http://10.0.0.4:9696/ | http://172.16.2.4:9696/ | http://172.16.2.4:9696/ | bd349e71ccac4a898b74e3d0188a6d06 | | 3daa8471b876434b93f4e8788c97adbe | regionOne | http://10.0.0.4:8041/ | http://172.16.2.4:8041/ | http://172.16.2.4:8041/ | d07e7b2f2cd848758ccca36b8a4d8656 | | 5561e21d06d3436fad1a9ac80f850bac | regionOne | http://10.0.0.4:8042/ | http://172.16.2.4:8042/ | http://172.16.2.4:8042/ | 49d3fbfad3594e7d95919d7ee4cc1e17 | | 6098686eca484ad5a89abb79c2a72905 | regionOne | http://10.0.0.4:8386/v1.1/%(tenant_id)s | http://172.16.2.4:8386/v1.1/%(tenant_id)s | http://172.16.2.4:8386/v1.1/%(tenant_id)s | 5b653b1fe68e40b9a43d4789051885f1 | | 6dfc15f98e2d4d1c91d54b2dd2504971 | regionOne | http://10.0.0.4:8776/v1/%(tenant_id)s | http://172.16.2.4:8776/v1/%(tenant_id)s | http://172.16.2.4:8776/v1/%(tenant_id)s | 2f9df21309964922bac0e549d69dea0c | | 82e94391c0ea4c9597abbe39c2d6d8cc | regionOne | http://10.0.0.4:8777/ | http://172.16.2.4:8777/ | http://172.16.2.4:8777/ | 7b68e12cd7f14098a4ae827f2b1bac15 | | 8b6efa8024bb4af1b6e791f3e37e3459 | regionOne | http://10.0.0.4:8774/v2.1/$(tenant_id)s | http://172.16.2.4:8774/v2.1/$(tenant_id)s | http://172.16.2.4:8774/v2.1/$(tenant_id)s | a4b09dc87715488694861a914f84f832 | | a7edd41fd13f4965b479a05b2459baf8 | regionOne | http://10.0.0.4:5000/v2.0 | http://172.16.2.4:5000/v2.0 | http://192.0.2.6:35357/v2.0 | e8eff6de777f4ff98e55d32ee986ab3d | | d59c31bc7d8f444b93416e4274b59e46 | regionOne | http://10.0.0.4:8080/v1/AUTH_%(tenant_id)s | http://172.16.1.4:8080/v1/AUTH_%(tenant_id)s | http://172.16.1.4:8080/v1 | 245f29807485429584a62dffd677bfa9 | | e585a654a2ad4e50b9c17dcd1c105dab | regionOne | http://10.0.0.4:8004/v1/%(tenant_id)s | http://172.16.2.4:8004/v1/%(tenant_id)s | http://172.16.2.4:8004/v1/%(tenant_id)s | c107f0b424c94918b7942f3211d584a9 | +----------------------------------+-----------+--------------------------------------------+----------------------------------------------+-------------------------------------------+----------------------------------+
A new option has been introduced since mitaka to force the creation of the keystone catalog [1] even if it's a stack update So using the '--force-postconfig' option during the last deploy command solves the issue (users, services and endpoints are created) : ---------------- Stack overcloud UPDATE_COMPLETE /home/stack/.ssh/known_hosts updated. Original contents retained as /home/stack/.ssh/known_hosts.old Skipping "horizon" postconfig because it wasn't found in the endpoint map output PKI initialization in init-keystone is deprecated and will be removed. The following cert files already exist, use --rebuild to remove the existing files before regenerating: /etc/keystone/ssl/certs/ca.pem already exists /etc/keystone/ssl/private/signing_key.pem already exists /etc/keystone/ssl/certs/signing_cert.pem already exists Connection to 192.0.2.6 closed. Overcloud Endpoint: http://10.0.0.4:5000/v2.0 Overcloud Deployed ---------------- Should we modify the upgrade documentation to integrate this option in the last update command ? [2] [1] https://github.com/openstack/python-tripleoclient/commit/7931486f8792d2af3d9256aaaef62efc2221bf22 [2] https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Major-Upgrading_the_Overcloud-Finalization
Having run into the same issue during our upgrade from OSP 8 to OSP 9, we were able to work around the issue by including "--force-postconfig" along with "major-upgrade-aodh.yaml" See exact command below. openstack overcloud deploy --force-postconfig --templates /home/stack/templates -e /home/stack/templates/allnode_post_config_env.yaml -e /home/stack/templates/network-environment.yaml -e /home/stack/templates/firstboot.yaml -e /home/stack/templates/environments/enable-tls.yaml -e /home/stack/templates/environments/inject-trust-anchor.yaml -e /home/stack/templates/environments/major-upgrade-aodh.yaml --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --control-flavor control --compute-flavor compute --ntp-server xx.xx.xx.xx --neutron-disable-tunneling --neutron-network-type vlan --neutron-network-vlan-ranges 'datacentre:1000:1050,datacentre:101:101' --debug --libvirt-type kvm
Is this a current reproducible bug?
@randy AFAIK it's still 100% reproducible
Confirmed this is 100% reproducible
Randy, did you try workaround from comment 8?
Yes the workaround works, when will it be fixed?
Sofer, Can you take a look at this? A couple questions that I think need answering: 1. Is the option from comment 8 a reasonable fix/workaround that we should document? 2. Are there any consequences to be aware of with that option? 3. Is there a more generic fix being worked on?
Hi Mike, 1. this is the fix. It's a documentation is bug is there https://bugzilla.redhat.com/show_bug.cgi?id=1377475 2. It recreates the endpoints and is especially tailored for this kind of upgrade case where new endpoint are needed. 3. This is the generic fix.
Taken.
Sofer, added the --force-postconfig command to both the Aodh upgrade command and to the final converge command: https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Major-Upgrading_the_Overcloud-Aodh https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Major-Upgrading_the_Overcloud-Finalization The second --force-postconfig might not be necessary but it should cover everything in a final sweep just in case. Was there anything else required for this BZ?
Hi, well it's not necessary and doesn't match upstream which might be confusing (http://tripleo.org/post_deployment/upgrade.html) Furthermore the --force-postconfig is also added to the keystone step. I think that we should just follow the upstream doc here, and remove the force-postconfig from keystone and convergence step.
For reference I think this should be not be closed https://bugzilla.redhat.com/show_bug.cgi?id=1377475
The --force-postconfig option is needed in the convergence step because sahara in deployed during the 8->9 upgrade.
The documentation states that you should check pacemaker on the overcloud between each and every step of the upgrade process step. I agree and consider this a best practice as you want to be able to catch any issues on the overcloud and troubleshoot them on each step. Because of this, if the endpoints for AODH are not created at during the appropriate step then pacemaker will show errors. This is why I would recommend the --force-postconfig option to be run on the AODH step. Of course, I could be remembering this incorrectly. If I have a chance I will run through the process again and make sure that pacemaker is unhappy when the endpoints are missing.
Okay, based on Dimitri comments in comment #24, we'll leave the --force-postconfig at the last step for the sahara endpoints. However, as Sofer pointed out, there's an added --force-postconfig for keystome, which is a mistake. Removing this.
Okay, I've removed the --force-postconfig for the keystone migration step. So now the following should be: Aodh step (includes --force-postconfig): https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Major-Upgrading_the_Overcloud-Aodh Keystone step (does not include --force-postconfig): https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Major-Upgrading_the_Overcloud-Keystone Converge step (includes --force-postconfig for sahara): https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Major-Upgrading_the_Overcloud-Finalization Sofer, Dimitri, and Chris -- How does this look to you guys?
Oki, looks good to me. Missed the comment about sahara. Fully trust Dimitri on this one. Thanks a lot Dan.
Looks good to me. Thanks Dan.
Dan - Has the documentation been pushed publicly then?
@Ben - Yep, it has. Closing this BZ (unless any objections from Chris).