Bug 1240461 - Running openstack overcloud deploy twice results in UPGRADE_FAILED
Summary: Running openstack overcloud deploy twice results in UPGRADE_FAILED
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-rdomanager-oscplugin
Version: Director
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ga
: Director
Assignee: Marek Aufart
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
: 1240123 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-07 00:26 UTC by Steve Baker
Modified: 2019-07-11 09:33 UTC (History)
17 users (show)

Fixed In Version: python-rdomanager-oscplugin-0.0.8-28.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the necessary parameters were not passed to Orchestration service correctly for an OpenStack Overcloud redeployment, resulting in an UPGRADE_FAILED message. With this update, a parameter 'Existing=True' is added to the Orchestration update parameters, resulting in a successful Overcloud deployment.
Clone Of:
Environment:
Last Closed: 2015-08-05 13:58:33 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Gerrithub.io 238958 None None None Never
Red Hat Bugzilla 1240123 None CLOSED Openstack overcloud deploy, cannot do a successful update of a deployment 2018-12-07 10:08:06 UTC
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Steve Baker 2015-07-07 00:26:50 UTC
After successfully running the following to a working overcloud:

  openstack overcloud deploy --use-tripleo-heat-templates

Running the above command again should just do a stack-update with no changes, however the update eventually fails with the following:

[cloud-user@undercloud ~]$ heat resource-list -n2 overcloud | grep "OS::Heat::StructuredDeployment " | grep UPDATE_FAILED
| 0                                           | 462c6dbc-73d2-41ef-b8a2-5f57a255d368          | OS::Heat::StructuredDeployment                    | UPDATE_FAILED   | 2015-07-07T00:23:39Z | ControllerOvercloudServicesDeployment_Step4 |

[cloud-user@undercloud ~]$ heat deployment-output-show --format raw 462c6dbc-73d2-41ef-b8a2-5f57a255d368 deploy_stderr
Warning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.
Error: Could not find data item keystone_signing_certificate in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/4918074e-d62c-44f3-a0cb-b9540f7e3a28.pp:426 on node overcloud-controller-0.localdomain
Error: Could not find data item keystone_signing_certificate in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/4918074e-d62c-44f3-a0cb-b9540f7e3a28.pp:426 on node overcloud-controller-0.localdomain

Comment 3 chris alfonso 2015-07-07 17:25:31 UTC
We probably need to take that flag out for GA.

Comment 4 Steve Baker 2015-07-08 04:15:34 UTC
I've just confirmed the exact same error message for occurs for both --use-tripleo-heat-templates and --plan overcloud stacks

Comment 5 Dougal Matthews 2015-07-08 08:38:39 UTC
I think the problem with --use-tripleo-heat-templates can be found here:

https://github.com/rdo-management/python-rdomanager-oscplugin/blob/master/rdomanager_oscplugin/v1/overcloud_deploy.py#L417-L422

The utils.create_environment_file re-creates the environment file on each deploy, and we only add the Keystone cert to it on a Heat CREATE. So the fix would be to update utils.create_environment_file to only create the file if it doesn't exist.

Comment 6 Jan Provaznik 2015-07-08 09:05:44 UTC
Because the env file generated by utils.create_environment_file stores mainly scale parameters I think this file should be updated on update operation too (even if it already exists), but the existing keystone params should be kept, not overwritten.

Comment 7 Marek Aufart 2015-07-08 11:54:38 UTC
(In reply to Steve Baker from comment #4)
> I've just confirmed the exact same error message for occurs for both
> --use-tripleo-heat-templates and --plan overcloud stacks

I tested it and tuskar repeated deploy worked. Have you destroyed heat stack overcloud after --use-tht and before running tuskar deploy?

Comment 8 Marek Aufart 2015-07-08 12:58:40 UTC
This bug should be fixed with https://bugzilla.redhat.com/show_bug.cgi?id=1240123

Comment 10 Mike Burns 2015-07-08 14:59:29 UTC
*** Bug 1240123 has been marked as a duplicate of this bug. ***

Comment 11 John Fulton 2015-07-09 18:24:37 UTC
A partner and I tried the fix from the above bug and tried manually applying the
config change as described by Graeme in the duplicate bug: 

 https://bugzilla.redhat.com/show_bug.cgi?id=1240123#c6

We too see from `heat stack-show overcloud` that the SigningCertificate IS being passed from hiera.datafiles (which is what bz 1240123 fixes). However, puppet still cannot access it and we get the same error above Is the fix incomplete? Let me know what I can provide to help. 

  John

Comment 13 Marek Aufart 2015-07-10 11:00:40 UTC
(In reply to John Fulton from comment #11)
> A partner and I tried the fix from the above bug and tried manually applying
> the
> config change as described by Graeme in the duplicate bug: 
> 
>  https://bugzilla.redhat.com/show_bug.cgi?id=1240123#c6
> 
> We too see from `heat stack-show overcloud` that the SigningCertificate IS
> being passed from hiera.datafiles (which is what bz 1240123 fixes). However,
> puppet still cannot access it and we get the same error above Is the fix
> incomplete? Let me know what I can provide to help. 
> 
>   John

Fix should work, but I am not sure about SigningCertificate etc.

Better ask patch author Graeme..

Comment 14 Graeme Gillies 2015-07-12 23:06:59 UTC
Hi,


All my patch does is ensure that the variables originally given to heat on first deploy are provided to subsequent redeploys.

John are you able to give me some more background about the problem you are experiencing with the partner? Is the first overcloud deploy working successfully, and subsequent deployments failing? Are you able to give us a copy of

heat stack-show overcloud

Somewhere private, which should contain all the keys and certs that are being generated?

Also, are you seeing the values for these parameters being written to disk in heira under /etc/puppet/hieradata/ on the controller nodes in question?

Regards,

Graeme

Comment 16 John Fulton 2015-07-13 00:20:08 UTC
> Is the first overcloud deploy working successfully, and subsequent
> deployments failing? 

No, they are all failing and the partner has yet to successfully
deploy an overcloud on bare metal using OSP-Director Beta 2. 

- He tried a first deployment using the images that he had built 
- We then attributed this to needing to you Red Hat provided images 
- He then removed those images from glance, imported ours (case 01470774)
- He then attempted to redeploy using our images and had this issue

[stack@manager ~]$ heat stack-list
+----+------------+--------------+---------------+
| id | stack_name | stack_status | creation_time |
+----+------------+--------------+---------------+
+----+------------+--------------+---------------+
[stack@manager ~]$ openstack baremetal introspection bulk start
Waiting for discovery to finish
Discovery for UUID 829da90a-a0af-4036-90cc-bfb183c90760 finished successfully.
Discovery for UUID 9ad871c5-9a1b-4bb6-86da-def2af416633 finished successfully.
Discovery for UUID e4906227-388a-407c-afa3-0c631ede5c95 finished successfully.
[stack@manager ~]$ openstack management plan list
+--------------------------------------+-----------+-------------+------------------------------------------------------------------+
| uuid                                 | name      | description | roles                                                            |
+--------------------------------------+-----------+-------------+------------------------------------------------------------------+
| 297198d2-39ac-438a-8e07-e44d6924080a | overcloud | None        | Compute, Ceph-Storage, Cinder-Storage, Controller, Swift-Storage |
+--------------------------------------+-----------+-------------+------------------------------------------------------------------+

<--- snip --->
/tmp/tmpcP1zUP/hieradata/common.yaml
/tmp/tmpcP1zUP/puppet/manifests/ringbuilder.pp
/tmp/tmpcP1zUP/extraconfig/post_deploy/default.yaml
/tmp/tmpcP1zUP/net-config-noop.yaml
/tmp/tmpcP1zUP/puppet/all-nodes-config.yaml
/tmp/tmpcP1zUP/firstboot/userdata_default.yaml

ERROR: openstack Heat Stack create failed.

[stack@manager ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| ede7daa2-e5da-46c4-9823-1ed8cca9fe6e | overcloud  | CREATE_FAILED | 2015-07-07T21:21:58Z |
+--------------------------------------+------------+---------------+----------------------+

[stack@manager ~]$

Comment 17 Graeme Gillies 2015-07-13 00:34:57 UTC
(In reply to John Fulton from comment #16)
> > Is the first overcloud deploy working successfully, and subsequent
> > deployments failing? 
> 
> No, they are all failing and the partner has yet to successfully
> deploy an overcloud on bare metal using OSP-Director Beta 2. 
> 
> - He tried a first deployment using the images that he had built 
> - We then attributed this to needing to you Red Hat provided images 
> - He then removed those images from glance, imported ours (case 01470774)
> - He then attempted to redeploy using our images and had this issue
> 
> [stack@manager ~]$ heat stack-list
> +----+------------+--------------+---------------+
> | id | stack_name | stack_status | creation_time |
> +----+------------+--------------+---------------+
> +----+------------+--------------+---------------+
> [stack@manager ~]$ openstack baremetal introspection bulk start
> Waiting for discovery to finish
> Discovery for UUID 829da90a-a0af-4036-90cc-bfb183c90760 finished
> successfully.
> Discovery for UUID 9ad871c5-9a1b-4bb6-86da-def2af416633 finished
> successfully.
> Discovery for UUID e4906227-388a-407c-afa3-0c631ede5c95 finished
> successfully.
> [stack@manager ~]$ openstack management plan list
> +--------------------------------------+-----------+-------------+-----------
> -------------------------------------------------------+
> | uuid                                 | name      | description | roles    
> |
> +--------------------------------------+-----------+-------------+-----------
> -------------------------------------------------------+
> | 297198d2-39ac-438a-8e07-e44d6924080a | overcloud | None        | Compute,
> Ceph-Storage, Cinder-Storage, Controller, Swift-Storage |
> +--------------------------------------+-----------+-------------+-----------
> -------------------------------------------------------+
> 
> <--- snip --->
> /tmp/tmpcP1zUP/hieradata/common.yaml
> /tmp/tmpcP1zUP/puppet/manifests/ringbuilder.pp
> /tmp/tmpcP1zUP/extraconfig/post_deploy/default.yaml
> /tmp/tmpcP1zUP/net-config-noop.yaml
> /tmp/tmpcP1zUP/puppet/all-nodes-config.yaml
> /tmp/tmpcP1zUP/firstboot/userdata_default.yaml
> 
> ERROR: openstack Heat Stack create failed.
> 
> [stack@manager ~]$ heat stack-list
> +--------------------------------------+------------+---------------+--------
> --------------+
> | id                                   | stack_name | stack_status  |
> creation_time        |
> +--------------------------------------+------------+---------------+--------
> --------------+
> | ede7daa2-e5da-46c4-9823-1ed8cca9fe6e | overcloud  | CREATE_FAILED |
> 2015-07-07T21:21:58Z |
> +--------------------------------------+------------+---------------+--------
> --------------+
> 
> [stack@manager ~]$

I would say this might have to be a separate bug lodged as this isn't related to doing overcloud deploy twice. In fact, the bug fix I did only does anything if an existing overcloud exists in heat stack-list when you run overcloud deploy.

Would probably be the best way to get more exposure and someone to properly work through the troubleshooting for the different occurrence here (initial deploy failing with keystone parameters not being passed through)

Comment 18 Marek Aufart 2015-07-14 11:38:56 UTC
Adding bug documentation (not sure if needed).

Comment 19 Alexander Chuzhoy 2015-07-21 19:55:15 UTC
Verified:

Environment:
instack-undercloud-2.1.2-21.el7ost.noarch


Deploying the overcloud several times results in "Overcloud Deployed".

Comment 21 errata-xmlrpc 2015-08-05 13:58:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549

Comment 22 John Fulton 2015-08-31 13:03:04 UTC
I agreed with the comment by Graeme Gillies on 2015-07-12 20:34:57 EDT. BZ closed as per errata.


Note You need to log in before you can comment on or make changes to this bug.