Bug 1753251 - [RHOSP 13] Redeployment of Overcloud Fails with "KeyError: 'passwords'" After Cancelling STACK_UPDATE Early
Summary: [RHOSP 13] Redeployment of Overcloud Fails with "KeyError: 'passwords'" After...
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Luke Short
QA Contact: Sasha Smolyak
Depends On:
TreeView+ depends on / blocked
Reported: 2019-09-18 13:34 UTC by vivek koul
Modified: 2019-11-07 14:02 UTC (History)
6 users (show)

Fixed In Version: python-tripleoclient-9.3.1-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-11-07 14:02:10 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Launchpad 1819672 None None None 2019-10-08 17:37:46 UTC
OpenStack gerrit 686202 'None' MERGED Handle a deployment that was manually cancelled. 2020-03-05 16:28:57 UTC
OpenStack gerrit 687659 'None' MERGED Handle a deployment that was manually cancelled. 2020-03-05 16:28:57 UTC
OpenStack gerrit 687661 'None' MERGED Handle a deployment that was manually cancelled. 2020-03-05 16:28:57 UTC
OpenStack gerrit 687662 'None' MERGED Handle a deployment that was manually cancelled. 2020-03-05 16:28:57 UTC
Red Hat Product Errata RHBA-2019:3794 None None None 2019-11-07 14:02:37 UTC

Description vivek koul 2019-09-18 13:34:58 UTC
Description of problem:

It's a very common issue if the deployment gets canceled for any reason/ctl C, when redeployed it gets failed with the below error.

Starting new HTTP connection (1): "GET /v1/AUTH_9ee3af68b5ec4bbd9fbfcfea344c67f2/overcloud/plan-environment.yaml HTTP/1.1" 200 233
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 973, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 428, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 453, in _deploy_tripleo_heat_templates
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py", line 170, in update_plan_from_templates
    passwords = _load_passwords(swift_client, name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py", line 256, in _load_passwords
    return plan_env['passwords']
KeyError: 'passwords'

I have tried a few workarounds, it did not work for me:

1) A workaround is to delete swift container :
Following the kcs[1] and bugzilla[2] deleted swift containers and redeployment gets failed with the issue mentioned in bugzilla[3]
[1] https://access.redhat.com/solutions/3714651
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1622725
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1638363

2) Tried upgrading undercloud but it does not reflect anything in plans.

3) Using the below command we were successfully able to populate password section in plan-environment.yaml but parameter_default section is still not there and hence failed:

openstack workflow execution create tripleo.plan_management.v1.update_deployment_plan '{"container": "overcloud"}'

Version-Release number of selected component (if applicable):

How reproducible:
Run the deployment command again and when it starts updating then plan press ctl C. 

Steps to Reproduce:

Actual results:

Expected results:
Need steps to regenerate the plan-environmnet.yaml or where ever it made changes after pressing CTL c

Additional info:
We also tried updating the plan but no use.
All these workarounds were done on our lab environment before suggesting the same to Cu
Deleting the overcloud is not possible as VM's are running on the environment.
If any data is required, please let me know.

Comment 11 errata-xmlrpc 2019-11-07 14:02:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.