Bug 1762318

Summary: A change in the tripleo-common zaqar message format is crashing tripleoclient
Product: Red Hat OpenStack Reporter: Dougal Matthews <dmatthew>
Component: python-tripleoclientAssignee: Adriano Petrich <apetrich>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: apetrich, fhubik, hbrock, jhajyahy, jschluet, jslagle, lmiccini, mburns, ramishra, slinaber
Target Milestone: z9Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-9.3.1-4.el7ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1763168 (view as bug list) Environment:
Last Closed: 2019-11-07 14:03:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1763168    

Description Dougal Matthews 2019-10-16 14:08:34 UTC
Description of problem:

The overcloud deploy command crashes with the output 'execution.



Version-Release number of selected component (if applicable):


How reproducible: 100%


Steps to Reproduce:
1. Attempt a overcloud deploy with --debug to get the full traceback.

Actual results:


Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: 612a6471-d049-4673-b7df-79744c8cf366                                                                                                                         [6/1859]│
Workflow Started                                                                                                                                                                                                                               │
'execution'                                                                                                                                                                                                                                    │
Traceback (most recent call last):                                                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand                                                                                                                                                            │
    result = cmd.run(parsed_args)                                                                                                                                                                                                              │
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run                                                                                                                                                            │
    super(Command, self).run(parsed_args)                                                                                                                                                                                                      │
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run                                                                                                                                                          │
    return super(Command, self).run(parsed_args)                                                                                                                                                                                               │  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run                                                                                                                                                                   │
    return_code = self.take_action(parsed_args) or 0                                                                                                                                                                                           │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1006, in take_action                                                                                                                                      │
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 435, in _deploy_tripleo_heat_templates_tmpdir                                                                                                             │
    new_tht_root, tht_root)                                                                                                                                                                                                                    │  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 532, in _deploy_tripleo_heat_templates                                                                                                                    │
    parsed_args.plan_environment_file)                                                                                                                                                                                                         │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 547, in _try_overcloud_deploy_with_compat_yaml                                                                                                            │
    plan_env_file)                                                                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 310, in _heat_deploy                                                                                                                                      │    skip_deploy_identifier=skip_deploy_identifier)                                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 78, in deploy_and_wait                                                                                                                                   │
    deploy(clients, **workflow_input)                                                                                                                                                                                                          │
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 53, in deploy                                                                                                                                            │
    360):                                                                                                                                                                                                                                      │
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 66, in wait_for_messages                                                                                                                                       │
    if payload['execution']['id'] != execution.id:                                                                                                                                                                                             │
KeyError: 'execution'                                                                                                                                                                                                                          │
clean_up DeployOvercloud: 'execution'                                                                                                                                                                                                          │Traceback (most recent call last):                                                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run                                                                                                                                                                   │
    ret_val = super(OpenStackShell, self).run(argv)                                                                                                                                                                                            │
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run                                                                                                                                                                       │
    result = self.run_subcommand(remainder)                                                                                                                                                                                                    │
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand                                                                                                                                                        │
    ret_value = super(OpenStackShell, self).run_subcommand(argv)                                                                                                                                                                               │
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand                                                                                                                                                            │
    result = cmd.run(parsed_args)                                                                                                                                                                                                              │
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run                                                                                                                                                            │
    super(Command, self).run(parsed_args)                                                                                                                                                                                                      │
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run                                                                                                                                                          │
    return super(Command, self).run(parsed_args)                                                                                                                                                                                               │
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run                                                                                                                                                                   │
    return_code = self.take_action(parsed_args) or 0                                                                                                                                                                                           │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1006, in take_action                                                                                                                                      │
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 435, in _deploy_tripleo_heat_templates_tmpdir                                                                                                             │
    new_tht_root, tht_root)                                                                                                                                                                                                                    │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 532, in _deploy_tripleo_heat_templates                                                                                                                    │
    parsed_args.plan_environment_file)                                                                                                                                                                                                         │  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 547, in _try_overcloud_deploy_with_compat_yaml                                                                                                            │
    plan_env_file)                                                                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 310, in _heat_deploy                                                                                                                                      │    skip_deploy_identifier=skip_deploy_identifier)                                                                                                                                                                                             │
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 78, in deploy_and_wait                                                                                                                                   │    deploy(clients, **workflow_input)                                                                                                                                                                                                          │
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 53, in deploy                                                                                                                                            │    360):                                                                                                                                                                                                                                      │
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 66, in wait_for_messages                                                                                                                                       │
    if payload['execution']['id'] != execution.id:                                                                                                                                                                                             │
KeyError: 'execution'                                                                                                                                                                                                                          │                                                                                                                                                                                                                                               │
END return value: 1

Comment 1 Dougal Matthews 2019-10-16 14:09:47 UTC
I think this was likely caused by this backport but I am still investigating https://review.opendev.org/#/c/680688/

Comment 4 Dougal Matthews 2019-10-16 15:16:48 UTC
The root cause of tis seems to be that we are missing a backport. So the original bug I described above and the exception have been previously fixed

Comment 8 Jad Haj Yahya 2019-10-22 11:43:46 UTC
Verified on 13  -p 2019-10-16.2

Comment 9 Alex McLeod 2019-10-31 11:29:22 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Comment 11 errata-xmlrpc 2019-11-07 14:03:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3794