Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1805507 - Mistral workflow: tripleo.overcloud.workflow_tasks.step2 (WorkflowTasks_Step2) is deleted causing failure on subsequent overcloud deployments.
Summary: Mistral workflow: tripleo.overcloud.workflow_tasks.step2 (WorkflowTasks_Step2...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Rabi Mishra
QA Contact: Jad Haj Yahya
URL:
Whiteboard:
: 1845480 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-20 22:03 UTC by Matt Flusche
Modified: 2020-06-24 11:44 UTC (History)
9 users (show)

Fixed In Version: openstack-heat-10.0.3-10.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-24 11:44:24 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 710060 0 None MERGED Handle OS::Mistral::Workflow resource replacement properly 2020-11-11 16:39:58 UTC
Red Hat Product Errata RHBA-2020:2723 0 None None None 2020-06-24 11:44:52 UTC

Description Matt Flusche 2020-02-20 22:03:46 UTC
Description of problem:

In this specific environment the mistral workflows: tripleo.overcloud.workflow_tasks.step2 & tripleo.overcloud.workflow_tasks.step5 are being removed even during a successful overcloud deployments.  This is causing the following error during subsequent deployments:

2020-02-19 14:06:31Z [overcloud-AllNodesDeploySteps-7xfftqh3grld.WorkflowTasks_Step2_Execution]: UPDATE_FAILED  StackValidationFailed: resources.WorkflowTasks_Step2_Execution: Property error: Properties.actions.CREATE.workflow: Error validating value 'tripleo.overcloud.workflow_tasks.step2': The Workflow (tripleo.overcloud.workflow_tasks.step2) could not be found.
2020-02-19 14:06:32Z [overcloud-AllNodesDeploySteps-7xfftqh3grld]: UPDATE_FAILED  Resource UPDATE failed: StackValidationFailed: resources.WorkflowTasks_Step2_Execution: Property error: Properties.actions.CREATE.workflow: Error validating value 'tripleo.overcloud.workflow_tasks.step2': The Workflow (tripleo.overcloud.workflow_tasks.ste
2020-02-19 14:06:32Z [AllNodesDeploySteps]: UPDATE_FAILED  StackValidationFailed: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2_Execution: Property error: Properties.actions.CREATE.workflow: Error validating value 'tripleo.overcloud.workflow_tasks.step2': The Workflow (tripleo.overcloud.workflow_tas
2020-02-19 14:06:33Z [overcloud]: UPDATE_FAILED  Resource UPDATE failed: StackValidationFailed: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2_Execution: Property error: Properties.actions.CREATE.workflow: Error validating value 'tripleo.overcloud.workflow_tasks.step2': The Workflow (triple

The work-around here is to mark the workflows as unhealthy in heat (before each deployment). The workflows are then recreated successfully and the deployment can complete.  So far this behavior cannot be reproduced in a lab.  

From the heat-engine logs we can see heat deleting the workflows during the overcloud deployment:

$ grep WorkflowTasks_Step 0720-heat-engine.log-20200219  |grep delete |grep complete
2020-02-18 20:01:42.492 52146 DEBUG heat.engine.scheduler [req-31219d30-c747-47e4-84a6-9482cb72fb68 - admin - default default] Task delete from MistralExternalResource "WorkflowTasks_Step5_Execution" [ecfaee96-4cfc-4747-a3a1-4c5d26aa9357] Stack "overcloud-AllNodesDeploySteps-7xfftqh3grld" [5fd70d60-7d6d-4dbd-82f4-2cc9be17d745] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:215
2020-02-18 20:01:45.974 52151 DEBUG heat.engine.scheduler [req-31219d30-c747-47e4-84a6-9482cb72fb68 - admin - default default] Task delete from Workflow "WorkflowTasks_Step5" [tripleo.overcloud.workflow_tasks.step5] Stack "overcloud-AllNodesDeploySteps-7xfftqh3grld" [5fd70d60-7d6d-4dbd-82f4-2cc9be17d745] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:215
2020-02-18 20:01:49.349 52149 DEBUG heat.engine.scheduler [req-31219d30-c747-47e4-84a6-9482cb72fb68 - admin - default default] Task delete from MistralExternalResource "WorkflowTasks_Step2_Execution" [4423d9b2-4ac7-4dea-b3d4-440d047e3e2f] Stack "overcloud-AllNodesDeploySteps-7xfftqh3grld" [5fd70d60-7d6d-4dbd-82f4-2cc9be17d745] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:215
2020-02-18 20:01:49.998 52153 DEBUG heat.engine.scheduler [req-31219d30-c747-47e4-84a6-9482cb72fb68 - admin - default default] Task delete from MistralExternalResource "WorkflowTasks_Step2_Execution" [98c7d3cf-722c-4e33-99fa-9a5370bf683a] Stack "overcloud-AllNodesDeploySteps-7xfftqh3grld" [5fd70d60-7d6d-4dbd-82f4-2cc9be17d745] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:215
2020-02-18 20:01:52.645 52148 DEBUG heat.engine.scheduler [req-31219d30-c747-47e4-84a6-9482cb72fb68 - admin - default default] Task delete from Workflow "WorkflowTasks_Step2" [tripleo.overcloud.workflow_tasks.step2] Stack "overcloud-AllNodesDeploySteps-7xfftqh3grld" [5fd70d60-7d6d-4dbd-82f4-2cc9be17d745] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:215


I'll attach the complete heat-engine.log to the case for review.  I would appreciate additional review of this log to understand why this occurs in this environment.


Version-Release number of selected component (if applicable):
$ grep heat installed-rpms 
heat-cfntools-1.3.0-2.el7ost.noarch                         Wed Oct 17 22:56:43 2018
openstack-heat-api-10.0.3-8.el7ost.noarch                   Wed Nov 20 10:02:59 2019
openstack-heat-api-cfn-10.0.3-8.el7ost.noarch               Wed Nov 20 10:02:59 2019
openstack-heat-common-10.0.3-8.el7ost.noarch                Wed Nov 20 10:02:29 2019
openstack-heat-engine-10.0.3-8.el7ost.noarch                Wed Nov 20 10:02:59 2019
openstack-tripleo-heat-templates-8.4.1-16.el7ost.noarch     Fri Dec 20 05:33:44 2019
puppet-heat-12.4.1-0.20190214021237.a7ed720.el7ost.noarch   Wed Nov 20 10:01:34 2019
python2-heatclient-1.14.1-1.el7ost.noarch                   Wed May  1 03:41:40 2019
python-heat-agent-1.5.4-1.el7ost.noarch                     Wed Nov 20 10:01:20 2019


How reproducible:
this specific environment - every deployment

Steps to Reproduce:
1. Unknown

Comment 15 Rabi Mishra 2020-06-10 04:30:33 UTC
*** Bug 1845480 has been marked as a duplicate of this bug. ***

Comment 20 errata-xmlrpc 2020-06-24 11:44:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2723


Note You need to log in before you can comment on or make changes to this bug.