Bug 1563628

Summary: [UPDATE osp13 update from core paddle 2018-03-16.1 fails with error resources.WorkflowTasks_Step2_Execution: ERROR
Product: Red Hat OpenStack Reporter: Raviv Bar-Tal <rbartal>
Component: rhosp-directorAssignee: Jiri Stransky <jstransk>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Raviv Bar-Tal <rbartal>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: ccamacho, dbecker, gfidente, johfulto, lbezdick, mburns, morazi, rbartal, rhel-osp-director-maint, sathlang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-23 20:38:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/home/stack
none
ceph sosreport
none
compute logs
none
controller logs none

Description Raviv Bar-Tal 2018-04-04 11:09:33 UTC 
Description of problem:
trying to update osp13 from 2018-03-16.1 to 2018-03-29.1 fails 
2018-04-04 10:39:58Z [overcloud-AllNodesDeploySteps-25fgux5nj775.WorkflowTasks_Step2_Execution]: UPDATE_COMPLETE  The Resource WorkflowTasks_Step2_Execution requires replacement.
2018-04-04 10:39:59Z [overcloud-AllNodesDeploySteps-25fgux5nj775.WorkflowTasks_Step2_Execution]: CREATE_IN_PROGRESS  state changed
2018-04-04 10:43:11Z [overcloud-AllNodesDeploySteps-25fgux5nj775.WorkflowTasks_Step2_Execution]: CREATE_FAILED  resources.WorkflowTasks_Step2_Execution: ERROR
2018-04-04 10:43:11Z [overcloud-AllNodesDeploySteps-25fgux5nj775]: UPDATE_FAILED  Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR
2018-04-04 10:43:11Z [AllNodesDeploySteps]: UPDATE_FAILED  resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR
2018-04-04 10:43:11Z [overcloud]: UPDATE_FAILED  Resource UPDATE failed: resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR

 Stack overcloud UPDATE_FAILED 

overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
  resource_type: OS::Mistral::ExternalResource
  physical_resource_id: 2110b7a1-08e8-4062-b56a-99c5ad5cfde7
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step2_Execution: ERROR
Heat Stack update failed.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. install osp13 puddle 2018-03-16.1
2. run the update procedure
3.

Actual results:
update failed

Expected results:


Additional info:
Comment 1 Raviv Bar-Tal 2018-04-04 11:15:00 UTC 
Created attachment 1417216 [details]
/home/stack
Comment 2 Raviv Bar-Tal 2018-04-04 11:26:02 UTC 
Created attachment 1417239 [details]
ceph sosreport
Comment 3 Raviv Bar-Tal 2018-04-04 11:29:29 UTC 
Created attachment 1417241 [details]
compute logs
Comment 4 Raviv Bar-Tal 2018-04-04 11:30:31 UTC 
Created attachment 1417243 [details]
controller logs
Comment 5 Jiri Stransky 2018-04-13 09:23:05 UTC 
Hi Raviv,

can you please provide some more info?

1) Mistral logs from undercloud would be useful (/var/log/mistral/*).

2) What were the `overcloud deploy` and `overcloud update` commands that were ran.

3) Was this an environment with Ceph? The error looks like it could be related to ceph-ansible, as WorkflowTasks_Step2_Execution is where ceph-ansible runs when it's enabled.

I understand we may not have the environment anymore but in case we hit this again, please add the info.
Comment 6 Lukas Bezdicka 2018-04-16 12:40:25 UTC 
Please add logs from /var/log/mistral on undercloud
Comment 7 Carlos Camacho 2018-04-23 13:53:58 UTC 
Hey Raviv can you add last logs Lukas asked?

Thanks!
Comment 8 John Fulton 2018-04-23 20:38:23 UTC 
Hi Raviv,

I'm just closing this for now, but I will be happy to re-open if you provide the data requested in 10 days ago in #5.

Thanks,
  John
Comment 9 Raviv Bar-Tal 2018-04-26 12:20:11 UTC 
Sure, I have mover to a newer build, which doe's not had this problem.