Bug 1731045

Summary: Deployment fails after including barbican environment files
Product: Red Hat OpenStack Reporter: PURANDHAR SAIRAM MANNIDI <pmannidi>
Component: openstack-tripleo-heat-templatesAssignee: RHOS Maint <rhos-maint>
Status: CLOSED NOTABUG QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: emacchi, hrybacki, mburns
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-25 11:41:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description PURANDHAR SAIRAM MANNIDI 2019-07-18 08:25:05 UTC
Description of problem:
Deployment fails after including barbican environment files to the existing env.

2019-07-18 08:06:44Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv.2]: UPDATE_IN_PROGRESS  state changed
2019-07-18 08:07:29Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv.1]: SIGNAL_IN_PROGRESS  Signal: deployment 4c787c90-ee92-49e6-8779-8caadc951635 succeeded
2019-07-18 08:07:29Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv.1]: UPDATE_COMPLETE  state changed
2019-07-18 08:07:52Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv.2]: SIGNAL_IN_PROGRESS  Signal: deployment 04f4a4cb-0690-4527-af13-06c5ac515bde succeeded
2019-07-18 08:07:52Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv.2]: UPDATE_COMPLETE  state changed
2019-07-18 08:07:57Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv.0]: SIGNAL_IN_PROGRESS  Signal: deployment c4f55c8d-12d1-4e5d-9687-2ff1d2c9d85d failed (2)
2019-07-18 08:07:58Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv.0]: UPDATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2019-07-18 08:07:58Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh-ControllerDeployment_Step3-jqomefgqg2xv]: UPDATE_FAILED  Resource UPDATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2019-07-18 08:07:58Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh.ControllerDeployment_Step3]: UPDATE_FAILED  Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2019-07-18 08:07:58Z [overcloud-AllNodesDeploySteps-zs5m6iznukhh]: UPDATE_FAILED  Resource UPDATE failed: Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2019-07-18 08:07:59Z [AllNodesDeploySteps]: UPDATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2019-07-18 08:07:59Z [overcloud]: UPDATE_FAILED  Resource UPDATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2

 Stack overcloud UPDATE_FAILED 

overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: c4f55c8d-12d1-4e5d-9687-2ff1d2c9d85d
  status: UPDATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "stdout: 5865f9c0cd5f1904af138cd38a1ed555d22db7ff58e7066e344adf69f64cee4b", 
            "stdout: fc48cd132bc7b71e48ab6cba59db952bfdc6252a1750856bee8b7d505ce1ce0c", 
            "stdout: 3222d213f3d5140e63ba5a9d3b403e1f81597e4567aa0001714d2613652ea00e"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/1e27cef6-48ff-4685-a861-3b0b09a4bbdd_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=8    changed=3    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |

Heat Stack update failed.
Heat Stack update failed.


Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform 13

How reproducible:
Always

Steps to Reproduce:
1. Deploy Overcloud without barbican
2. Include Barbican env files
3. Redeploy

Actual results:
Deployment fails

Expected results:
Deployment should succeed

Additional info:

Barbican DB isn't created, so is barbican galera user. After manually creating Barbican DB with privileges, stopping existing barbican_worker and barbican_keystone_listener and redeploying with same templates resolves the issue.

Comment 1 Harry Rybacki 2019-07-19 13:55:41 UTC
Purandhar, thank you for raising this issue. 

Could you provide us with specifics for reproducing your failure e.g. which environment files did you include, were there any modifications to them, etc. Additionally, any logs (or a sosreport) would be helpful.

Thanks!

Comment 2 PURANDHAR SAIRAM MANNIDI 2019-07-25 11:41:23 UTC
The issue is because of --skip-deploy-identifier in the deploy command. Redeploying without that option has passed and deployment was successful.