Bug 1437222

Summary: OSP10 -> OSP11 upgrade fails on environment using external loadbalancer due to missing haproxy service running on controller nodes
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED ERRATA QA Contact: Amit Ugol <augol>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 11.0 (Ocata)CC: bperkins, dbecker, jcoufal, jschluet, mburns, morazi, rhel-osp-director-maint, sathlang, slinaber
Target Milestone: rcKeywords: Regression, Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-6.0.0-4.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-17 20:16:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2017-03-29 19:56:49 UTC
Description of problem:
OSP10 -> OSP11 upgrade fails on environment using external loadbalancer due to missing haproxy service running on controller nodes:

[stack@undercloud-0 ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerUpgrade_Step4.1:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 590915dd-fbcd-4692-ba17-65ecb673051f
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    
    TASK [Check pacemaker resource] ************************************************
    ok: [localhost] => (item=rabbitmq)
    failed: [localhost] (item=haproxy) => {"failed": true, "item": "haproxy", "msg": "Resource 'haproxy' doesn't exist in the cib.", "resource_name": "haproxy", "resource_type": null}
    ok: [localhost] => (item=galera)
    	to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/082a4c01-dadc-401b-b899-d0bb868222fb_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=3    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step4.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 3b5fb93a-0e66-4e83-bb83-f543918d21bc
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    
    TASK [Check pacemaker resource] ************************************************
    ok: [localhost] => (item=rabbitmq)
    failed: [localhost] (item=haproxy) => {"failed": true, "item": "haproxy", "msg": "Resource 'haproxy' doesn't exist in the cib.", "resource_name": "haproxy", "resource_type": null}
    ok: [localhost] => (item=galera)
    	to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/93cd7e9c-915d-43b5-8201-d4478052a430_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=7    changed=4    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step4.2:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: cddc9c0e-71ae-41b8-85dc-3628e3c42f02
  status: CREATE_FAILED
  status_reason: |
    Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    
    TASK [Check pacemaker resource] ************************************************
    ok: [localhost] => (item=rabbitmq)
    failed: [localhost] (item=haproxy) => {"failed": true, "item": "haproxy", "msg": "Resource 'haproxy' doesn't exist in the cib.", "resource_name": "haproxy", "resource_type": null}
    ok: [localhost] => (item=galera)
    	to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/8672385f-986d-4346-9e87-87e44d3a523b_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=3    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-6.0.0-0.6.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 with external loadbalancer
2. Upgrade to OSP11

Actual results:
Upgrade fails

Expected results:
Upgrade succeeds.

Additional info:
In https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/pacemaker.yaml#L90-L93 the PacemakerResources list contains haproxy which doesn't run on the controller nodes when an external loadbalancer is used.

Comment 3 errata-xmlrpc 2017-05-17 20:16:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245