Bug 1320333

Summary: rhel-osp-director: 7.3->8.0 overcloud upgrade fails on " Property error: resources[1].properties: Property HAProxyStatsPassword not assigned"
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Marios Andreou <mandreou>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: high    
Version: 8.0 (Liberty)CC: dbecker, mandreou, mburns, morazi, rhel-osp-director-maint
Target Milestone: ga   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-23 13:09:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
heat-engine.log from the undercloud. none

Description Alexander Chuzhoy 2016-03-22 21:10:16 UTC 
rhel-osp-director: 7.3->8.0 overcloud upgrade fails on " Property error: resources[1].properties: Property HAProxyStatsPassword not assigned"

Environment:
openstack-tripleo-heat-templates-0.8.12-2.el7ost.noarch
instack-undercloud-2.2.6-1.el7ost.noarch
openstack-puppet-modules-7.0.15-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.12-2.el7ost.noarch


Steps to reproduce:
1. deploy overcloud 7.3
openstack overcloud deploy --templates --control-scale 3 --compute-scale 2    --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml

2. update the undercloud to 8.0

3. Run:
openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network-environment.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml -e rhos-release-8.yaml

Result:

2016-03-22 19:21:23 [EndpointMap]: UPDATE_COMPLETE  state changed
2016-03-22 19:21:23 [Controller]: UPDATE_IN_PROGRESS  state changed
2016-03-22 19:21:25 [Controller]: UPDATE_FAILED  resources.Controller: Property error: resources[1].properties: Property HAProxyStatsPassword not assigned
2016-03-22 19:21:26 [overcloud]: UPDATE_FAILED  resources.Controller: Property error: resources[1].properties: Property HAProxyStatsPassword not assigned
Stack overcloud UPDATE_FAILED
Heat Stack update failed.
[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time        | stack_name                                                                    |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------+
| Controller                                | 9d9cbc1e-6eac-4e64-bfaa-25c00423126a          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-03-22T19:21:23 | overcloud                                                                     |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------+
[stack@instack ~]$

Expected result:
The command should complete successfully.
Comment 2 Alexander Chuzhoy 2016-03-22 21:14:41 UTC 
Created attachment 1139233 [details]
heat-engine.log from the undercloud.
Comment 4 Marios Andreou 2016-03-23 07:44:54 UTC 
Hi Sasha, thanks for filing the bug which is valid though luckily an easy fix/perhaps already fixed. The tl;dr is we need a new build for python-tripleoclient because of the changes recently that added the haproxystats user and password:

1. (tripleo-heat-templates), "Wire in HAProxy stats user and password" https://review.openstack.org/#/c/295442/

2. (python-tripleoclient), "Add password for HAProxy stats" https://review.openstack.org/#/c/295441/1

3. (puppet-tripleo), "Allow enabling authentication on haproxy.stats" https://review.openstack.org/#/c/290896/2

(@mburns needinfo please ^^^ can we check we have these in the relevant latest builds)

The tripleo-heat-templates change *did* land into stable/liberty (which you are using for the upgrade), but didn't land into the python-tripleoclient-0.3.1-1.el7ost.noarch package yet. Hence there indeed was a parameter "HAProxyStatsPassword" in the templates that you passed to heat which isn't defaulted to anything and also isn't set in the version of the client you have.

After I manually applied https://review.openstack.org/#/c/295441/ to /usr/lib/python2.7/site-packages/tripleoclient/ the upgrade init completed fine in ~ 10 mins:

openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network-environment.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml -e rhos-release-8.yaml
Deploying templates in the directory /home/stack/tripleo-heat-templates

2016-03-23 07:22:21 [overcloud]: UPDATE_IN_PROGRESS  Stack UPDATE started
...
2016-03-23 07:32:11 [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully
Stack overcloud UPDATE_COMPLETE

thanks, marios
Comment 5 Marios Andreou 2016-03-23 10:39:09 UTC 
> After I manually applied https://review.openstack.org/#/c/295441/ to
> /usr/lib/python2.7/site-packages/tripleoclient/ the upgrade init completed
> fine in ~ 10 mins:
> 

as a temp workaround you can do that ^^^ like

sudo su
pushd /usr/lib/python2.7/site-packages/tripleoclient/
curl -o utils.py
"https://review.openstack.org/gitweb?p=openstack/python-tripleoclient.git;a=blob_plain;f=tripleoclient/utils.py;h=bfd60cb3c2b45f84f616691b14fa59f5b1a30197;hb=d5bb743cbccd93df709f3d0815ce7b2acf416c8d"
curl -o v1/overcloud_deploy.py
"https://review.openstack.org/gitweb?p=openstack/python-tripleoclient.git;a=blob_plain;f=tripleoclient/v1/overcloud_deploy.py;h=f6b95e3de8e1d29b93646d196c4c81cf8adbdbd7;hb=d5bb743cbccd93df709f3d0815ce7b2acf416c8d"
popd
exit # from sudo su
Comment 6 Mike Burns 2016-03-23 13:08:50 UTC 
So this bug came about because all the changes from 1311383 haven't landed downstream yet.

The test was done with THT from upstream stable/liberty (which has part of the fix) and not downstream THT package (which doesn't) which causes the issue.  When 1311383 lands downstream, this won't happen anymore, so closing as a duplicate
Comment 7 Mike Burns 2016-03-23 13:09:31 UTC 

*** This bug has been marked as a duplicate of bug 1311383 ***