Bug 1354046

Summary: rhel-osp-director: 8.0->9.0 upgrade, Keystone WSGI migration step fails, httpd resource is down on the cluster and fails to start.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Michele Baldessari <michele>
Status: CLOSED ERRATA QA Contact: Rodrigo Duarte <rduartes>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: dbecker, dcritch, jason.dobies, jjoyce, kbasil, mburns, michele, morazi, nlevinki, rcritten, rduartes, rhel-osp-director-maint, sasha, sgordon, tvignaud
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-21.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 11:35:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Alexander Chuzhoy 2016-07-08 20:05:16 UTC
rhel-osp-director:  8.0->9.0 upgrade, Keystone WSGI migration step fails, httpd resource is down on the cluster and fails to start.


Environment:
httpd-tools-2.4.6-40.el7_2.1.x86_64
python-keystoneclient-1.7.2-1.el7ost.noarch
python-keystoneauth1-1.1.0-4.el7ost.noarch
python-keystone-8.0.1-1.el7ost.noarch
httpd-2.4.6-40.el7_2.1.x86_64
python-keystonemiddleware-2.3.1-1.el7ost.noarch
openstack-keystone-8.0.1-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-2.0.0-13.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-13.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch
openstack-puppet-modules-8.1.2-1.el7ost.noarch
instack-undercloud-4.0.0-5.el7ost.noarch



Steps to reproduce:
1. Deploy 8.0 with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

2. Upgrade the undercloud 
3. Successfully pass the step with "-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml"
4. Attempt to do the keystone wsgi migration step (includes  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-keystone-liberty-mitaka.yaml)
Result:
2016-07-08 19:23:50 [ControllerDeployment]: SIGNAL_COMPLETE Unknown
2016-07-08 19:23:52 [1]: SIGNAL_COMPLETE Unknown
2016-07-08 19:23:54 [1]: SIGNAL_COMPLETE Unknown
Stack overcloud UPDATE_FAILED
Deployment failed:  Heat Stack update failed.

Checked the pcs resource status. Several resources were stopped.
ran "pcs resource cleanup"
All but one resource successfully started. 

[root@overcloud-controller-1 ~]# pcs status|grep -B1 -i stop
 Clone Set: httpd-clone [httpd]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]


Was unable to start this resource:
(98)Address already in use: AH00072: make_sock: could not bind to address 192.168.100.12:5000



turned out "/usr/bin/python2/usr/bin/keystone-all" was already using port 5000.

Comment 2 Alexander Chuzhoy 2016-07-08 20:33:25 UTC
The workaround is to stop the openstack-keystone.service on all controllers and then to start the httpd resource.

Comment 19 errata-xmlrpc 2016-08-11 11:35:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html