Bug 1285596

Summary: rhel-osp-director: 7.2 - HA overcloud deployment doesn't return the prompt, although reported in heat stack-list as completed.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: hbrock, mburns, mcornea, rhel-osp-director-maint, srevivo
Target Milestone: y3   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-26 18:42:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs from a controller with the failed heat. none

Description Alexander Chuzhoy 2015-11-26 01:54:24 UTC
rhel-osp-director: 7.2 - HA overcloud deployment  doesn't return the prompt, although reported in heat stack-list as completed.


Environment:
openstack-heat-engine-2015.1.2-2.el7ost.noarch
instack-undercloud-2.1.2-34.el7ost.noarch
openstack-heat-api-2015.1.2-2.el7ost.noarch
openstack-heat-common-2015.1.2-2.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-85.el7ost.noarch
openstack-heat-templates-0-0.7.20150605git.el7ost.noarch


Steps to reproduce:
Deploy HA overcloud  with network isolation, 2 computes and  3 ceph nodes.

Result:
The deployment doesn't end, although the timeout is set to 90 minutes.
running "heat stack-list" shows:
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| e4234dce-744d-45c2-8a11-6a06b4ed7d89 | overcloud  | CREATE_COMPLETE | 2015-11-25T23:22:03Z |
+--------------------------------------+------------+-----------------+----------------------+

Running commands against the overcloud results in:
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
ERROR (ConnectionRefused): Unable to establish connection to http://10.19.184.180:5000/v2.0/tokens


openstack-heat-api.service was failed on one controller out of 3.


Expected result:
The deployment should complete successfully.

Comment 1 Alexander Chuzhoy 2015-11-26 01:56:17 UTC
Created attachment 1099072 [details]
logs from a controller with the failed heat.

Comment 2 Alexander Chuzhoy 2015-11-26 01:57:02 UTC
The deployment command was:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 2  --ceph-storage-scale 3 --ntp-server 10.5.26.10  --timeout 90  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml   -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml  -e /home/stack/network-environment.yaml

Comment 3 Alexander Chuzhoy 2015-11-26 02:55:04 UTC
Here's a result from another deployment on the same setup, although without ceph nodes:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 2 --ntp-server 10.5.26.10  --timeout 90  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml  -e /home/stack/network-environment.yaml
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
/home/stack/.ssh/known_hosts updated.
Original contents retained as /home/stack/.ssh/known_hosts.old
PKI initialization in init-keystone is deprecated and will be removed.



ssh: connect to host 10.19.94.10 port 22: Connection timed out
ERROR: openstack Command '['ssh', '-oStrictHostKeyChecking=no', '-t', '-l', 'heat-admin', u'10.19.94.10', 'sudo', 'keystone-manage', 'pki_setup', '--keystone-user', "$(getent passwd | grep '^keystone' | cut -d: -f1)", '--keystone-group', "$(getent group | grep '^keystone' | cut -d: -f1)"]' returned non-zero exit status 255
[stack@undercloud ~]$ openstack overcloud deploy --templates --control-scale 3 --compute-scale 2 --ntp-server 10.5.26.10  --timeout 90  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml  -e /home/stack/network-environment.yaml
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates


Note: InternalApiNetCidr: 10.19.94.0/24
The undercloud can't connect there.

Comment 4 Alexander Chuzhoy 2015-11-26 18:42:15 UTC
It was due to an IP collision with another setup.
Able to deploy after the issue was fixed.