Bug 1322916

Summary: rhel-osp-director: After rebooting the OC nodes - unable to communicate with OC keystone.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 8.0 (Liberty)CC: dbecker, dyasny, jcoufal, jslagle, mburns, morazi, ohochman, rhel-osp-director-maint, sasha
Target Milestone: rcKeywords: TestOnly, Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-openstack-tripleo-heat-templates-5.1.0-6.el7ost.noarch-5.0.0-0.5.0rc3.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 15:29:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2016-03-31 16:16:28 UTC
rhel-osp-director: After rebooting the OC nodes  - unable to communicate with OC keystone.


Environment:
python-keystone-8.0.1-1.el7ost.noarch
python-keystoneclient-1.7.2-1.el7ost.noarch
openstack-keystone-8.0.1-1.el7ost.noarch
python-keystoneauth1-1.1.0-4.el7ost.noarch
python-keystonemiddleware-2.3.1-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-2.el7ost.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-2.el7ost.noarch
instack-undercloud-2.2.7-1.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud 7.3
2. Upgrade to 8.0
3. Reboot all nodes as part of testing.
4. Try to run any command against the overcloud.

Result:
[stack@instack ~]$ nova list
No handlers could be found for logger "keystoneauth.identity.generic.base"
ERROR (ConnectFailure): Unable to establish connection to https://192.168.200.180:13000/v2.0/tokens


Port 13000 isn't open on the controller(s).
No stopped pcs resources.


Expected result:
Should be able to run commands against the overcloud.

Comment 2 Mike Burns 2016-04-07 21:36:02 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 3 Dan Yasny 2016-05-18 00:33:39 UTC
I just reproduced it today, this basically means that none of the API endpoints for the overcloud work after a restart, because everything is tied into keystone. What is worse, this breaks automation in a bad way, we don't even get errors that we can parse, any python-*-client based command just hangs.

If there is no scalable workaround (manual service restart on multiple hosts isn't exactly scalable), I'd suggest the priority is set to the highest possible.

Comment 4 James Slagle 2016-10-14 15:31:50 UTC
please retest and confirm if this is still an issue with OSP10

Comment 7 Omri Hochman 2016-11-29 21:10:15 UTC
Verified with openstack-tripleo-heat-templates-5.1.0-6.el7ost.noarch

By doing :
- Upgrade 9->10.
- Reboot OC nodes.
- Check the ability to communicate with the overcloud upon reboot.

Comment 10 errata-xmlrpc 2016-12-14 15:29:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html