Bug 1263334 - Authentication required with error with heat when deploying the overcloud
Authentication required with error with heat when deploying the overcloud
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: Giulio Fidente
: Triaged
: 1263333 (view as bug list)
Depends On:
  Show dependency treegraph
Reported: 2015-09-15 10:47 EDT by Eduard Barrera
Modified: 2016-04-18 03:11 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-02-28 02:42:41 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Eduard Barrera 2015-09-15 10:47:43 EDT
Description of problem:

We are getting an authentication required when depolying the overcloud. We execture the following command to get it deployed:

# openstack overcloud deploy --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute --block-storage-scale 0 --swift-storage-scale 0 --ntp-server pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan

and we get the following error in the deployment log:

DEBUG: heatclient.common.http curl -g -i -X GET -H 'X-Auth-Token: {SHA1}9bfc40185f7d61788bf46e9dc7393207d1792804' -H 'Content-Type: application/json' -H 'X-Auth-Url:' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' --cacert /etc/haproxy/undercloud.pem   <=============================================================

/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
DEBUG: heatclient.common.http
HTTP/1.1 401 Unauthorized
date: Fri, 11 Sep 2015 11:35:19 GMT 
content-length: 23
content-type: text/plain
www-authenticate: Keystone uri=''
x-openstack-request-id: req-b8f86e2f-50cd-4530-b424-1c39b38d32d4

Authentication required

ERROR: openstack ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1
Authentication required
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 824, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 498, in _deploy_tripleo_heat_templates
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 438, in _heat_deploy
    orchestration_client, "overcloud")
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/utils.py", line 144, in wait_for_stack_ready
    stack = orchestration_client.stacks.get(stack_name)
  File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 202, in get
    resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id)
  File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 265, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 217, in _http_request
    'content': resp.content
HTTPUnauthorized: ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1
Authentication required

In heat-api side I could only find this related entry that return 200 OK, but nothing else

2015-09-11 11:35:19.775 1326 INFO eventlet.wsgi.server [req-d3880746-0d86-43b1-b1c5-d921f51db503 - admin] - - [11/Sep/2015 11:35:19] "GET /v1/fdb61c52c3c3431384960b4e5b83263f/stacks/overcloud/d79fd372-4cbc-487f-813f-5726599673c3 HTTP/1.1" 200 8813 0.210738

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:
Comment 3 Mike Burns 2015-09-15 10:48:42 EDT
*** Bug 1263333 has been marked as a duplicate of this bug. ***
Comment 5 chris alfonso 2015-09-17 12:50:39 EDT
Whenever you see that authentication error, it's usually an indication you've run up against the keystone timeout threshhold. If that occurs, you can look into why the deployment is taking so long. This often happens due to a misconfiguration in the network config yaml. If all that looks fine, you can always increase the keystone token timeout value.
Comment 6 Eduard Barrera 2015-09-22 05:28:10 EDT
There is confirmation that the deployment fails after 3 hours, however the timeout is 4 hours

# Amount of time a token should remain valid (in seconds). (integer value)
#expiration = 3600
expiration = 14400

Somewhere else to look ?
Comment 9 Giulio Fidente 2015-09-28 05:43:31 EDT
Eduard, if you see the auth error after the 3h timeout, it is most probably just due to the token being invalidated.

The deployment shouldn't take that long; I suspect something might be timing out on one of the overcloud noded and never returning a complete/failed execution to heat.

Can you login on the overcloud nodes and collect journal logs or run a sosreport?
Comment 10 Eduard Barrera 2015-09-28 06:10:47 EDT
Link to Logs and sosreport are in comment 4
Comment 11 Eduard Barrera 2015-09-28 06:50:30 EDT
Comment 13 Hugh Brock 2016-02-28 02:42:41 EST
This is completely stale at this point. Please reopen if you are currently reproducing the underlying issue (the heat auth error is just a timeout).

Note You need to log in before you can comment on or make changes to this bug.