Description of problem: We are getting an authentication required when depolying the overcloud. We execture the following command to get it deployed: # openstack overcloud deploy --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute --block-storage-scale 0 --swift-storage-scale 0 --ntp-server pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan and we get the following error in the deployment log: DEBUG: heatclient.common.http curl -g -i -X GET -H 'X-Auth-Token: {SHA1}9bfc40185f7d61788bf46e9dc7393207d1792804' -H 'Content-Type: application/json' -H 'X-Auth-Url: https://10.135.2.36:13000/v2.0' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' --cacert /etc/haproxy/undercloud.pem https://10.135.2.36:13004/v1/fdb61c52c3c3431384960b4e5b83263f/stacks/overcloud <============================================================= /usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SecurityWarning DEBUG: heatclient.common.http HTTP/1.1 401 Unauthorized date: Fri, 11 Sep 2015 11:35:19 GMT content-length: 23 content-type: text/plain www-authenticate: Keystone uri='http://10.124.1.35:5000/v2.0' x-openstack-request-id: req-b8f86e2f-50cd-4530-b424-1c39b38d32d4 Authentication required ERROR: openstack ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1 Authentication required Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 824, in take_action self._deploy_tripleo_heat_templates(stack, parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 498, in _deploy_tripleo_heat_templates parsed_args.timeout) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 438, in _heat_deploy orchestration_client, "overcloud") File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/utils.py", line 144, in wait_for_stack_ready stack = orchestration_client.stacks.get(stack_name) File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 202, in get resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 265, in json_request resp = self._http_request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 217, in _http_request 'content': resp.content HTTPUnauthorized: ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1 Authentication required In heat-api side I could only find this related entry that return 200 OK, but nothing else 2015-09-11 11:35:19.775 1326 INFO eventlet.wsgi.server [req-d3880746-0d86-43b1-b1c5-d921f51db503 - admin] 256.256.33.35 - - [11/Sep/2015 11:35:19] "GET /v1/fdb61c52c3c3431384960b4e5b83263f/stacks/overcloud/d79fd372-4cbc-487f-813f-5726599673c3 HTTP/1.1" 200 8813 0.210738 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
*** Bug 1263333 has been marked as a duplicate of this bug. ***
Whenever you see that authentication error, it's usually an indication you've run up against the keystone timeout threshhold. If that occurs, you can look into why the deployment is taking so long. This often happens due to a misconfiguration in the network config yaml. If all that looks fine, you can always increase the keystone token timeout value.
There is confirmation that the deployment fails after 3 hours, however the timeout is 4 hours # Amount of time a token should remain valid (in seconds). (integer value) #expiration = 3600 expiration = 14400 Somewhere else to look ?
Eduard, if you see the auth error after the 3h timeout, it is most probably just due to the token being invalidated. The deployment shouldn't take that long; I suspect something might be timing out on one of the overcloud noded and never returning a complete/failed execution to heat. Can you login on the overcloud nodes and collect journal logs or run a sosreport?
Link to Logs and sosreport are in comment 4
-
This is completely stale at this point. Please reopen if you are currently reproducing the underlying issue (the heat auth error is just a timeout).