Bug 1258614

The original bug hits both openstack-puppet-modules and openstack-tripleo-heat-templates.  Cloning this so that the openstack-puppet-modules part gets fixed

+++ This bug was initially created as a clone of Bug #1252585 +++

Description of problem:

Deployment fails on one of the controller nodes. The deployment is on 3 controllers, 5 computes, and 2 ceph nodes at the moment. 

The error that causes the stack to fail comes from puppet on Controller-0

Aug 11 11:24:29 localhost os-collect-config: ::Server[6000]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6000]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.\n   (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')\u001b[0m\n\u001b[1;31mError: Received error response from Keystone server at http://172.17.0.10:35357/v3/domains: Unauthorized\u001b[0m\n\u001b[1;31mError: /Stage[main]/Heat::Keystone::Domain/Heat_domain_id_setter[heat_domain_id]/ensure: change from absent to present failed: Received error response from Keystone server at http://172.17.0.10:35357/v3/domains: Unauthorized\u001b[0m\n", "deploy_status_code": 6}

The interesting part of that error (formatted nicely):

Error: Received error response from Keystone server at http://172.17.0.10:35357/v3/domains: Unauthorized

Error: /Stage[main]/Heat::Keystone::Domain/Heat_domain_id_setter[heat_domain_id]/ensure: change from absent to present failed: Received error response from Keystone server at http://172.17.0.10:35357/v3/domains: Unauthorized


The connection to the keystone service works, as does everything else in the network I was able to test. 

In the Overcloud keystone logs, I see there's a login request for ceilometer that returns a 401:Unauthorized and seems to be unrelated, but there don't seem to be other failing requests.


--- Additional comment from Joe Talerico on 2015-08-11 18:00:03 EDT ---

Hitting this in the BAGL lab.

Aug 11 16:59:55 localhost os-collect-config: [2015-08-11 16:59:55,117] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/43fe7cb8-0d1f-473a-b056-4364ee7f4e8d.pp. [6]
Aug 11 16:59:55 localhost os-collect-config: ^[[1;31mError: Received error response from Keystone server at http://172.21.33.11:35357/v3/domains: Unauthorized^[[0m

--- Additional comment from Gaëtan Trellu on 2015-08-12 17:32:50 EDT ---

Hi,

We had the same issue.
Your undercloud should be able to communicate with your overcloud external network.

--- Additional comment from Emilien Macchi on 2015-08-18 15:01:39 EDT ---

This is not a networking or communication issue I think, because we have "Unauthorized". I guess this is a domain configuration issue...

--- Additional comment from James Slagle on 2015-08-18 15:22:38 EDT ---

fwiw, i saw this error when i had not specified a value for --ntp-server during a deployment

--- Additional comment from Martin Magr on 2015-08-19 03:39:59 EDT ---

I believe this is caused by heat domain creation script due to asynchronous corner cases during multi-controller deployment. Once [1] and [2] will be used, then this issue should perish. Will rebase [2] and run some tests.


[1] https://review.openstack.org/#/c/204541/
[2] https://review.openstack.org/#/c/180566/17



--- Additional comment from Shiva Prasad Rao on 2015-08-28 02:31:29 EDT ---

Is there a workaround for this issue? We have been hitting this issue on most of the deployments.

--- Additional comment from Martin Magr on 2015-08-28 11:24:43 EDT ---

Sadly, I'm not aware about any workaround. We might remove the #180566 patch from RHOSd until the latest patchset will be fully tested, but that will mean that customer will have to create domain manually and we can instead just save configuration from heat::keystone::domain class [1]. Derek, does that make sense?

[1] https://github.com/openstack/puppet-heat/blob/master/manifests/keystone/domain.pp#L78

--- Additional comment from Derek Higgins on 2015-08-28 11:34:08 EDT ---

Any time I've seen this it has been an underlying issue with the keystone setup on the controllers, the domain creation script just happens to be the first thing to use keystone.

Can you confirm that the time on each host is in sync? As times out of sync can cause tokens to be seen as invalid.