Description of problem: Customer is implementing RHEL OSP 7 to migrate existing OSP workloads from existing OSP 4 and OSP 5 environments. It is a very high priority for the customer to enable the transition to OSP 7 with as little interferance to end users as possible. The customer have LDAP integration working in current production platforms. With OSP 7 we have issues with Heat authentication when using LDAP as the Default driver. When using LDAP as the Default driver, we get this error: 2016-03-02 17:32:12.785 7079 TRACE heat.engine.service AuthorizationFailure: Authorization failed: Could not find domain: 04854fca39dc4bbeaa187475c4530061 (HTTP 404) (Request-ID: req-a1a9cf2c-1b9d-4efd-a964-1e26f08441fc) (HTTP 404) 2016-03-02 17:32:12.785 7079 TRACE heat.engine.service --- We can succesfully make it work by using domain-specific identity drivers and configure SQL as the default, and LDAP as an additional "Users" domain. However: In the configuration where LDAP is not the Default domain the end-user will have to specify the domain in client's (CLI, Horizon, Python scripts etc). Example: “openstack user list --domains default/users” The customer is not satisfied with the requirement to change the client configuration in their environment, existing scripting in CI environments etc. Is there any way we can make this work, without the need for changes client side? Version-Release number of selected component (if applicable): OSP-7, deployed with OSP-d 7.3.
While it is possible, understand that the V2 API in Keystone is deprecated, and the most forward-compatible approach is to specify the domain and use the V3 API. Multiple Domains are a very powerful concept. I would suggest pointing out the benefits, to include having the service users separated from Heat specific entries, and being able to allocate user entries for any third part automation. It also will allow for multiple LDAP servers and Federation for third party Identity Providers. To Continue using the V2 API means that the non-LDAP backed accounts all need to use the V3 API, and then the LDAP backed domain gets switched over to be the default. I am not certain that this is well enough tested to suggest it be run in production. Please test in a staging environment first.
I've sent Harald a longer summary via email, but I think we have two options that do not involve using the unsupported, out-of-tree hybrid driver: 1. Create service users using the default SQL-backed domain, and use the LDAP-backed domain for other users. This requires those users to use the v3 API so they can specify a non-default domain. 2. Use a single, default LDAP-backed domain, which would mean creating all users (including service users) in LDAP. This is a very common approach amongst our customers. It would mean that all users could continue using the v2 API to authenticate.
I think what was clarified separately was that the OSP4 environment used (2) and the preference would be to continue using that approach, but a Heat issue was observed on OSP7. We should get to the bottom of that Heat issue before considering other approaches, I think. https://bugs.launchpad.net/openstack-ansible/+bug/1447768 was mentioned as a similar Heat issue.
The issue is likely rooted in the newer heat version's use of trusts. Heat has to create/delete users for every stack so LDAP users can delegate permissions for stack actions. There's an older setting where heat instead stores the user's (encrypted) password such that it can still take actions on behalf of the user without needing write access to the domain (which the LDAP backend lacks). When setting up heat, they could use password as the deferred_auth_method instead of trusts, which would make heat continue to work until they're able to change their scripts to handle multidomain and move off the V2 API.
Right, Heat has switch to a more secure method of performing operations on behalf of the user, but this new method is not compatible with using a single read-only LDAP identity domain. So something like: $> crudini --set /etc/heat/heat.conf DEFAULT deferred_auth_method password should do restore the same behavior as OSP4. We still recommend moving back to the more secure "trusts" method once using multiple identity domains is a workable approach for this deployment Closing as notabug for now, but please do reopen if this doesn't resolve the issue. Thanks!
Note that comment #7 is not quite accurate - trusts are unrelated to the heat usage of domains so setting deferred_auth_method will not help return to behaviour which only expects the default domain. Instead, you need to *not* set stack_user_domain_id or stack_user_domain_name in heat.conf, then we fall back to the old legacy path: https://github.com/openstack/heat/blob/master/heat/common/heat_keystoneclient.py#L92 https://github.com/openstack/heat/blob/master/heat/common/heat_keystoneclient.py#L310 Note this still creates a user for the in-instance credentials, but it's in the same domain/project as the stack owner. This is VERY BAD from a security perspective if you allow creating resources that then create users and deploy credentials inside your VMs (it also requires all heat users to be admins). That said, I can only assume in this case that those resources which require creating users are not being used at all, otherwise we'd still fail creating users in the (read only) LDAP backed keystone? More context of why this works like this, and why the heat domain is a good idea: https://bugs.launchpad.net/heat/+bug/1089261 https://blueprints.launchpad.net/heat/+spec/instance-users http://hardysteven.blogspot.co.uk/2014/04/heat-auth-model-updates-part-2-stack.html
Apologies for the conflicting recommendations here. We are confident that Steve is correct, the Heat error very likely related to the concept of "Stack Domain Users" which is very confusingly similar (but unrelated to) the way Heat does deferred authentication. To confirm this suspicion, you should be able to do: $> crudini --del /etc/heat/heat.conf DEFAULT stack_user_domain_name and the "could not find domain" error above will go away, but you will see this harmless warning in heat-engine.log: WARNING heat.common.config [-] stack_user_domain_id or stack_user_domain_name not set in heat.conf falling back to using default With that confirmed, the next step would be to ensure put in place a stack user domain. The idea would be to have a read-only, LDAP-backed domain as the default domain (including service users) but to also have a writable, non-default, SQL-backed 'heat_stack' domain so that Heat can create users and projects when needed.