Bug 1113534

Summary: LDAP misconfiguration should be handled better
Product: Red Hat OpenStack Reporter: Udi Kalifon <ukalifon>
Component: openstack-keystoneAssignee: Nathan Kinder <nkinder>
Status: CLOSED ERRATA QA Contact: Udi Kalifon <ukalifon>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.0 (RHEL 7)CC: aberezin, ajeain, apevec, ayoung, dnavale, jtrowbri, nkinder, sclewis, ukalifon, yeylon
Target Milestone: z2Keywords: OtherQA, ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-keystone-2014.1.3-1.el7ost Doc Type: Bug Fix
Doc Text:
Previously, when LDAP was used as a backend, Identity service attributes that mapped to 'None' in LDAP were not implicitly ignored. So, a 500 error was encountered when performing operations in Identity service that retrieve entries from LDAP. With this bug fix, attributes mapped to 'None' are now implicitly ignored, even if the admin left the attribute out of the ignore list. As a result, errors are not encountered when entries are retrieved from LDAP.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-03 08:47:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1128799    
Attachments:
Description Flags
script to install freeipa and configure a schema for keystone
none
keystone configuration for LDAP - part 1
none
keystone configuration for LDAP - part 2 none

Description Udi Kalifon 2014-06-26 11:14:05 UTC
Description of problem:
When configuring the keystone client to work with LDAP, you see the following error recurring in keystone.log:

2014-06-26 13:55:16.401 27975 ERROR keystone.common.wsgi [-] 'NoneType' object has no attribute 'lower'
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 207, in __call__
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     result = method(context, **params)
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/controllers.py", line 98, in authenticate
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     context, auth)
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/controllers.py", line 263, in _authenticate_local
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     username, CONF.identity.default_domain_id)
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 189, in wrapper
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     return f(self, *args, **kwargs)
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 315, in get_user_by_name
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     ref = driver.get_user_by_name(user_name, domain_id)
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/backends/ldap.py", line 86, in get_user_by_name
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     return identity.filter_user(self.user.get_by_name(user_name))
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/ldap/core.py", line 624, in get_by_name
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     res = self.get_all(query)
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/ldap/core.py", line 1008, in get_all
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     if x[0] != self.enabled_emulation_dn]
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/backends/ldap.py", line 210, in _ldap_res_to_model
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     obj = super(UserApi, self)._ldap_res_to_model(res)
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/ldap/core.py", line 502, in _ldap_res_to_model
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi     v = lower_res[self.attribute_mapping.get(k, k).lower()]
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi AttributeError: 'NoneType' object has no attribute 'lower'
2014-06-26 13:55:16.401 27975 TRACE keystone.common.wsgi 
2014-06-26 13:55:16.403 27975 INFO eventlet.wsgi.server [-] 10.35.64.157 - - [26/Jun/2014 13:55:16] "POST /v2.0/tokens HTTP/1.1" 500 291 0.010635

When running the keystone commands user-list or tenant-list you get the error:
An unexpected error prevented the server from fulfilling your request. (HTTP 500)

Setting up the LDAP support is done using an automatic script which works in Havana and used to work in Icehouse. The script creates tenants and then fails after trying to call tenant-list.


Version-Release number of selected component (if applicable):
openstack-keystone-2014.1-5.el7ost.noarch
python-keystone-2014.1-5.el7ost.noarch
python-keystoneclient-0.9.0-1.el7ost.noarch


How reproducible:
100%

Comment 1 Alan Pevec 2014-06-26 12:40:14 UTC
> File
> "/usr/lib/python2.7/site-packages/keystone/common/ldap/core.py", line 502,
> in _ldap_res_to_model
>      v =
> lower_res[self.attribute_mapping.get(k, k).lower()]
>  AttributeError:
> 'NoneType' object has no attribute 'lower'

This line is from a patch for a bug 1073011
take from stable/icehouse, so that'd mean 2014.1.1 would be broken the same way if we rebased.

As a quickfix, we can revert that patch but also need to investigate this upstream.

Please provide your "automatic script" so this can be replicated. Thanks.

Comment 2 Udi Kalifon 2014-06-26 12:51:22 UTC
Created attachment 912412 [details]
script to install freeipa and configure a schema for keystone

Comment 3 Udi Kalifon 2014-06-26 12:53:12 UTC
Created attachment 912413 [details]
keystone configuration for LDAP - part 1

Comment 4 Udi Kalifon 2014-06-26 12:54:55 UTC
Created attachment 912414 [details]
keystone configuration for LDAP - part 2

Comment 5 Udi Kalifon 2014-06-26 13:27:39 UTC
The scripts are attached to the bug, for anyone who is interested in replicating the setup. The server script is intended to run on a clean RHEL7 machine and the client setup is intended to run on a RHEL7 setup with a default keystone installation (Havana or Icehouse).

To install a FreeIPA server with the script:
1) Extract the tar.bz archive, it will create an "ldap" directory
2) In the ldap directory, run the runme-server.sh script which will run all the other scripts in order
3) A new file, called runonce-hostname-dns.txt, will be created and you need to copy it over to the keystone machine

To configure the client machine:
1) It is recommended to backup your keystone.conf file, so that you can revert the configuration changes
2) Make sure you copied the file runonce-hostname-dns.txt from the server
3) Run the 2 scripts in order: configure-client.sh and configure-client2.sh

Comment 6 Nathan Kinder 2014-06-26 13:59:30 UTC
I'll need to set up a test system to see what is happening, but there are a few things I notice:

- The error seems to be that a mapping isn't found, which indicates that this might be schema/config related.

- We definitely should be handlign the case where a mapping isn't found better (checking that before calling lower()).

- LDAP is being configured for the assignment backend in addition to identity.  This is not a normal use-case, so this problem might be assignment specific.

Comment 7 Nathan Kinder 2014-06-26 17:18:58 UTC
The problem here seems to be triggered by a configuration setting.  I added some instrumentation to see what mapping it was having a problem with, and the following was reported for both 'user-list' and 'tenant-list':

----------------------------------------------------------------------------
2014-06-26 19:51:43.339 13649 INFO keystone.common.ldap.core [-] mapping is {'email': 'mail', 'password': 'userPassword', 'enabled': 'enabled', 'default_project_id': None, 'name': 'uid'}
2014-06-26 19:51:43.340 13649 INFO keystone.common.ldap.core [-] k is id
2014-06-26 19:51:43.340 13649 INFO keystone.common.ldap.core [-] k is name
2014-06-26 19:51:43.340 13649 INFO keystone.common.ldap.core [-] k is domain_id
2014-06-26 19:51:43.340 13649 INFO keystone.common.ldap.core [-] k is password
2014-06-26 19:51:43.341 13649 INFO keystone.common.ldap.core [-] k is description
2014-06-26 19:51:43.341 13649 INFO keystone.common.ldap.core [-] k is email
2014-06-26 19:51:43.341 13649 INFO keystone.common.ldap.core [-] k is enabled
2014-06-26 19:51:43.341 13649 INFO keystone.common.ldap.core [-] k is default_project_id
2014-06-26 19:51:43.342 13649 ERROR keystone.common.wsgi [-] 'NoneType' object has no attribute 'lower'
----------------------------------------------------------------------------

As can be seen the mapping for the 'default_project_id' is None, and things break when this mapping is processed.  Keystone's default settings show this:

----------------------------------------------------
#user_attribute_ignore=default_project_id,tenants
#user_default_project_id_attribute=<None>
----------------------------------------------------

With these default settings, the 'default_project_id' is ignored, which causes us to skip processing the mapping.  This avoids the issue reported in this bug.  The test scripts in this bug don't ignore the 'default_project_id' attribute though, which triggers the 500 error:

----------------------------------------------------
user_attribute_ignore = tenant_id,tenants
----------------------------------------------------

If 'default_project_id' is added to the list of user attributes to ignore, the 500 error doesn't occur.

There is still a bug here, which is to improve the handling of a mapping that is set to None that is not ignored.  This is a misconfiguration, but we should handle it better.

Comment 13 Nathan Kinder 2014-09-20 02:36:38 UTC
This has merged in stable/icehouse upstream and will be included in the upstream 2014.1.3 release of Keystone.

Comment 14 Nathan Kinder 2014-10-08 23:05:20 UTC
*** Bug 1128916 has been marked as a duplicate of this bug. ***

Comment 16 Ami Jeain 2014-10-26 11:22:22 UTC
no capacity to test this medium bug during Icehouse.

Comment 18 errata-xmlrpc 2014-11-03 08:47:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2014-1790.html