Bug 1120305
Summary: | Horizon failed to login - manually setting to use keystone v3 | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Ido Ovadia <iovadia> | ||||||||||
Component: | doc-Deployment_Guide | Assignee: | Suyog Sainkar <ssainkar> | ||||||||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Ido Ovadia <iovadia> | ||||||||||
Severity: | high | Docs Contact: | |||||||||||
Priority: | high | ||||||||||||
Version: | 5.0 (RHEL 6) | CC: | adahms, ajeain, aortega, apevec, ayoung, iovadia, jpichon, mrunge, sclewis, yeylon | ||||||||||
Target Milestone: | async | Keywords: | Documentation, Regression, Triaged, ZStream | ||||||||||
Target Release: | 5.0 (RHEL 6) | ||||||||||||
Hardware: | x86_64 | ||||||||||||
OS: | Unspecified | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2015-02-13 05:43:31 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | |||||||||||||
Bug Blocks: | 1186518 | ||||||||||||
Attachments: |
|
Could you set the logging level to DEBUG in the settings - in the local_settings, in the LOGGING dictionary, making sure level is set to DEBUG for 'file' like this: 'file': { 'level': 'DEBUG', 'class': 'logging.FileHandler', 'filename': '/var/log/horizon.log', 'formatter': 'verbose', }, Then empty the contents of horizon.log, try to login again and attach the logs here? After that - does deleting the cookies help with login in? Restarting httpd? I assume, this is an issue with keystone v3. That setting was changed after packstack install: OPENSTACK_API_VERSION was set to 'identity': 3 Setting back/uncommenting makes that setup work immediately. I think, the issue is valid on rhel6 and rhel7. How to reproduce: 1. set up horizon via packstack 2. change in /etc/openstack_dashboard/local_settings: (lines 37 to 39 to read:) OPENSTACK_API_VERSIONS = { "identity": 3 } 3. restart httpd 4. Try to log in: after typing in username/password, nothing happens 5. comment in lines 37 to 39 6. restart httpd 7. log in To make it more clear: there is no issue, when using plain packstack install the reproducer does not work locally here on f20 and on rhel7 :-/ workaround? =========== After step 2: change in /etc/openstack_dashboard/local_settings: (lines 37 to 39 to read:) OPENSTACK_API_VERSIONS = { "identity": 3 } Reboot the system Ido, to clarify: #c7 is, what what you did, to produce this issue? Afterwards, the same steps solved it? Because that whole thing is still a bit unclear to me, I set up a RHOS-5 system on RHEL6, enabled ssl for httpd and changed /etc/openstack-dashboard/local_settings according to #c7. I was immediately able to log into the system. Ido, did you change on your system something else as well? Ido, could you provide logs from around the time when the problem occurs (failure to login): both the horizon logs (in debug mode ideally, so we can see the requests sent with the keystone client), as well as the keystone logs when trying to log in? I let the system idle for the last few days and haven't managed to reproduce yet. I tried to properly log out then logged in again, I tried to let my session time out then log in. What kind of activities were performed on the system before the failure started? Another important package version I forgot to note in comment 15: python-django-openstack-auth-1.1.5-2.el7ost.noarch Ido (or anyone who can reproduce), in addition to logs could you also provide version numbers for these packages? Thank you. This bug referce to RHEL 6 I will try to reproduce it till weekend This bug tested both: RHEL 6 and RHEL 7 and it is only broken on RHEL 6 The bug reproduced only on RHEL 6 It was last reproduced on: ========================== OpenStack-5.0-RHEL-6 Puddle: 2014-08-22.1 python-django-horizon-2014.1.2-1.el6ost.noarch openstack-dashboard-theme-2014.1.2-1.el6ost.noarch openstack-dashboard-2014.1.2-1.el6ost.noarch This is very strange. Ido gave me access to the system where the problem is occurring, I can see it although there's nothing obvious in the horizon, httpd or keystone logs as to the cause. Switching the Horizon setting between Keystone v2 and v3 does make the problem go away and reappear. I set up 2 RHEL 6.5 environments and ran packstack on both, one where Horizon is set up with SSL (CONFIG_HORIZON_SSL=y in packstack) and one where it isn't. I changed the Keystone setting to v3, restarted httpd and I still cannot reproduce the problem. The django, horizon, django_openstack_auth, keystone and keystone client package versions are the same as on Ido's system. I ran yum update and rebooted, still I can login just fine. Ido indicated that the problem could be reproduced as soon as the version was changed this time, but I'll still let the systems idle for a couple of days to see if that changes something, and also see if I can figure out the differences with Ido's environment. Created attachment 930781 [details]
answer file, all-in-one, no SSL, Heat=y
Reproduced again: allinone CONFIG_HORIZON_SSL=n, CONFIG_HEAT_INSTALL=y Same version: OpenStack-5.0-RHEL-6 Puddle: 2014-08-22.1 Julie, try to use the new enclosed answer file. I will try to reproduce without Heat Created attachment 930794 [details]
RHEL 6.5 - rpm list
Created attachment 930795 [details]
RHEL 7 - rpm list
This bug reproduced on RHEL 6.5 only, I used the enclosed answerfile for both RHEL 6.5 and RHEL 7. rpm lists are enclosed too I was able to reproduce on RHEL 6.5 with CONFIG_HORIZON_SSL=y, CONFIG_HEAT_INSTALL=y. I'm assuming something is configured differently when also installing Heat, although there's no differences standing out in the configuration files so far and I can't quite explain why it would only reproduce on RHEL 6 either. This is not related to heat in particular but to the keystone catalogue size. If there are 10 or more services, the login issue with Keystone v3 happens on RHEL 6. After deleting services and endpoints so that there are only 9 services left in the catalogue, I can login. In the RHEL 7 environment, testing with 11 services there are no issues. > This is not related to heat in particular but to the keystone catalogue > size. If there are 10 or more services, the login issue with Keystone v3 > happens on RHEL 6. After deleting services and endpoints so that there are > only 9 services left in the catalogue, I can login. In the RHEL 7 > environment, testing with 11 services there are no issues. Catalog size solution in PKI tokens is "resolved" in upstream Juno by moving back to UUID tokens as default: https://review.openstack.org110488 https://bugs.launchpad.net/keystone/+bug/1350000 Could RHEL6 vs RHEL7 be due to different configured http header size in httpd config? RHEL6 and RHEL7 use different versions (or shoud I say: generations) of httpd. httpd-2.2.15-x vs. httpd-2.4.6-x Header sizes in both installations are not modified from default, which is still the same in both versions. Should we document this as a known issue, and recommend to use uuid tokens for this use case and release? This doesn't seem to be a PKI vs UUID issue, I tried setting the provider to keystone.token.providers.uuid.Provider in keystone.conf and restarting openstack-keystone, and still couldn't login. The only consistent workaround so far is to not set Horizon to use Identity v3. I wonder if there are other differences in the httpd configurations we could look into. Here's another workaround: changing the session engine from the "signed_cookie" default to using memcached or a database backend. Memcached is already installed and configured when using packstack so this may be easier. Memcached workaround: 1. Add the following to /etc/openstack-dashboard/local_settings: SESSION_ENGINE = 'django.contrib.sessions.backends.cache' 2. Restart httpd Database workaround: 1. Add a database dictionary to /etc/openstack-dashboard/local_settings (see https://docs.djangoproject.com/en/1.6/ref/settings/#databases for the expected format). I tested this with the mysql backend. 2. Add also: SESSION_ENGINE = 'django.contrib.sessions.backends.db' 3. Go to /usr/share/openstack-dashboard and run the following: $ python manage.py syncdb (no need to create a django superuser when asked) 4. Restart httpd When switching to either I was able to login again. There is some additional information in the Horizon upstream docs about the different implications for each backend: http://docs.openstack.org/developer/horizon/topics/deployment.html#session-storage . Notably this includes recommendations against using memcached in HA scenarios. (There's still something more to investigate because as far as I can tell we also use the signed_cookie backend on RHEL 7, without seeing the same issue there.) I think I found a workaround: switch to database backed sessions. This won't impact nothing else, it just needs to be documented. Esp. this doesn't need to be a 'special' database. How to do it: in /usr/share/openstack-dashboard/openstack_dashboard/settings.py add a setting: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'horizondb', 'USER': '...', 'PASSWORD': '...', 'HOST': 'localhost', } } and change SESSION_ENGINE = .... to: SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db' then: mysql > create database horizondb; exit cd /usr/share/openstack-dashboard/openstack_dashboard ./manage.py syncdb (you don't need to create a superuser, so answer 'n' to the question) finally: service httpd restart (In reply to Matthias Runge from comment #37) > I think I found a workaround: switch to database backed sessions. > This won't impact nothing else, it just needs to be documented. > Esp. this doesn't need to be a 'special' database. > > How to do it: > > in /usr/share/openstack-dashboard/openstack_dashboard/settings.py > add a setting: > > DATABASES = { > 'default': { > 'ENGINE': 'django.db.backends.mysql', > 'NAME': 'horizondb', > 'USER': '...', > 'PASSWORD': '...', > 'HOST': 'localhost', > } > } > > and change > SESSION_ENGINE = .... > to: > SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db' > > then: > mysql > > create database horizondb; > exit > > cd /usr/share/openstack-dashboard/openstack_dashboard > ./manage.py syncdb > > (you don't need to create a superuser, so answer 'n' to the question) > > finally: service httpd restart This workaround has been verified: ================================== openstack-dashboard-theme-2014.1.2-2.el6ost.noarch openstack-dashboard-2014.1.2-2.el6ost.noarch python-django-horizon-2014.1.2-2.el6ost.noarch What is the next step since this is resolved by a configuration change? Documentation, or changing the default? It seems, this is more a configuration issue than an issue with the software itself. comment 38 has everything required here. Hi Matthias, Thank you for comment 40 - we are looking to scope this content for inclusion in the documentation, and I have a quick question about what this configuration change addresses. In short, can we consider this required configuration for logging in to the Dashboard with Keystone v3? Kind regards, Andrew Assigning to Suyog, who is the assigned author for Keystone-related content. Suyog, see comment 38 for the details - looks like this is a procedure we may have to add to the Proof of Concept guide for Packstack installations, and touches on authentication with Keystone. Kind regards, Andrew Andrew, this config change is necessary for this kind of config (having looots of components enabled in keystone). The underlying issue is, that cookies become too long. And in tests, this wasn't reproducible on RHEL7 Anyways, keystone will switch back to uuid tokens, if I'm not completely wrong here. I would suggest to add a paragraph for Horizon: In rare cases, when having many or all services enabled all together, esp. when using keystone v3 authentication, it may be required to do ..... Hi Matthias, Thank you for your response! We'll look at getting adding some content on this procedure in the documentation. Kind regards, Andrew Hi Matthias, I'm mindful of the workaround verified above in Comment 38, and we are considering getting this content added in the documentation. However, I wanted to quickly check on if there's a fix scheduled at some stage for this issue? Kind regards, Suyog The fix is, upstream moved back to using UUIDs for keystone for quite a while now. I can't predict, how this will change in the future. |
Created attachment 918477 [details] Answer file Description of problem: ======================= Horizon failed to login Version-Release number of selected component: ============================================= python-django-horizon-2014.1.1-2.el6ost.noarch openstack-dashboard-2014.1.1-2.el6ost.noarch openstack-dashboard-theme-2014.1.1-2.el6ost.noarch How reproducible: Steps to Reproduce: =================== 1. Afater using Horizon over time, it logs out and can't login Actual results: =============== Can't login Expected results: ================= successfully login Additional info: ================ CONFIG_HORIZON_SSL=y Setup: Controller Networker 2*Compute An answer-file is enclosed