Bug 1120305

Summary: Horizon failed to login - manually setting to use keystone v3
Product: Red Hat OpenStack Reporter: Ido Ovadia <iovadia>
Component: doc-Deployment_GuideAssignee: Suyog Sainkar <ssainkar>
Status: CLOSED CURRENTRELEASE QA Contact: Ido Ovadia <iovadia>
Severity: high Docs Contact:
Priority: high    
Version: 5.0 (RHEL 6)CC: adahms, ajeain, aortega, apevec, ayoung, iovadia, jpichon, mrunge, sclewis, yeylon
Target Milestone: asyncKeywords: Documentation, Regression, Triaged, ZStream
Target Release: 5.0 (RHEL 6)   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-13 05:43:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186518    
Attachments:
Description Flags
Answer file
none
answer file, all-in-one, no SSL, Heat=y
none
RHEL 6.5 - rpm list
none
RHEL 7 - rpm list none

Description Ido Ovadia 2014-07-16 17:01:29 UTC
Created attachment 918477 [details]
Answer file

Description of problem:
=======================
Horizon failed to login

Version-Release number of selected component:
=============================================
python-django-horizon-2014.1.1-2.el6ost.noarch
openstack-dashboard-2014.1.1-2.el6ost.noarch
openstack-dashboard-theme-2014.1.1-2.el6ost.noarch

How reproducible:

Steps to Reproduce:
===================
1. Afater using Horizon over time, it logs out and can't login


Actual results:
===============
Can't login

Expected results:
=================
successfully login  

Additional info:
================
CONFIG_HORIZON_SSL=y

Setup: 
Controller
Networker
2*Compute

An answer-file is enclosed

Comment 2 Julie Pichon 2014-07-17 08:13:47 UTC
Could you set the logging level to DEBUG in the settings - in the local_settings, in the LOGGING dictionary, making sure level is set to DEBUG for 'file' like this:

        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/var/log/horizon.log',
            'formatter': 'verbose',
        },

Then empty the contents of horizon.log, try to login again and attach the logs here?

After that - does deleting the cookies help with login in? Restarting httpd?

Comment 3 Matthias Runge 2014-07-17 10:34:51 UTC
I assume, this is an issue with keystone v3.
That setting was changed after packstack install:
OPENSTACK_API_VERSION was set to 'identity': 3

Setting back/uncommenting makes that setup work immediately.

I think, the issue is valid on rhel6 and rhel7.

How to reproduce:
1. set up horizon via packstack
2. change in /etc/openstack_dashboard/local_settings:
(lines 37 to 39 to read:)
OPENSTACK_API_VERSIONS = {
    "identity": 3
}
3. restart httpd
4. Try to log in: after typing in username/password, nothing happens
5. comment in lines 37 to 39
6. restart httpd
7. log in

Comment 4 Matthias Runge 2014-07-17 10:35:42 UTC
To make it more clear: there is no issue, when using plain packstack install

Comment 5 Matthias Runge 2014-07-17 11:43:05 UTC
the reproducer does not work locally here on f20 and on rhel7 :-/

Comment 7 Ido Ovadia 2014-07-17 15:24:53 UTC
workaround?
===========

After step 2:
  change in /etc/openstack_dashboard/local_settings:
  (lines 37 to 39 to read:)
  OPENSTACK_API_VERSIONS = {
      "identity": 3
  }

Reboot the system

Comment 8 Matthias Runge 2014-07-17 18:18:22 UTC
Ido,

to clarify: #c7 is, what what you did, to produce this issue?

Afterwards, the same steps solved it?

Comment 9 Matthias Runge 2014-07-18 08:51:55 UTC
Because that whole thing is still a bit unclear to me, I set up a RHOS-5 system on RHEL6, enabled ssl for httpd and changed /etc/openstack-dashboard/local_settings according to #c7.

I was immediately able to log into the system.

Ido, did you change on your system something else as well?

Comment 14 Julie Pichon 2014-08-11 12:19:59 UTC
Ido, could you provide logs from around the time when the problem occurs (failure to login): both the horizon logs (in debug mode ideally, so we can see the requests sent with the keystone client), as well as the keystone logs when trying to log in?

Comment 19 Julie Pichon 2014-08-15 09:51:08 UTC
I let the system idle for the last few days and haven't managed to reproduce yet. I tried to properly log out then logged in again, I tried to let my session time out then log in. What kind of activities were performed on the system before the failure started?

Another important package version I forgot to note in comment 15:
python-django-openstack-auth-1.1.5-2.el7ost.noarch

Ido (or anyone who can reproduce), in addition to logs could you also provide version numbers for these packages? Thank you.

Comment 22 Ido Ovadia 2014-08-18 15:01:01 UTC
This bug referce to RHEL 6

I will try to reproduce it till weekend

Comment 23 Ido Ovadia 2014-08-25 14:34:02 UTC
This bug tested both: RHEL 6 and RHEL 7 and it is only broken on RHEL 6

The bug reproduced only on RHEL 6

It was last reproduced on:
========================== 
OpenStack-5.0-RHEL-6 Puddle: 2014-08-22.1

python-django-horizon-2014.1.2-1.el6ost.noarch
openstack-dashboard-theme-2014.1.2-1.el6ost.noarch
openstack-dashboard-2014.1.2-1.el6ost.noarch

Comment 24 Julie Pichon 2014-08-25 17:12:11 UTC
This is very strange. Ido gave me access to the system where the problem is occurring, I can see it although there's nothing obvious in the horizon, httpd or keystone logs as to the cause. Switching the Horizon setting between Keystone v2 and v3 does make the problem go away and reappear.

I set up 2 RHEL 6.5 environments and ran packstack on both, one where Horizon is set up with SSL (CONFIG_HORIZON_SSL=y in packstack) and one where it isn't. I changed the Keystone setting to v3, restarted httpd and I still cannot reproduce the problem. The django, horizon, django_openstack_auth, keystone and keystone client package versions are the same as on Ido's system. I ran yum update and rebooted, still I can login just fine. Ido indicated that the problem could be reproduced as soon as the version was changed this time, but I'll still let the systems idle for a couple of days to see if that changes something, and also see if I can figure out the differences with Ido's environment.

Comment 25 Ido Ovadia 2014-08-26 08:09:25 UTC
Created attachment 930781 [details]
answer file, all-in-one, no SSL, Heat=y

Comment 26 Ido Ovadia 2014-08-26 08:13:32 UTC
Reproduced again: allinone CONFIG_HORIZON_SSL=n, CONFIG_HEAT_INSTALL=y
Same version: OpenStack-5.0-RHEL-6 Puddle: 2014-08-22.1

Julie, try to use the new enclosed answer file.

I will try to reproduce without Heat

Comment 27 Ido Ovadia 2014-08-26 09:37:10 UTC
Created attachment 930794 [details]
RHEL 6.5 - rpm list

Comment 28 Ido Ovadia 2014-08-26 09:37:38 UTC
Created attachment 930795 [details]
RHEL 7 - rpm list

Comment 29 Ido Ovadia 2014-08-26 09:44:52 UTC
This bug reproduced on RHEL 6.5 only, I used the enclosed answerfile for both RHEL 6.5 and RHEL 7.

rpm lists are enclosed too

Comment 30 Julie Pichon 2014-08-26 09:55:24 UTC
I was able to reproduce on RHEL 6.5 with CONFIG_HORIZON_SSL=y, CONFIG_HEAT_INSTALL=y. I'm assuming something is configured differently when also installing Heat, although there's no differences standing out in the configuration files so far and I can't quite explain why it would only reproduce on RHEL 6 either.

Comment 31 Julie Pichon 2014-08-26 15:35:00 UTC
This is not related to heat in particular but to the keystone catalogue size. If there are 10 or more services, the login issue with Keystone v3 happens on RHEL 6. After deleting services and endpoints so that there are only 9 services left in the catalogue, I can login. In the RHEL 7 environment, testing with 11 services there are no issues.

Comment 33 Alan Pevec 2014-09-08 14:27:32 UTC
> This is not related to heat in particular but to the keystone catalogue
> size. If there are 10 or more services, the login issue with Keystone v3
> happens on RHEL 6. After deleting services and endpoints so that there are
> only 9 services left in the catalogue, I can login. In the RHEL 7
> environment, testing with 11 services there are no issues.

Catalog size solution in PKI tokens is "resolved" in upstream Juno by moving back to UUID tokens as default: https://review.openstack.org110488
https://bugs.launchpad.net/keystone/+bug/1350000

Could RHEL6 vs RHEL7 be due to different configured http header size in httpd config?

Comment 34 Matthias Runge 2014-09-09 07:04:50 UTC
RHEL6 and RHEL7 use different versions (or shoud I say: generations) of httpd.

httpd-2.2.15-x
vs.
httpd-2.4.6-x

Header sizes in both installations are not modified from default, which is still the same in both versions.

Should we document this as a known issue, and recommend to use uuid tokens for this use case and release?

Comment 35 Julie Pichon 2014-09-11 06:52:55 UTC
This doesn't seem to be a PKI vs UUID issue, I tried setting the provider to keystone.token.providers.uuid.Provider in keystone.conf and restarting openstack-keystone, and still couldn't login. The only consistent workaround so far is to not set Horizon to use Identity v3.

I wonder if there are other differences in the httpd configurations we could look into.

Comment 36 Julie Pichon 2014-09-15 01:22:12 UTC
Here's another workaround: changing the session engine from the "signed_cookie" default to using memcached or a database backend. Memcached is already installed and configured when using packstack so this may be easier.

Memcached workaround:
1. Add the following to /etc/openstack-dashboard/local_settings:
    SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
2. Restart httpd

Database workaround:
1. Add a database dictionary to /etc/openstack-dashboard/local_settings (see https://docs.djangoproject.com/en/1.6/ref/settings/#databases for the expected format). I tested this with the mysql backend.
2. Add also:
    SESSION_ENGINE = 'django.contrib.sessions.backends.db'
3. Go to /usr/share/openstack-dashboard and run the following:
    $ python manage.py syncdb
    (no need to create a django superuser when asked)
4. Restart httpd

When switching to either I was able to login again.

There is some additional information in the Horizon upstream docs about the different implications for each backend: http://docs.openstack.org/developer/horizon/topics/deployment.html#session-storage . Notably this includes recommendations against using memcached in HA scenarios.

(There's still something more to investigate because as far as I can tell we also use the signed_cookie backend on RHEL 7, without seeing the same issue there.)

Comment 37 Matthias Runge 2014-10-02 10:47:07 UTC
I think I found a workaround: switch to database backed sessions.
This won't impact nothing else, it just needs to be documented.
Esp. this doesn't need to be a 'special' database.

How to do it:

in /usr/share/openstack-dashboard/openstack_dashboard/settings.py
add a setting:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'horizondb',
        'USER': '...',
        'PASSWORD': '...',
        'HOST': 'localhost',
    }
}

and change
SESSION_ENGINE = ....
to:
SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'

then:
mysql >
create database horizondb;
exit

cd /usr/share/openstack-dashboard/openstack_dashboard
./manage.py syncdb

(you don't need to create a superuser, so answer 'n' to the question)

finally: service httpd restart

Comment 38 Ido Ovadia 2014-10-06 14:40:38 UTC
(In reply to Matthias Runge from comment #37)
> I think I found a workaround: switch to database backed sessions.
> This won't impact nothing else, it just needs to be documented.
> Esp. this doesn't need to be a 'special' database.
> 
> How to do it:
> 
> in /usr/share/openstack-dashboard/openstack_dashboard/settings.py
> add a setting:
> 
> DATABASES = {
>     'default': {
>         'ENGINE': 'django.db.backends.mysql',
>         'NAME': 'horizondb',
>         'USER': '...',
>         'PASSWORD': '...',
>         'HOST': 'localhost',
>     }
> }
> 
> and change
> SESSION_ENGINE = ....
> to:
> SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'
> 
> then:
> mysql >
> create database horizondb;
> exit
> 
> cd /usr/share/openstack-dashboard/openstack_dashboard
> ./manage.py syncdb
> 
> (you don't need to create a superuser, so answer 'n' to the question)
> 
> finally: service httpd restart


This workaround has been verified:
==================================
openstack-dashboard-theme-2014.1.2-2.el6ost.noarch
openstack-dashboard-2014.1.2-2.el6ost.noarch
python-django-horizon-2014.1.2-2.el6ost.noarch

Comment 39 Julie Pichon 2014-11-27 16:47:18 UTC
What is the next step since this is resolved by a configuration change? Documentation, or changing the default?

Comment 40 Matthias Runge 2014-12-02 08:26:31 UTC
It seems, this is more a configuration issue than an issue with the software itself.

comment 38 has everything required here.

Comment 42 Andrew Dahms 2014-12-05 03:36:06 UTC
Hi Matthias,

Thank you for comment 40 - we are looking to scope this content for inclusion in the documentation, and I have a quick question about what this configuration change addresses.

In short, can we consider this required configuration for logging in to the Dashboard with Keystone v3?

Kind regards,

Andrew

Comment 43 Andrew Dahms 2014-12-05 03:37:13 UTC
Assigning to Suyog, who is the assigned author for Keystone-related content.

Suyog, see comment 38 for the details - looks like this is a procedure we may have to add to the Proof of Concept guide for Packstack installations, and touches on authentication with Keystone.

Kind regards,

Andrew

Comment 44 Matthias Runge 2014-12-05 07:35:57 UTC
Andrew,

this config change is necessary for this kind of config (having looots of components enabled in keystone). The underlying issue is, that cookies become too long. And in tests, this wasn't reproducible on RHEL7

Anyways, keystone will switch back to uuid tokens, if I'm not completely wrong here.

I would suggest to add a paragraph for Horizon:

In rare cases, when having many or all services enabled all together, esp. when using keystone v3 authentication, it may be required to do .....

Comment 45 Andrew Dahms 2014-12-09 11:49:29 UTC
Hi Matthias,

Thank you for your response! We'll look at getting adding some content on this procedure in the documentation.

Kind regards,

Andrew

Comment 46 Suyog Sainkar 2015-02-03 06:58:21 UTC
Hi Matthias, 

I'm mindful of the workaround verified above in Comment 38, and we are considering getting this content added in the documentation. However, I wanted to quickly check on if there's a fix scheduled at some stage for this issue?

Kind regards,

Suyog

Comment 47 Matthias Runge 2015-02-03 07:05:44 UTC
The fix is, upstream moved back to using UUIDs for keystone for quite a while now.

I can't predict, how this will change in the future.