Bug 1492135 - OSP11 -> OSP12 upgrade: Horizon returns Bad Request (400) post upgrade
Summary: OSP11 -> OSP12 upgrade: Horizon returns Bad Request (400) post upgrade
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
Target Milestone: beta
: 12.0 (Pike)
Assignee: Emilien Macchi
QA Contact: Marius Cornea
Depends On:
TreeView+ depends on / blocked
Reported: 2017-09-15 14:49 UTC by Marius Cornea
Modified: 2018-02-05 19:12 UTC (History)
19 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.3-6.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-12-13 22:08:57 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Launchpad 1719590 0 None None None 2017-09-26 12:39:52 UTC
OpenStack gerrit 511425 0 None MERGED Fix permissions for dockerized horizon 2020-06-26 17:31:25 UTC
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Marius Cornea 2017-09-15 14:49:37 UTC
Description of problem:
OSP11 -> OSP12 upgrade: Horizon returns Bad Request (400) post upgrade

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Deploy OSP11 with 3 controllers, 2 computes, 3 ceph nodes
2. Upgrade to OSP12
3. Try reaching overcloud Horizon:
curl -L

Actual results:
<h1>Bad Request (400)</h1>[

Expected results:
Horizon dashboard gets loaded.

Additional info:
I couldn't find much releavant info in the logs. Please let me know what further info I can provide.

/var/log/httpd/horizon_access.log inside container: - - [15/Sep/2017:14:47:54 +0000] "GET /dashboard HTTP/1.1" 400 26 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"


()[root@controller-0 /]# cat /etc/httpd/conf.d/10-horizon_vhost.conf    
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************

  ServerName controller-0.internalapi.localdomain

  ## Vhost docroot
  DocumentRoot "/var/www/"
  ## Alias declarations for resources outside the DocumentRoot
  Alias /dashboard/static "/usr/share/openstack-dashboard/static"

  ## Directories, there should at least be a declaration for /var/www/

  <Directory "/var/www/">
    Options FollowSymLinks MultiViews
    AllowOverride None
    Require all granted

  ## Logging
  ErrorLog "/var/log/httpd/horizon_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/horizon_access.log" "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" 

  ## RedirectMatch rules
  RedirectMatch permanent  ^/$ /dashboard

  ## Server aliases
  ServerAlias controller-0.localdomain
  WSGIApplicationGroup %{GLOBAL}
  WSGIDaemonProcess apache group=apache processes=3 threads=10 user=apache
  WSGIProcessGroup apache
  WSGIScriptAlias /dashboard "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi"

Comment 1 Marius Cornea 2017-09-15 14:50:33 UTC
Upgrade process completes ok and horizon container is running so I'm suspecting this might happend on deployment as well.

Comment 2 Alexander Chuzhoy 2017-09-15 16:01:11 UTC
Reproducing "Bad Request (400)" on fresh deployment of OSP12 with SSL:


Comment 3 Radomir Dopieralski 2017-09-19 13:54:13 UTC
Do you think you could enable DEBUG in /etc/openstack_dashboard/local_settings and try again, to possibly get a better error message?

Comment 4 Radomir Dopieralski 2017-09-19 13:55:48 UTC
Also, can you try deleting the Horizon cookies from your browser?

Comment 5 Marius Cornea 2017-09-19 14:34:20 UTC
DEBUG = True is already set in /etc/openstack-dashboard/local_settings but /var/log/horizon/horizon.log(/var/log/containers/horizon/horizon.log on host) is empty on all nodes

Only in /var/log/httpd/horizon_access.log I can see: - - [19/Sep/2017:14:29:44 +0000] "GET /dashboard HTTP/1.1" 400 26 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"

Comment 6 Marius Cornea 2017-09-19 14:34:46 UTC
/etc/openstack-dashboard/local_settings :


Comment 7 Radomir Dopieralski 2017-09-19 18:22:24 UTC
I have it reproduced, and I'm trying to figure out what is wrong.

So far I have ruled out haproxy — because the same error happens when trying to access Horizon directly — and I think we can rule out Apache itself — because the redirect from / to /dashboard works properly (ok, at least the early stages of Apache request handling).

That practically leaves mod_wsgi and Django.

Comment 8 Radomir Dopieralski 2017-09-20 07:53:55 UTC
I replaced the horizon wsgi with a simple test wsgi script to see if WSGI is working correctly and to see what the application gets in the request. Here's the environment the application is getting:

[root@controller-1 ~]# curl
mod_wsgi.listener_port: 80
SCRIPT_NAME: /dashboard
mod_wsgi.enable_sendfile: 0
HTTP_USER_AGENT: curl/7.29.0
mod_wsgi.queue_start: 1505893886406747
mod_wsgi.request_handler: wsgi-script
wsgi.url_scheme: http
mod_wsgi.callable_object: application
wsgi.multiprocess: True
mod_wsgi.input_chunked: 0
DOCUMENT_ROOT: /var/www/
mod_wsgi.process_group: apache
SCRIPT_FILENAME: /usr/share/openstack-dashboard/openstack_dashboard/wsgi/test.wsgi
SERVER_ADMIN: [no address given]
wsgi.input: <mod_wsgi.Input object at 0x7fa2db2410b0>
wsgi.multithread: True
REQUEST_URI: /dashboard
wsgi.version: (1, 0)
wsgi.run_once: False
wsgi.errors: <mod_wsgi.Log object at 0x7fa2db2410f0>
mod_wsgi.version: (3, 4)
mod_wsgi.script_reloading: 1
wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fa2db23aaf8>

Comment 9 Radomir Dopieralski 2017-09-20 08:31:21 UTC
Found a clue: https://code.djangoproject.com/ticket/27760

Comment 10 Radomir Dopieralski 2017-09-20 08:52:38 UTC
It seems that for some reason Horizon is ignoring the "ALLOWED_HOSTS = ['*', ]" setting in its local_settings file. Adding that line to its settings.py makes it work, but that's of course not a proper solution.

I'm still investigating what is happening there.

Comment 11 Radomir Dopieralski 2017-09-20 09:22:37 UTC
This gets stranger and stranger. It seems that Horizon is failing to import local_settings file (ImportError: No module named local_settings), but only when it's running as a WSGI script from Apache — it imports it fine when run from the command line. The sys.path looks correct: ['/usr/share/openstack-dashboard', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old
', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages', '/usr/share/openstack-dashboard/openstack_dashboard']

Comment 12 Radomir Dopieralski 2017-09-20 09:36:34 UTC
OK, figured it out.

The "apache" user doesn't have permissions to read "/etc/openstack-dashboard" or "/etc/openstack-dashboard/local_settings".

Comment 13 Radomir Dopieralski 2017-09-20 09:41:44 UTC
Looks like this: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/horizon.yaml#L107-L110 is not getting executed after upgrade.

Comment 14 Radomir Dopieralski 2017-09-20 09:44:20 UTC
This wasn't logged, because Horizon also didn't have access to the log file.

Comment 15 Radomir Dopieralski 2017-09-20 11:26:24 UTC
The workaround is to run this on every controller:

# docker exec -t -i horizon /bin/bash
$ touch /var/log/horizon/horizon.log
$ chown -R apache:apache /var/log/horizon
$ chmod -R a+rx /etc/openstack-dashboard
$ apachectl -k graceful
$ exit

Comment 16 Radomir Dopieralski 2017-09-20 11:27:25 UTC
The above should be run as root.

Comment 17 Dan Prince 2017-09-25 21:20:20 UTC
Could we put some ansible tasks into the upgrade_tasks section of the heat-template here:


Specifically so that the file gets created, chown and chmod'd correctly.

Comment 18 Radomir Dopieralski 2017-09-26 08:21:54 UTC
Proposed https://review.openstack.org/#/c/507408/ as a fix.

Comment 19 Radomir Dopieralski 2017-09-26 10:40:16 UTC
The patch doesn't seem to help the problem.

Comment 20 Radomir Dopieralski 2017-09-26 11:02:23 UTC
Hmm, it seems that this problem happens not only after upgrade, but also with the initial install.

Comment 21 Radomir Dopieralski 2017-10-11 11:54:01 UTC
The patch has merged upstream.

Comment 24 Jason E. Rist 2017-11-07 03:54:59 UTC
Says that it needs 3 controllers, 2 computes, and 3 ceph nodes.

Is this necessary to test this issue?

What steps do you go through for the upgrade, specifically?

This is extremely urgent. Thanks for your help.

Comment 25 Radomir Dopieralski 2017-11-07 07:45:54 UTC
I think that you can test it with any dockerized setup that has Horizon running. It doesn't even have to be an upgrade — the error was present also in normal install.

Comment 26 Marius Cornea 2017-11-07 13:59:15 UTC

Inside the Horizon container:

()[root@controller-0 /]# ls -ld /etc/openstack-dashboard
drwxr-xr-x. 1 horizon horizon 28 Nov  7 11:16 /etc/openstack-dashboard
()[root@controller-0 /]# ls -l /etc/openstack-dashboard 
total 100
-rw-r-----. 1 horizon horizon  5283 Aug 30 11:08 cinder_policy.json
drwxr-xr-x. 2 horizon horizon  4096 Nov  3 21:46 enabled
-rw-r-----. 1 horizon horizon  1361 Aug 30 11:08 glance_policy.json
-rw-r-----. 1 horizon horizon  4544 Aug 30 11:08 heat_policy.json
-rw-r-----. 1 horizon horizon  9822 Aug 30 11:08 keystone_policy.json
-rwxr-xr-x. 1 apache  apache  30541 Nov  7 11:01 local_settings
drwxr-x---. 2 horizon horizon  4096 Nov  3 21:46 local_settings.d
-rw-r-----. 1 horizon horizon  9551 Aug 30 11:08 neutron_policy.json
-rw-r-----. 1 horizon horizon 15717 Aug 30 11:08 nova_policy.json

()[root@controller-0 /]# ls -ld /var/log/horizon/
drwxr-xr-x. 2 apache apache 25 Nov  7 11:11 /var/log/horizon/
()[root@controller-0 /]# ls -l /var/log/horizon/ 
total 8
-rw-r--r--. 1 apache apache 7600 Nov  7 13:14 horizon.log

the initial 400 is not reproducible anymore, seeing another not related issue

(undercloud) [stack@undercloud-0 ~]$ curl -L -I
HTTP/1.1 301 Moved Permanently
Content-length: 0
Connection: close

HTTP/1.1 301 Moved Permanently
Date: Tue, 07 Nov 2017 13:58:45 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1
Set-Cookie: SERVERID=controller-1; path=/
Cache-control: private

HTTP/1.1 302 FOUND
Date: Tue, 07 Nov 2017 13:58:45 GMT
Server: Apache
Vary: Accept-Language,Cookie
X-Frame-Options: SAMEORIGIN
Content-Language: en
Content-Type: text/html; charset=utf-8
Set-Cookie: SERVERID=controller-1; path=/

Date: Tue, 07 Nov 2017 13:58:47 GMT
Server: Apache
Vary: Accept-Language,Cookie
X-Frame-Options: SAMEORIGIN
Content-Language: en
Content-Type: text/html
Set-Cookie: SERVERID=controller-1; path=/

Comment 27 Marius Cornea 2017-11-07 16:30:49 UTC
Moving this back to ON_QA as it seems that on fresh deployments we still get the error reported initially.

Comment 28 Marius Cornea 2017-11-07 16:44:16 UTC
https://review.openstack.org/#/c/511425/ is still not present in the latest build

Comment 29 Jon Schlueter 2017-11-08 02:58:30 UTC
stable/pike patch merged 2017-11-01

Comment 34 Will Kline 2017-11-28 18:50:00 UTC
I was experiencing this bug with my 12-beta test deployment.

Installed Packages
Name        : openstack-tripleo-heat-templates
Arch        : noarch
Version     : 7.0.3
Release     : 0.20171024200823.el7ost
Size        : 2.7 M
Repo        : installed
From repo   : rhel-7-server-openstack-beta-rpms
Summary     : Heat templates for TripleO
URL         : https://wiki.openstack.org/wiki/TripleO
License     : ASL 2.0
Description : OpenStack TripleO Heat Templates is a collection of templates and 
            : tools for building Heat Templates to do deployments of OpenStack.

I was able to get past it and successfully deploy horizon using the upstream template (https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/docker/services/horizon.yaml) after I removed the Installed plugins lines 153-163.

Comment 37 errata-xmlrpc 2017-12-13 22:08:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.