Description of problem: OSP11 -> OSP12 upgrade: Horizon returns Bad Request (400) post upgrade Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-7.0.0-0.20170913050523.0rc2.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP11 with 3 controllers, 2 computes, 3 ceph nodes 2. Upgrade to OSP12 3. Try reaching overcloud Horizon: curl -L http://10.0.0.109 Actual results: <h1>Bad Request (400)</h1>[ Expected results: Horizon dashboard gets loaded. Additional info: I couldn't find much releavant info in the logs. Please let me know what further info I can provide. /var/log/httpd/horizon_access.log inside container: 10.0.0.1 - - [15/Sep/2017:14:47:54 +0000] "GET /dashboard HTTP/1.1" 400 26 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" vhost: ()[root@controller-0 /]# cat /etc/httpd/conf.d/10-horizon_vhost.conf # ************************************ # Vhost template in module puppetlabs-apache # Managed by Puppet # ************************************ <VirtualHost 172.17.1.25:80> ServerName controller-0.internalapi.localdomain ## Vhost docroot DocumentRoot "/var/www/" ## Alias declarations for resources outside the DocumentRoot Alias /dashboard/static "/usr/share/openstack-dashboard/static" ## Directories, there should at least be a declaration for /var/www/ <Directory "/var/www/"> Options FollowSymLinks MultiViews AllowOverride None Require all granted </Directory> ## Logging ErrorLog "/var/log/httpd/horizon_error.log" ServerSignature Off CustomLog "/var/log/httpd/horizon_access.log" "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" ## RedirectMatch rules RedirectMatch permanent ^/$ /dashboard ## Server aliases ServerAlias controller-0.localdomain WSGIApplicationGroup %{GLOBAL} WSGIDaemonProcess apache group=apache processes=3 threads=10 user=apache WSGIProcessGroup apache WSGIScriptAlias /dashboard "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi" </VirtualHost>
Upgrade process completes ok and horizon container is running so I'm suspecting this might happend on deployment as well.
Reproducing "Bad Request (400)" on fresh deployment of OSP12 with SSL: Environment: instack-undercloud-7.3.1-0.20170830213703.el7ost.noarch openstack-tripleo-heat-templates-7.0.0-0.20170901051303.0rc1.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch
Do you think you could enable DEBUG in /etc/openstack_dashboard/local_settings and try again, to possibly get a better error message?
Also, can you try deleting the Horizon cookies from your browser?
DEBUG = True is already set in /etc/openstack-dashboard/local_settings but /var/log/horizon/horizon.log(/var/log/containers/horizon/horizon.log on host) is empty on all nodes Only in /var/log/httpd/horizon_access.log I can see: 10.0.0.1 - - [19/Sep/2017:14:29:44 +0000] "GET /dashboard HTTP/1.1" 400 26 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
/etc/openstack-dashboard/local_settings : http://paste.openstack.org/show/621437/
I have it reproduced, and I'm trying to figure out what is wrong. So far I have ruled out haproxy — because the same error happens when trying to access Horizon directly — and I think we can rule out Apache itself — because the redirect from / to /dashboard works properly (ok, at least the early stages of Apache request handling). That practically leaves mod_wsgi and Django.
I replaced the horizon wsgi with a simple test wsgi script to see if WSGI is working correctly and to see what the application gets in the request. Here's the environment the application is getting: [root@controller-1 ~]# curl http://172.17.1.10:80/dashboard mod_wsgi.listener_port: 80 CONTEXT_DOCUMENT_ROOT: /var/www/ SERVER_SOFTWARE: Apache SCRIPT_NAME: /dashboard mod_wsgi.enable_sendfile: 0 mod_wsgi.handler_script: SERVER_SIGNATURE: REQUEST_METHOD: GET PATH_INFO: SERVER_PROTOCOL: HTTP/1.1 QUERY_STRING: HTTP_USER_AGENT: curl/7.29.0 SERVER_NAME: 172.17.1.10 REMOTE_ADDR: 172.17.1.10 mod_wsgi.queue_start: 1505893886406747 mod_wsgi.request_handler: wsgi-script wsgi.url_scheme: http mod_wsgi.callable_object: application SERVER_PORT: 80 wsgi.multiprocess: True mod_wsgi.input_chunked: 0 SERVER_ADDR: 172.17.1.10 DOCUMENT_ROOT: /var/www/ mod_wsgi.process_group: apache SCRIPT_FILENAME: /usr/share/openstack-dashboard/openstack_dashboard/wsgi/test.wsgi SERVER_ADMIN: [no address given] wsgi.input: <mod_wsgi.Input object at 0x7fa2db2410b0> HTTP_HOST: 172.17.1.10 CONTEXT_PREFIX: wsgi.multithread: True REQUEST_URI: /dashboard HTTP_ACCEPT: */* wsgi.version: (1, 0) GATEWAY_INTERFACE: CGI/1.1 wsgi.run_once: False wsgi.errors: <mod_wsgi.Log object at 0x7fa2db2410f0> REMOTE_PORT: 39566 mod_wsgi.listener_host: 172.17.1.10 REQUEST_SCHEME: http mod_wsgi.version: (3, 4) mod_wsgi.application_group: mod_wsgi.script_reloading: 1 wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fa2db23aaf8> UNIQUE_ID: WcId-spDDqygEy2ulzAXqQAAAAM
Found a clue: https://code.djangoproject.com/ticket/27760
It seems that for some reason Horizon is ignoring the "ALLOWED_HOSTS = ['*', ]" setting in its local_settings file. Adding that line to its settings.py makes it work, but that's of course not a proper solution. I'm still investigating what is happening there.
This gets stranger and stranger. It seems that Horizon is failing to import local_settings file (ImportError: No module named local_settings), but only when it's running as a WSGI script from Apache — it imports it fine when run from the command line. The sys.path looks correct: ['/usr/share/openstack-dashboard', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old ', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages', '/usr/share/openstack-dashboard/openstack_dashboard']
OK, figured it out. The "apache" user doesn't have permissions to read "/etc/openstack-dashboard" or "/etc/openstack-dashboard/local_settings".
Looks like this: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/horizon.yaml#L107-L110 is not getting executed after upgrade.
This wasn't logged, because Horizon also didn't have access to the log file.
The workaround is to run this on every controller: # docker exec -t -i horizon /bin/bash $ touch /var/log/horizon/horizon.log $ chown -R apache:apache /var/log/horizon $ chmod -R a+rx /etc/openstack-dashboard $ apachectl -k graceful $ exit
The above should be run as root.
Could we put some ansible tasks into the upgrade_tasks section of the heat-template here: http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/horizon.yaml#n149 Specifically so that the file gets created, chown and chmod'd correctly.
Proposed https://review.openstack.org/#/c/507408/ as a fix.
The patch doesn't seem to help the problem.
Hmm, it seems that this problem happens not only after upgrade, but also with the initial install.
The patch has merged upstream.
Says that it needs 3 controllers, 2 computes, and 3 ceph nodes. Is this necessary to test this issue? What steps do you go through for the upgrade, specifically? This is extremely urgent. Thanks for your help.
I think that you can test it with any dockerized setup that has Horizon running. It doesn't even have to be an upgrade — the error was present also in normal install.
openstack-tripleo-heat-templates-7.0.3-0.20171024200823.el7ost.noarch: Inside the Horizon container: ()[root@controller-0 /]# ls -ld /etc/openstack-dashboard drwxr-xr-x. 1 horizon horizon 28 Nov 7 11:16 /etc/openstack-dashboard ()[root@controller-0 /]# ls -l /etc/openstack-dashboard total 100 -rw-r-----. 1 horizon horizon 5283 Aug 30 11:08 cinder_policy.json drwxr-xr-x. 2 horizon horizon 4096 Nov 3 21:46 enabled -rw-r-----. 1 horizon horizon 1361 Aug 30 11:08 glance_policy.json -rw-r-----. 1 horizon horizon 4544 Aug 30 11:08 heat_policy.json -rw-r-----. 1 horizon horizon 9822 Aug 30 11:08 keystone_policy.json -rwxr-xr-x. 1 apache apache 30541 Nov 7 11:01 local_settings drwxr-x---. 2 horizon horizon 4096 Nov 3 21:46 local_settings.d -rw-r-----. 1 horizon horizon 9551 Aug 30 11:08 neutron_policy.json -rw-r-----. 1 horizon horizon 15717 Aug 30 11:08 nova_policy.json ()[root@controller-0 /]# ls -ld /var/log/horizon/ drwxr-xr-x. 2 apache apache 25 Nov 7 11:11 /var/log/horizon/ ()[root@controller-0 /]# ls -l /var/log/horizon/ total 8 -rw-r--r--. 1 apache apache 7600 Nov 7 13:14 horizon.log the initial 400 is not reproducible anymore, seeing another not related issue (undercloud) [stack@undercloud-0 ~]$ curl -L http://10.0.0.101 -I HTTP/1.1 301 Moved Permanently Content-length: 0 Location: https://10.0.0.101/ Connection: close HTTP/1.1 301 Moved Permanently Date: Tue, 07 Nov 2017 13:58:45 GMT Server: Apache Location: https://10.0.0.101/dashboard Content-Type: text/html; charset=iso-8859-1 Set-Cookie: SERVERID=controller-1; path=/ Cache-control: private HTTP/1.1 302 FOUND Date: Tue, 07 Nov 2017 13:58:45 GMT Server: Apache Vary: Accept-Language,Cookie X-Frame-Options: SAMEORIGIN Content-Language: en Location: https://10.0.0.101/dashboard/auth/login/?next=/dashboard/ Content-Type: text/html; charset=utf-8 Set-Cookie: SERVERID=controller-1; path=/ HTTP/1.1 500 INTERNAL SERVER ERROR Date: Tue, 07 Nov 2017 13:58:47 GMT Server: Apache Vary: Accept-Language,Cookie X-Frame-Options: SAMEORIGIN Content-Language: en Content-Type: text/html Set-Cookie: SERVERID=controller-1; path=/
Moving this back to ON_QA as it seems that on fresh deployments we still get the error reported initially.
https://review.openstack.org/#/c/511425/ is still not present in the latest build
stable/pike patch merged 2017-11-01
I was experiencing this bug with my 12-beta test deployment. Installed Packages Name : openstack-tripleo-heat-templates Arch : noarch Version : 7.0.3 Release : 0.20171024200823.el7ost Size : 2.7 M Repo : installed From repo : rhel-7-server-openstack-beta-rpms Summary : Heat templates for TripleO URL : https://wiki.openstack.org/wiki/TripleO License : ASL 2.0 Description : OpenStack TripleO Heat Templates is a collection of templates and : tools for building Heat Templates to do deployments of OpenStack. I was able to get past it and successfully deploy horizon using the upstream template (https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/docker/services/horizon.yaml) after I removed the Installed plugins lines 153-163.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462