Bug 1028678

Summary: Accessing horizon using hostname instead of ip ends with exception
Product: Red Hat OpenStack Reporter: Rami Vaknin <rvaknin>
Component: openstack-packstackAssignee: Julie Pichon <jpichon>
Status: CLOSED ERRATA QA Contact: Rami Vaknin <rvaknin>
Severity: high Docs Contact:
Priority: high    
Version: 4.0CC: aortega, athomas, breeler, derekh, hateya, jdexter, jpichon, mmagr, mrunge, oblaut, ramon, rhos-maint, rvaknin, sgordon, yeylon
Target Milestone: rcKeywords: Regression
Target Release: 4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-packstack-2013.2.1-0.13.dev876.el6ost Doc Type: Known Issue
Doc Text:
Horizon is built on top of Django. Django 1.5 makes the ALLOWED_HOSTS setting mandatory as a security measure. (see https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts) Also, recent changes in PackStack cause this variable to be set when using Django 1.4. The consequence is that the Horizon Dashboard only becomes accessible when using one of the host names explicitly defined in ALLOWED_HOSTS. Otherwise, an "error 500" occurs. Workaround: Update ALLOWED_HOSTS in /etc/openstack-dashboard/local_settings to include the hostname(s) you wish to use for the Dashboard. Then restart httpd. The dashboard will now work when accessing it using the desired hostname(s).
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-20 00:34:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rami Vaknin 2013-11-09 13:16:50 UTC
Version
=======
rhos 4.0 on rhel 6.5, puddle 2013-11-08.1
python-django-horizon-2013.2-3.el6ost.noarch

Description
===========
When accessing horizon using the ip address (http://10.35.160.29/dashboard) - it works well, however, when trying to access it via hostname (http://puma10.scl.lab.tlv.redhat.com/dashboard/) - it fails, the following error messsage appears:

"Something went wrong!
An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator."

Also, openstack-status can't detect horizon as active:

# openstack-status | grep dashboard
openstack-dashboard:          500


from /var/log/horizon/horizon.log
=================================
2013-11-09 11:43:37,017 2176 ERROR django.request Internal Server Error: /dashboard/
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 89, in get_response
    response = middleware_method(request)
  File "/usr/lib/python2.6/site-packages/django/middleware/common.py", line 55, in process_request
    host = request.get_host()
  File "/usr/lib/python2.6/site-packages/django/http/__init__.py", line 223, in get_host
    "Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): %s" % host)
SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): puma10.scl.lab.tlv.redhat.com

Comment 1 Matthias Runge 2013-11-11 09:13:53 UTC
Did you install via packstack?

Is ALLOWED_HOSTS in /etc/openstack-dashboard configured? If yes, please list the value here.

Comment 2 Rami Vaknin 2013-11-11 09:37:39 UTC
I don't have the original env, the output is from a similar env:

# grep ALLO /etc/openstack-dashboard/*
/etc/openstack-dashboard/local_settings:#ALLOWED_HOSTS = ['horizon.example.com', ]
/etc/openstack-dashboard/local_settings:ALLOWED_HOSTS = ['10.35.160.29', ]

# rpm -qa | grep -i django
python-django-appconf-0.5-2.el6.noarch
python-django-horizon-2013.2-3.el6ost.noarch
Django14-1.4.8-1.el6ost.noarch
python-django-openstack-auth-1.1.2-1.el6ost.noarch
python-django-compressor-1.3-1.el6ost.noarch

Comment 3 Julie Pichon 2013-11-11 09:51:24 UTC
This is probably a regression introduced by bug 988316.

I'm not sure what's the best way to handle it. Should a new hostname variable be added to the packstack config file, to set it if it differs from the default fqdn? (And then use that instead at the end of the packstack output, for directing people to the console.)

Perhaps we could set ALLOWED_HOSTS to '*' as was initially suggested in the review, and leave updating the ALLOWED_HOSTS variable as an instruction in the documentation on how to secure the dashboard, if such docs exist. Or is the expectation that Packstack can somehow do it all?

Comment 4 Matthias Runge 2013-11-11 11:08:58 UTC
I wouldn't recommend to allow all hosts for allowed hosts.

From the corresponding release notes (added to Django-1.4.4)

Host header poisoning

Some parts of Django – independent of end-user-written applications – make use of full URLs, including domain name, which are generated from the HTTP Host header. Django’s documentation has for some time contained notes advising users on how to configure webservers to ensure that only valid Host headers can reach the Django application. However, it has been reported to us that even with the recommended webserver configurations there are still techniques available for tricking many common webservers into supplying the application with an incorrect and possibly malicious Host header.

For this reason, Django 1.4.4 adds a new setting, ALLOWED_HOSTS, containing an explicit list of valid host/domain names for this site. A request with a Host header not matching an entry in this list will raise SuspiciousOperation if request.get_host() is called. For full details see the documentation for the ALLOWED_HOSTS setting.

The default value for this setting in Django 1.4.4 is ['*'] (matching any host), for backwards-compatibility, but we strongly encourage all sites to set a more restrictive value.

This host validation is disabled when DEBUG is True or when running tests.


So, it's a security hardening method.

Comment 6 Julie Pichon 2013-11-15 13:21:58 UTC
I don't think the resolution is appropriate, this seems like a consequence of that patch, rather than a symptom: it *works* with the IP but not with the hostname.

Alvaro was suggesting doing something like `socket.gethostbyaddr("1.1.1.1")` to guess at the hostname but I'm not sure how far we want to go to try and guess what the deployer actually wants.

Comment 7 Scott Lewis 2013-11-18 20:35:38 UTC
Adding to beta milestone for MODIFIED/ON-QA bugs

Comment 9 Rami Vaknin 2013-11-20 10:49:35 UTC
Could you please update about the status of this bug? it's still in MODIFIED but I wonder why it's not fixed in the openstack-packstack that appears in the "Fixed in Version" above, is that a release note only solution?

Comment 10 Julie Pichon 2013-11-20 11:12:59 UTC
After chatting with mrunge and mmagr, I'll try and tweak the patch to allow both the Horizon host IP and fqdn. The Doc Text note should still be included I think, as we can't necessarily guess what the user will want and ALLOWED_HOSTS will likely need to be tweaked.

Comment 11 Julie Pichon 2013-12-04 14:57:47 UTC
The patch is included at least in the latest build, openstack-packstack-2013.2.1-0.13.dev876.el6ost - changing status to MODIFIED.

Comment 13 Rami Vaknin 2013-12-08 11:48:11 UTC
Now I'm able to login to the dashboard successfully, however openstack-status still does not report horizon as active:

# openstack-status | grep dashboard
openstack-dashboard:                    500

What are we going to do with that? should we report that in another bug and close this one as verified?

Tested on rhos 4.0 on rhel 6.5 with 2013-12-06.3 puddle, openstack-packstack-2013.2.1-0.13.dev876.el6ost.

Comment 14 Rami Vaknin 2013-12-08 15:49:39 UTC
btw, when installing with nova-network multihost, I get the following:

[Sun Dec 08 17:46:45 2013] [warn] [client 10.35.7.239] incomplete redirection target of '/dashboard/' for URI '/' modified to 'http://puma10.scl.lab.tlv.redhat.com/dashboard/'
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239] mod_wsgi (pid=2011): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239] Traceback (most recent call last):
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]   File "/usr/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 219, in __call__
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]     self.load_middleware()
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]   File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 39, in load_middleware
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]     for middleware_path in settings.MIDDLEWARE_CLASSES:
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]   File "/usr/lib/python2.6/site-packages/django/utils/functional.py", line 184, in inner
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]     self._setup()
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]   File "/usr/lib/python2.6/site-packages/django/conf/__init__.py", line 42, in _setup
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]     self._wrapped = Settings(settings_module)
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]   File "/usr/lib/python2.6/site-packages/django/conf/__init__.py", line 95, in __init__
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239]     raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
[Sun Dec 08 17:46:45 2013] [error] [client 10.35.7.239] ImportError: Could not import settings 'openstack_dashboard.settings' (Is it on sys.path?): No module named pbr.version

Comment 15 Rami Vaknin 2013-12-08 20:48:42 UTC
I reported the dashboard exception in:
https://bugzilla.redhat.com/show_bug.cgi?id=1039368

I think that the openstack-status issue should be fixed in this patch too hence failing this bug verification.

From /usr/bin/openstack-status:

if test "$dashboard"; then
  printf "== Horizon service ==\n"
  horizon_status="$(curl -s -w '%{http_code}\n' http://localhost/dashboard -o /dev/null)"
  [ "$horizon_status" = 200 ] && horizon_status=active
  printf '%-40s%s\n' "openstack-dashboard:" "$horizon_status"
fi


The fix in https://review.openstack.org/#/c/57693/1/packstack/puppet/templates/horizon.pp should also contain "localhost" in addition to the ip and fqdn of the horizon machine, this way openstack-status will have an access to query horizon and will mark the openstack-dashboard service as "active".

Comment 16 Martin Magr 2013-12-09 13:49:08 UTC
If the problem is only in openstack-status, then please open new bug for that. You can verify this bug just with accessing horizon via your browser.

Comment 17 Rami Vaknin 2013-12-09 15:21:46 UTC
The bug originally contained two symptoms, caused by the very same reason in the same time, and the fix should be in the same line, I don't think this bug should be verified while only one of those symptoms (the more severe) was fixed while the second one not.

Yaniv, could I get your advice here?

Comment 18 Julie Pichon 2013-12-09 18:33:32 UTC
I would also have preferred this to be 2 separate bugs since the severity is different and they may have different solutions. Ideally we'd be able to configure which URL openstack-utils uses but it doesn't seem to be possible, so perhaps the path of least resistance at the moment is to also add localhost to the ALLOWED_HOSTS list.

I proposed https://review.openstack.org/#/c/60898/ , after running packstack with this patch included openstack-status shows the following output:

== Horizon service ==
openstack-dashboard:          active

Comment 19 Rami Vaknin 2013-12-09 21:09:08 UTC
Verified based on comment#13.

I've opened the following bug for the openstack-status issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1039727

Comment 20 Rami Vaknin 2013-12-09 21:09:42 UTC
Clearing the needinfo flag.

Comment 21 Julie Pichon 2013-12-12 17:14:58 UTC
Thank you for the doc text update Bruce. Packstack does its best to set sensible defaults, however it is quite possible a user will want to use another hostname. Because of this, I think it would be good to include the doc text in order to highlight this new setting, either as a known issue or as a release note.

Comment 23 errata-xmlrpc 2013-12-20 00:34:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Comment 24 Don Domingo 2014-07-09 03:33:38 UTC
*** Bug 1040688 has been marked as a duplicate of this bug. ***