Hide Forgot
Description of problem: Installed Mitaka in a virtualized environment according to https://www.rdoproject.org/testday/mitaka/milestone2/ and https://www.rdoproject.org/rdo-manager/. After rebooting the controller, when trying to access horizon I see: 503 Service Unavailable No server is available to handle this request. There are exceptions in /var/log/httpd/horizon_error.log: mod_wsgi (pid=7703): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 170, in __call__ self.load_middleware() File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 52, in load_middleware mw_instance = mw_class() File "/usr/lib/python2.7/site-packages/django/middleware/locale.py", line 24, in __init__ for url_pattern in get_resolver(None).url_patterns: File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 401, in url_patterns patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module) File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 395, in urlconf_module self._urlconf_module = import_module(self.urlconf_name) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/urls.py", line 35, in <module> url(r'^api/', include('openstack_dashboard.api.rest.urls')), File "/usr/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 33, in include urlconf_module = import_module(urlconf_module) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/__init__.py", line 36, in <module> from openstack_dashboard.api import cinder File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/cinder.py", line 30, in <module> from cinderclient.v2.contrib import list_extensions as cinder_list_extensions File "/usr/lib/python2.7/site-packages/cinderclient/v2/__init__.py", line 17, in <module> from cinderclient.v2.client import Client # noqa File "/usr/lib/python2.7/site-packages/cinderclient/v2/client.py", line 16, in <module> from cinderclient import client File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 52, in <module> from eventlet import sleep File "/usr/lib/python2.7/site-packages/eventlet/__init__.py", line 10, in <module> from eventlet import convenience File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 3, in <module> from eventlet import greenio File "/usr/lib/python2.7/site-packages/eventlet/greenio/__init__.py", line 3, in <module> from eventlet.greenio.base import * # noqa File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 441, in <module> from OpenSSL import SSL File "/usr/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import rand, crypto, SSL File "/usr/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module> from OpenSSL._util import ( File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 7, in <module> binding = Binding() File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 63, in __init__ self._ensure_ffi_initialized() File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 95, in _ensure_ffi_initialized cls._register_osrandom_engine() File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 70, in _register_osrandom_engine raise RuntimeError("osrandom engine already registered") RuntimeError: osrandom engine already registered mod_wsgi (pid=7703): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 170, in __call__ self.load_middleware() File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 52, in load_middleware mw_instance = mw_class() File "/usr/lib/python2.7/site-packages/django/middleware/locale.py", line 24, in __init__ for url_pattern in get_resolver(None).url_patterns: File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 401, in url_patterns patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module) File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 395, in urlconf_module self._urlconf_module = import_module(self.urlconf_name) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/urls.py", line 35, in <module> url(r'^api/', include('openstack_dashboard.api.rest.urls')), File "/usr/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 33, in include urlconf_module = import_module(urlconf_module) File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/__init__.py", line 34, in <module> from openstack_dashboard.api import base ImportError: cannot import name base Version-Release number of selected component (if applicable): python-django-horizon-9.0.0.0b3-dev67.el7.centos.noarch How reproducible: 100% Steps to Reproduce: 1. Install Mitaka according to https://www.rdoproject.org/testday/mitaka/milestone2/ and https://www.rdoproject.org/rdo-manager/ 2. Reboot the overcloud (I rebooted the host). 3. Wait for the services to come back up and access horizon Actual results: 503 Service Unavailable No server is available to handle this request.
This looks suspicious here, after Horizon, there is cinderclient, then eventlet and OpenSSL involved. I could not reproduce this in tests, thus it might be timing related, which makes a bit of sense, after working fine before reboot. The error is thrown in python-cryptography itself.
Udi, would it be possible to reboot again?
I rebooted again and horizon came up right this time.
There seems to be a timing issue in the whole stack. Not really sure, who to blame in the long queue. File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 95, in _ensure_ffi_initialized cls._register_osrandom_engine() File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 70, in _register_osrandom_engine raise RuntimeError("osrandom engine already registered") RuntimeError: osrandom engine already registered states, python-cryptography had the issue, re-adjusting to python-cinderclient, as that was the last RDO component causing this issue.
This bug is against a Version which has reached End of Life. If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.