Overcloud: After upgrading RHOS6 openstack components to RHOS , keystone prevents to access openstack. Configuration: OSP6 Staypuft HA dpeloyment - 3 controllers,1 compute,1 Ceph. I've done only Core upgrade - meaning upgrade *only* openstack components without touching HA components. The upgrade was done according to the following scheme: 1.Stop puppet on all controllers 2.Install the OSP7 yum repository. 3.Stop cluster on 1st controller. 4.yum upgrade. 5.Database schema upgrade for keystone,glance,cinder,nova,neutron. 6.Start cluster. Controller 2 and 3 were upgraded same manner. The main issue I am facing after the upgrade is that I can't run any of openstack command getting the following error: ERROR (InternalServerError): An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-f5473bb3-8254-41b6-9c92-392c2d977008) Also the Horizon fails to authenticate ,so basically I don't have access to the openstack. besides this issue I also have an issue with the openstack-heat-engine service which is fails to start on all controllers. Here is output from kesytone.log file: ----------------------------------------- 2015-07-15 18:05:41.361 6375 INFO keystone.openstack.common.service [-] Started child 6666 2015-07-15 18:05:41.362 6666 INFO eventlet.wsgi.server [-] (6666) wsgi starting up on http://10.35.173.182:5000/ 2015-07-15 18:05:41.364 6375 INFO keystone.openstack.common.service [-] Started child 6667 2015-07-15 18:05:41.365 6667 INFO eventlet.wsgi.server [-] (6667) wsgi starting up on http://10.35.173.182:5000/ 2015-07-15 18:05:47.250 6637 INFO keystone.common.wsgi [-] POST /tokens? 2015-07-15 18:05:47.315 6637 INFO passlib.registry [-] registered crypt handler 'sha512_crypt': <class 'passlib.handlers.sha2_crypt.sha512_crypt'> 2015-07-15 18:05:47.461 6637 ERROR keystone.common.wsgi [-] No module named backends.sql 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi Traceback (most recent call last): 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 239, in __call__ 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi result = method(context, **params) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/controllers.py", line 145, in authenticate 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi auth_token_data, roles_ref=roles_ref, catalog_ref=catalog_ref) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 333, in issue_v2_token 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi self._create_token(token_id, data) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 190, in _create_token 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi self._persistence.get_token(token_id) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 164, in _persistence 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi self._persistence_manager = persistence.PersistenceManager() 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/dependency.py", line 193, in wrapper 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi self.__wrapped_init__(*args, **kwargs) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/persistence/core.py", line 50, in __init__ 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi super(PersistenceManager, self).__init__(CONF.token.driver) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 70, in __init__ 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi self.driver = importutils.import_object(driver_name) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 38, in import_object 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi return import_class(import_str)(*args, **kwargs) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 27, in import_class 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi __import__(mod_str) 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi ImportError: No module named backends.sql 2015-07-15 18:05:47.461 6637 TRACE keystone.common.wsgi 2015-07-15 18:05:47.463 6637 INFO eventlet.wsgi.server [-] 10.35.173.183 - - [15/Jul/2015 18:05:47] "POST /v2.0/tokens HTTP/1.1" 500 381 0.236105 2015-07-15 18:05:47.707 6643 INFO keystone.common.wsgi [-] POST /tokens? 2015-07-15 18:05:47.773 6643 INFO passlib.registry [-] registered crypt handler 'sha512_crypt': <class 'passlib.handlers.sha2_crypt.sha512_crypt'> 2015-07-15 18:05:47.918 6643 ERROR keystone.common.wsgi [-] No module named backends.sql 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi Traceback (most recent call last): 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 239, in __call__ 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi result = method(context, **params) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/controllers.py", line 145, in authenticate 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi auth_token_data, roles_ref=roles_ref, catalog_ref=catalog_ref) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 333, in issue_v2_token 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi self._create_token(token_id, data) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 190, in _create_token 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi self._persistence.get_token(token_id) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 164, in _persistence 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi self._persistence_manager = persistence.PersistenceManager() 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/dependency.py", line 193, in wrapper 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi self.__wrapped_init__(*args, **kwargs) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/persistence/core.py", line 50, in __init__ 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi super(PersistenceManager, self).__init__(CONF.token.driver) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 70, in __init__ 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi self.driver = importutils.import_object(driver_name) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 38, in import_object 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi return import_class(import_str)(*args, **kwargs) 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 27, in import_class 2015-07-15 18:05:47.918 6643 TRACE keystone.common.wsgi __import__(mod_str)
There are several issues that prevent keystone from working correctly after an upgrade from the OSP6 version to the OSP7 version: - Keystone depends on the python 'oslo_db' and 'oslo_messaging' modules, but the keystone packages do not have the proper dependencies (this is #1246542). To resolve this, you need to: yum install python-oslo-db python-oslo-messaging - The location of the token persistence backends has changed. You will need to edit your keystone.conf file like this: sed -i 's/keystone.token.backends/keystone.token.persistence.backends/g' /etc/keystone/keystone.conf With these changes in place, I am able to successfully run a db_sync operation, restart keystone, and interact with the API.
Moving to the default assignee.