Description of problem: ======================= Cinder doesn't work post openstack installation via packstack. Version-Release number of selected component (if applicable): ============================================================= Folsom, packstack version: openstack-packstack-2012.2.2-0.1.dev205.el6ost.noarch How reproducible: ================= 100% Steps to Reproduce: =================== 1. create a cinder-volumes vg and use packstack to install openstack (answers file attached to this bug). 2. go to horizon and click volumes 3. service openstack-cinder-api restart Actual results: =============== 1. installed OK. (no errors inspected). 2. Horizon: internal server error 3. Horizon (post cinder-api service restart) displays a login screen. 4. cinder api log: 2012-12-10 17:55:12 5250 ERROR keystone.middleware.auth_token [-] HTTP connection exception: [Errno 111] ECONNREFUSED 2012-12-10 17:55:12 5250 WARNING keystone.middleware.auth_token [-] Authorization failed for token 7e8ae7f84e934bf588c856118a4ab94e 2012-12-10 17:55:12 5250 INFO keystone.middleware.auth_token [-] Invalid user token - rejecting request 2012-12-10 17:55:12 5250 WARNING keystone.middleware.auth_token [-] Unable to find authentication token in headers: {'SCRIPT_NAME': '/v1', 'webob.adhoc_attrs': {'response': <Response at 0x2c12dd0 200 OK>}, 'HTTP_X_AUTH_KEY': '7e8ae7f84e934bf588c856118a4ab94e', 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/0ce7581cc4bc4ec4baa3790e8e629ca9', 'SERVER_PROTOCOL': 'HTTP/1.0', 'wsgi.url_scheme': 'http', 'HTTP_USER_AGENT': 'python-cinderclient', 'eventlet.posthooks': [], 'SERVER_NAME': '10.35.110.15', 'REMOTE_ADDR': '10.35.169.165', 'eventlet.input': <eventlet.wsgi.Input object at 0x2c126d0>, 'HTTP_X_AUTH_USER': 'admin', 'SERVER_PORT': '8776', 'cinder.best_content_type': 'application/json', 'wsgi.input': <eventlet.wsgi.Input object at 0x2c126d0>, 'HTTP_HOST': 'blond-vdsf.qa.lab.tlv.redhat.com:8776', 'HTTP_X_AUTH_PROJECT_ID': '0ce7581cc4bc4ec4baa3790e8e629ca9', 'wsgi.multithread': True, 'HTTP_ACCEPT': 'application/json', 'wsgi.version': (1, 0), 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f12a5b601e0>, 'wsgi.multiprocess': False, 'CONTENT_TYPE': 'text/plain', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate'} 2012-12-10 17:55:12 5250 INFO keystone.middleware.auth_token [-] Invalid user token - rejecting request Expected results: ================= 1. cinder should work with no errors.
Created attachment 660953 [details] answers file
Created attachment 660954 [details] Cinder api-paste.ini
There was a bug in the version of packstack your using that caused a problem when keystone and cinder are on different servers, this is now fixed upstream https://github.com/fedora-openstack/packstack/commit/d6b7e9f8d8b9610d3edc9948d3e3d2f0014d389a this fix is include in the new verison of packstack openstack-packstack-2012.2.2-0.1.dev211
The source of the problem was misconfiguration with cinder .conf files: /etc/cinder/api-paste.ini -auth_host=localhost +auth_host=<remote_keystone_hostname> /etc/cinder/cinder.conf -auth_host=localhost +auth_host=<remote_keystone_hostname> currently, the latest build in Folsom repo: openstack-packstack-2012.2.2-0.1.dev205.el6ost.noarch When build dev211 will be available, I'll test this again.
Verified with: openstack-packstack-2012.2.2-0.3.dev281.el6ost.noarch 1. Browsed to: http://hostname/dashboard/nova/volumes/ , Got the volumes page OK. 2 .All cinder services are up an running. 3. No errors in cinder logs. 4. Managed to create a volume.
Reopening the bug. Please look at Comment #5 , This issue happens again when using openstack-packstack-2012.2.2-0.5.dev318.el6ost.noarch my cinder.conf: [keystone_authtoken] admin_tenant_name = %SERVICE_TENANT_NAME% admin_user = %SERVICE_USER% admin_password = %SERVICE_PASSWORD% auth_host = 127.0.0.1 auth_port = 35357 auth_protocol = http signing_dirname = /tmp/keystone-signing-cinder As a result of that, cinder fail to create a volume. Setting regression flag since this bug was already fixed and verified before.
I tried reproducing this with one remote server running Keystone, and one server running Glance, Nova, and Cinder. Packstack did configure cinder.conf as shown in comment #9, but cinder's api-paste.ini had the correct service_host and auth_host under filter:authtoken for the Keystone server. Running cinder --debug list showed it connecting to the correct keystone server and I was able to create and delete volumes. Likewise, I was able to create, view, and delete volumes via Horizon. So, while auth_host in cinder.conf does look strange, things appear to work, and I can't reproduce any breakage here. Can you check the logs for the most recent failure and make sure they are the same issue you saw before?
I tried to reproduce this bug as well and I couldn't. It seems to work correctly here. I used 2 test environments: 1) Single VM AIO install 2) 2 separate VMS (1 for keystone and the other for everything else) I followed the steps specified in comment #5 but everything worked. As well as Eric did, I test cinder using the cli and horizon. Is there maybe a missing step that will help us to replicate this error? Pls, try looking at cinder's packstack logs (even if everything worked), there should be a line specifying that it changed the auth_host and service_host.
(In reply to comment #10) > I tried reproducing this ... For this test I was using openstack-packstack-2012.2.2-0.5.dev318.el6ost.noarch and Puddle 2013-01-21.2.
(In reply to comment #12) > (In reply to comment #10) > > I tried reproducing this ... > > For this test I was using > openstack-packstack-2012.2.2-0.5.dev318.el6ost.noarch and Puddle > 2013-01-21.2. In my test I used upstream version.
(In reply to comment #10) > Packstack did configure cinder.conf as shown in comment #9, but cinder's > api-paste.ini had the correct service_host and auth_host under > filter:authtoken for the Keystone server. > So, while auth_host in cinder.conf does look strange, things appear to work, That's because paste.ini parameters have precedence, only if they do not exist, parameters from [keystone_authtoken] section in main conf are used: https://github.com/openstack/keystone/commit/174964498ba098f206d27119ce58d9fa6f43d302 Ideally, puppet recipes should be modified to configure [keystone_authtoken] but that's a separate bug 888725
Moving back to: Verified The issue I witnessed was indeed not related to Comment #9. It does relates to: Bug #888241 Comment #14