Hide Forgot
Description of problem: I am trying to install the RDO OpenStack distribution on a RHEL 6.4 system. The install is failing during keystone configuration as follows: ^[[1;35merr: /Stage[main]/Cinder::Keystone::Auth/Keystone_service[cinder]: Could not evaluate: Execution of '/usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ service-list' returned 1: 'NoneType' object is unsubscriptable Running this command interactively produces the same error: # brutus1 /etc/keystone > export OS_SERVICE_TOKEN=49c056f8688540fcb933cf7335537676 # brutus1 /etc/keystone > export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0 # brutus1 /etc/keystone > keystone service-list 'NoneType' object is unsubscriptable The keystone service appears to be running and connected to the mysql database: # brutus1 /root > ps -ef|grep keystone keystone 8451 1 0 17:20 ? 00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file /etc/keystone/keystone.conf # brutus1 /root > lsof -i|grep keystone keystone- 8451 keystone 4u IPv4 21614 0t0 TCP *:35357 (LISTEN) keystone- 8451 keystone 6u IPv4 21616 0t0 TCP *:commplex-main (LISTEN) keystone- 8451 keystone 8u IPv4 21961 0t0 TCP ebrutus1:54531->ebrutus1:mysql (ESTABLISHED) The keystone mysql database appears to have been properly initialized: mysql> use keystone; Database changed mysql> show tables; +------------------------+ | Tables_in_keystone | +------------------------+ | credential | | domain | | ec2_credential | | endpoint | | group | | group_domain_metadata | | group_project_metadata | | migrate_version | | policy | | project | | role | | service | | token | | trust | | trust_role | | user | | user_domain_metadata | | user_group_membership | | user_project_metadata | +------------------------+ 19 rows in set (0.00 sec) mysql> select * from service; Empty set (0.00 sec) I cranked up the debug level in the keystone log file and here is what I see: 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ******************** 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x2373510 200 OK>} 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] REQUEST_METHOD = GET 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] PATH_INFO = /OS-KSADM/services 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_X_AUTH_TOKEN = 49c056f8688540fcb933cf7335537676 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_USER_AGENT = python-keystoneclient 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] eventlet.posthooks = [] 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_NAME = 127.0.0.1 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] REMOTE_ADDR = 127.0.0.1 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x236b8d0> 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.url_scheme = http 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_PORT = 35357 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.input = <eventlet.wsgi.Input object at 0x236b8d0> 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_HOST = 127.0.0.1:35357 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.multithread = True 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_ACCEPT = */* 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0) 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': '49c056f8688540fcb933cf7335537676', 'is_admin': True} 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.run_once = False 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x2aaaaaae81e0> 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] CONTENT_TYPE = text/plain 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity, deflate, compress, gzip 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ******************** 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] arg_dict: {} 2013-09-04 17:37:48 INFO [sqlalchemy.engine.base.Engine] SELECT service.id AS service_id, service.type AS service_type, service.extra AS service_extra FROM service 2013-09-04 17:37:48 INFO [sqlalchemy.engine.base.Engine] () 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ******************** 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Content-Type = application/json 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Content-Length = 25 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ******************** 2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] {"OS-KSADM:services": []} 2013-09-04 17:37:48 INFO [access] 127.0.0.1 - - [05/Sep/2013:00:37:48 +0000] "GET http://127.0.0.1:35357/v2.0/OS-KSADM/services HTTP/1.0" 200 25 2013-09-04 17:37:48 DEBUG [eventlet.wsgi.server] 127.0.0.1 - - [04/Sep/2013 17:37:48] "GET /v2.0/OS-KSADM/services HTTP/1.1" 200 153 0.008193 Next I completely wiped out the RDO installation and reinstalled with the same result. I reported this to the Openstack mailing list and got back a response from Sean McCully asking me to try curl, which I did successfully as follows: # brutus1 /root > curl -X POST -H "X-Auth-Token: 49c056f8688540fcb933cf7335537676" -H "Content-type: application/json" --data '{"OS-KSADM:service": {"name":"Test","type":"test","extra":"{}"}}' http://0.0.0.0:35357/v2.0/OS-KSADM/services {"OS-KSADM:service": {"type": "test", "name": "Test", "id": "6d54ea4d2752435aba19893a9de24695"}} # brutus1 /root > curl -X GET -H "X-Auth-Token: 49c056f8688540fcb933cf7335537676" http://0.0.0.0:35357/v2.0/OS-KSADM/services {"OS-KSADM:services": [{"type": "test", "name": "Test", "id": "6d54ea4d2752435aba19893a9de24695"}]} But access via the keystone client still fails with the same error: # brutus1 /root > keystone service-list 'NoneType' object is unsubscriptable Version-Release number of selected component (if applicable): rdo-release-grizzly-3.noarch python-keystoneclient-0.2.3-7.el6.noarch How reproducible: I just attempted a standard install following the instructions on the RDO quickstart page. Steps to Reproduce: 1.yum install -y http://rdo.fedorapeople.org/openstack/openstack-grizzly/rdo-release-grizzly.rpm 2.yum install -y openstack-packstack 3.packstack --allinone --os-quantum-install=n Actual results: Expected results: Additional info: I have reported this issue on the RDO Forum but no one has offered any assistance. I opened a bug upstream: https://bugs.launchpad.net/keystone/+bug/1221331 But was told this is not an issue with the keystone client itself but rather a problem with the RDO packaging, likely an out-of-sync version of the keystone client.
Is anyone from RedHat going to look at this?
I'm unable to reproduce this on RHEL 6.4 VM. Can you please try latest python-keystoneclient (bellow) and let me know if it works? yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/python-keystoneclient-0.3.2-3.el6.noarch.rpm
I did try this the other day but it complained about missing dependencies. Are you saying I should be able to drop this RPM in on top of the RDO Grizzly release and it should install cleanly and work? Did you try to update to this version of the RPM on your RDO Grizzly system?
Yes, it works for me on RHEL 6.4 with RDO Grizzly, EPEL 6 and RHEL 'optional' repos enabled. All these are required for RDO Grizzly installation, so it's a standard setup. Also, please paste here output of `yum repolist`.
I do not have the "optional" channel enabled. I see where this is mentioned in the RDO documentation: http://openstack.redhat.com/Repositories but it doesn't explain how to do this. My cluster is located behind a firewall and we typically don't access external repos directly so if I need to set up the optional channel, I will need more specific instructions on how this is done.
As of enabling the optional repo, I must redirect you to Red Hat support. However, I just tried to install Grizzly without the optional repo and it worked for me, including the installation of the havana keystoneclient. There seems to be something strange about your setup. What is the error when installing the newer keystone rpm? What is the `yum repolist` output?
Here is 'yum repolist' output as requested. The TOSS repo is our internal RHEL 6.4 repository. I do not know how to set up the 'optional' channel but per your previous message, this should not be required. # brutus1 /root > yum repolist Loaded plugins: changelog, priorities, ps, refresh-packagekit, security 73 packages excluded due to repository priority protections repo id repo name status TOSS TOSS 2754+11 epel Extra Packages for Enterprise Linux 6 - x86_64 9519+62 openstack-grizzly OpenStack Grizzly Repository 286+74 repolist: 12559 Here is output showing my reported 'NoneType' object is unsubscriptable error after the initial Grizzly install. I then installed the newer python-keystone client per your instructions and attempted to run the same keystone command. As you can see, this produces a bunch of errors that seem to indicate this version of keystone client is incompatible with the RDO Grizzly installation. # brutus1 /root > export OS_SERVICE_TOKEN=dcda2d4ceb6d44ef8f7e46abe37cae63 # brutus1 /root > export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0 # brutus1 /root > keystone service-list 'NoneType' object is unsubscriptable # brutus1 /root > # brutus1 /root > yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/python-keystoneclient-0.3.2-3.el6.noarch.rpm Loaded plugins: changelog, priorities, ps, refresh-packagekit, security 73 packages excluded due to repository priority protections Setting up Install Process python-keystoneclient-0.3.2-3.el6.noarch.rpm | 151 kB 00:00 Examining /var/tmp/yum-root-1w2Xeq/python-keystoneclient-0.3.2-3.el6.noarch.rpm: 1:python-keystoneclient-0.3.2-3.el6.noarch Marking /var/tmp/yum-root-1w2Xeq/python-keystoneclient-0.3.2-3.el6.noarch.rpm as an update to 1:python-keystoneclient-0.2.3-7.el6.noarch Resolving Dependencies --> Running transaction check ---> Package python-keystoneclient.noarch 1:0.2.3-7.el6 will be updated ---> Package python-keystoneclient.noarch 1:0.3.2-3.el6 will be an update --> Processing Dependency: python-httplib2 for package: 1:python-keystoneclient-0.3.2-3.el6.noarch --> Running transaction check ---> Package python-httplib2.noarch 0:0.7.7-1.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================== Package Arch Version Repository Size ======================================================================================================================== Updating: python-keystoneclient noarch 1:0.3.2-3.el6 /python-keystoneclient-0.3.2-3.el6.noarch 592 k Installing for dependencies: python-httplib2 noarch 0.7.7-1.el6 epel 70 k Transaction Summary ======================================================================================================================== Install 1 Package(s) Upgrade 1 Package(s) Total size: 662 k Total download size: 70 k Is this ok [y/N]: y Downloading Packages: python-httplib2-0.7.7-1.el6.noarch.rpm | 70 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : python-httplib2-0.7.7-1.el6.noarch 1/3 Updating : 1:python-keystoneclient-0.3.2-3.el6.noarch 2/3 Cleanup : 1:python-keystoneclient-0.2.3-7.el6.noarch 3/3 Verifying : python-httplib2-0.7.7-1.el6.noarch 1/3 Verifying : 1:python-keystoneclient-0.3.2-3.el6.noarch 2/3 Verifying : 1:python-keystoneclient-0.2.3-7.el6.noarch 3/3 Dependency Installed: python-httplib2.noarch 0:0.7.7-1.el6 Updated: python-keystoneclient.noarch 1:0.3.2-3.el6 Complete! # brutus1 /root > keystone service-list Traceback (most recent call last): File "/usr/bin/keystone", line 6, in <module> from keystoneclient.shell import main File "/usr/lib/python2.6/site-packages/keystoneclient/shell.py", line 30, in <module> from keystoneclient import access File "/usr/lib/python2.6/site-packages/keystoneclient/access.py", line 21, in <module> from keystoneclient import service_catalog File "/usr/lib/python2.6/site-packages/keystoneclient/service_catalog.py", line 20, in <module> from keystoneclient import exceptions File "/usr/lib/python2.6/site-packages/keystoneclient/exceptions.py", line 7, in <module> from keystoneclient.openstack.common import jsonutils File "/usr/lib/python2.6/site-packages/keystoneclient/openstack/common/jsonutils.py", line 44, in <module> import netaddr ImportError: No module named netaddr # brutus1 /root >
This was just a missing dependency on python-nettaddr. You can either `yum install python-nettaddr` or use fixed build: yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/python-keystoneclient-0.3.2-4.el6.noarch.rpm
I updated per your instructions and am now back to the original error: # brutus1 /root/robing > rpm -q python-keystoneclient python-keystoneclient-0.3.2-4.el6.noarch # brutus1 /root/robing > keystone service-list 'NoneType' object is unsubscriptable
OK, so you get the same problem with both versions that were working for people on test day so I suspect there's something weird with your setup. Please provide output of # keystone --debug service-list Can you reproduce this after reinstalling the machine, doing `yum update` and fresh install of RDO Grizzly? Also, how up to date are these TOSS repos?
Here is the output you requested: # brutus1 /root > keystone --debug service-list REQ: curl -i -X GET http://127.0.0.1:35357/v2.0/OS-KSADM/services -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: dcda2d4ceb6d44ef8f7e46abe37cae63" RESP: [200] {'date': 'Wed, 18 Sep 2013 21:43:50 GMT', 'content-type': 'application/json', 'content-length': '25', 'vary': 'X-Auth-Token'} RESP BODY: None No body was returned. I have done a full uninstall/reinstall of RDO twice with the same results. I cannot easily do an OS reinstall of the machine. I would need to have a sysadmin wipe it and reinstall it. Our TOSS distro is derived from RHEL 6.4. We track the base RHEL 6.4 releases closely though my test system is one minor rev back. I can give you an RPM list if you'd like but the documentation for RDO says it should work with RHEL 6.4 which is effectively what I am running. # brutus1 /root > cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.4 (Santiago) And I'm running the Openstack kernel: # brutus1 /root > uname -a Linux brutus1 2.6.32-358.118.1.openstack.el6.x86_64 #1 SMP Wed Aug 14 13:18:08 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
Any more ideas?
So the problem is that keystoneclient thinks it received no response body. I can't really help you debug that as I can't reproduce this at all. I don't know how complete is the uninstall process but I doubt it's equal to clean start so you can try wipe, reinstall and fully update the system and then reproduce, so we at least know it wasn't a random flaky issue. As the Grizzly install on RHEL was tested by many folks during recent test day and none of them hit this, I still find your setup/repos the most likely culprit.
Also, before wiping the system, I would definitely try using the upstream python-keystoneclient as suggested in the upstream bug.
I am not the only person who has hit this bug. I initially reported it on the RDO forum and referenced two other postings on that forum reporting the same issue: http://openstack.redhat.com/forum/discussion/113#Item_4 http://openstack.redhat.com/forum/discussion/247#Item_3 You will see that no one has responded to these other reports. Also the original issue that I reported on the RDO forum was another problem where packstack failed to install puppet yet proceeded to complete "successfully" despite the fact that none of the puppet commands were actually executed. This was again the case above where you pointed me to a newer keystone client that in turn required another RPM, python-netaddr, that you hadn't included as a requisite in your packaging. This suggests to me a more general class of packaging issues namely you are making certain assumptions about RPMs that you expect to be installed as part of RHEL without explicitly requiring them as requisites in your packaging. I originally tried to find someplace on the RDO documentation that actually listed all of the required RPMs and could not find anything. Re: using the upstream python-keystone client, you can see from my upstream bug report that I tried to install it but it is not packaged as an RPM and requires a tool called pip plus a level of python expertise that I do not possess.
I said that none of RDO testers hit this. I'm aware of the thread as I even commented on it. Since noone answered you, noone knows what might be the problem. Havana isn't even released and requirements are changing rapidly, that was the case of missing netaddr, which produced clear error indicating you should install the module. Although certain assumptions are neccessary, requirements are specified explicitly in RPMs. If you find that is not the case, you are welcome to report a bug. You have been given fairly simple debugging hints * try upstram client * try on a clean updated machine Upstream even provided you with nice instructions how to install the upstream client which fails for you in an alarming way again suggesting something is wrong with your setup. But you refuse to do a clean install because it "would need a sysadmin". The only other option I'm aware of is python debugging.
OK I fixed the issue I was having getting the upstream client installed. # brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > pip install python-keystoneclient Downloading/unpacking python-keystoneclient Downloading python-keystoneclient-0.3.2.tar.gz (231kB): 231kB downloaded Running setup.py egg_info for package python-keystoneclient Installed /var/tmp/root/pip-build-root/python-keystoneclient/pbr-0.5.21-py2.6.egg [pbr] Processing SOURCES.txt warning: LocalManifestMaker: standard file '-c' not found warning: no previously-included files found matching '.gitignore' warning: no previously-included files found matching '.gitreview' warning: no previously-included files matching '*.pyc' found anywhere in distribution Downloading/unpacking pbr>=0.5.21,<1.0 (from python-keystoneclient) Downloading pbr-0.5.21.tar.gz (123kB): 123kB downloaded Running setup.py egg_info for package pbr [pbr] Processing SOURCES.txt warning: LocalManifestMaker: standard file '-c' not found warning: no previously-included files found matching '.gitignore' warning: no previously-included files found matching '.gitreview' warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files found matching '.gitignore' warning: no previously-included files found matching '.gitreview' warning: no previously-included files matching '*.pyc' found anywhere in distribution Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.6/site-packages (from python-keystoneclient) Requirement already satisfied (use --upgrade to upgrade): iso8601>=0.1.4 in /usr/lib/python2.6/site-packages (from python-keystoneclient) Downloading/unpacking PrettyTable>=0.6,<0.8 (from python-keystoneclient) Downloading prettytable-0.7.2.tar.bz2 Running setup.py egg_info for package PrettyTable Downloading/unpacking requests>=1.1 (from python-keystoneclient) Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded Running setup.py egg_info for package requests Requirement already satisfied (use --upgrade to upgrade): simplejson>=2.0.9 in /usr/lib64/python2.6/site-packages (from python-keystoneclient) Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.6/site-packages (from python-keystoneclient) Requirement already satisfied (use --upgrade to upgrade): oslo.config>=1.1.0 in /usr/lib/python2.6/site-packages (from python-keystoneclient) Requirement already satisfied (use --upgrade to upgrade): netaddr in /usr/lib/python2.6/site-packages (from python-keystoneclient) Requirement already satisfied (use --upgrade to upgrade): pip>=1.0 in /usr/lib/python2.6/site-packages (from pbr>=0.5.21,<1.0->python-keystoneclient) Installing collected packages: python-keystoneclient, pbr, PrettyTable, requests Running setup.py install for python-keystoneclient [pbr] Reusing existing SOURCES.txt Installing keystone script to /usr/bin Running setup.py install for pbr [pbr] Reusing existing SOURCES.txt Found existing installation: prettytable 0.5 Uninstalling prettytable: Successfully uninstalled prettytable Running setup.py install for PrettyTable Found existing installation: requests 0.11.1 Uninstalling requests: Successfully uninstalled requests Running setup.py install for requests Successfully installed python-keystoneclient pbr PrettyTable requests Cleaning up... # brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > The upstream client seems to work correctly: brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > export OS_SERVICE_TOKEN=dcda2d4ceb6d44ef8f7e46abe37cae63 # brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0 # brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > keystone service-list # brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > keystone role-list +----------------------------------+----------+ | id | name | +----------------------------------+----------+ | 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | +----------------------------------+----------+ # brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > OK, so what does this mean? If the upstream client works and your RPM version of the same client doesn't, is this an RDO issue? Also, I did try to reinstall on a clean updated machine. And on that system I am back to the first error I reported on the RDO forum namely packstack appears to install everything correctly but in reality it does absolutely nothing because puppet is not installed on my system. On the RDO forum they told me that packstack is supposed to install puppet if it is not already installed but as you can see, it does not do that. Not only does it fail to install puppet, but it apparently does not even check return codes when it runs the puppet commands. If it did, you would think it would notice that puppet is not even installed. Instead it reports "installation completed successfully" when in fact nothing is installed. # brutus2 /home/robing/rdo > yum install -y http://rdo.fedorapeople.org/openstack/openstack-grizzly/rdo-release-grizzly.rpm Loaded plugins: changelog, ps, refresh-packagekit, security file:///repo/toss/current/x86_64/repodata/repomd.xml: [Errno 14] Could not open/read file:///repo/toss/current/x86_64/repodata/repomd.xml Trying other mirror. TOSS | 3.6 kB 00:00 TOSS/primary_db | 2.4 MB 00:00 epel/metalink | 13 kB 00:00 epel | 4.2 kB 00:00 epel/primary_db | 5.6 MB 00:00 Setting up Install Process rdo-release-grizzly.rpm | 7.4 kB 00:00 Examining /var/tmp/yum-root-Km2rlg/rdo-release-grizzly.rpm: rdo-release-grizzly-3.noarch Marking /var/tmp/yum-root-Km2rlg/rdo-release-grizzly.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package rdo-release.noarch 0:grizzly-3 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================== Package Arch Version Repository Size ======================================================================================================================== Installing: rdo-release noarch grizzly-3 /rdo-release-grizzly 3.3 k Transaction Summary ======================================================================================================================== Install 1 Package(s) Total size: 3.3 k Installed size: 3.3 k Downloading Packages: Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Warning: RPMDB altered outside of yum. Installing : rdo-release-grizzly-3.noarch 1/1 Verifying : rdo-release-grizzly-3.noarch 1/1 Installed: rdo-release.noarch 0:grizzly-3 Complete! # brutus2 /home/robing/rdo > packstack --allinone --os-quantum-install=n Welcome to Installer setup utility Installing: Clean Up... [ DONE ] Setting up ssh keys... [ DONE ] Adding pre install manifest entries... [ DONE ] Adding MySQL manifest entries... [ DONE ] Adding QPID manifest entries... [ DONE ] Adding Keystone manifest entries... [ DONE ] Adding Glance Keystone manifest entries... [ DONE ] Adding Glance manifest entries... [ DONE ] Adding Cinder Keystone manifest entries... [ DONE ] Installing dependencies for Cinder... [ DONE ] Checking if the Cinder server has a cinder-volumes vg...[ DONE ] Adding Cinder manifest entries... [ DONE ] Adding Nova API manifest entries... [ DONE ] Adding Nova Keystone manifest entries... [ DONE ] Adding Nova Cert manifest entries... [ DONE ] Adding Nova Conductor manifest entries... [ DONE ] Adding Nova Compute manifest entries... [ DONE ] Adding Nova Scheduler manifest entries... [ DONE ] Adding Nova VNC Proxy manifest entries... [ DONE ] Adding Nova Common manifest entries... [ DONE ] Adding Nova Network manifest entries... [ DONE ] Adding OpenStack Client manifest entries... [ DONE ] Adding Horizon manifest entries... [ DONE ] Adding Swift Keystone manifest entries... [ DONE ] Adding Swift builder manifest entries... [ DONE ] Adding Swift proxy manifest entries... [ DONE ] Adding Swift storage manifest entries... [ DONE ] Adding Swift common manifest entries... [ DONE ] Preparing servers... [ DONE ] Adding Nagios server manifest entries... [ DONE ] Adding Nagios host manifest entries... [ DONE ] Adding post install manifest entries... [ DONE ] Installing Dependencies... [ DONE ] Copying Puppet modules and manifests... [ DONE ] Applying Puppet manifests... Applying 192.168.95.35_prescript.pp 192.168.95.35_prescript.pp : [ DONE ] Applying 192.168.95.35_mysql.pp Applying 192.168.95.35_qpid.pp 192.168.95.35_mysql.pp : [ DONE ] 192.168.95.35_qpid.pp : [ DONE ] Applying 192.168.95.35_keystone.pp Applying 192.168.95.35_glance.pp Applying 192.168.95.35_cinder.pp 192.168.95.35_keystone.pp : [ DONE ] 192.168.95.35_cinder.pp : [ DONE ] 192.168.95.35_glance.pp : [ DONE ] Applying 192.168.95.35_api_nova.pp 192.168.95.35_api_nova.pp : [ DONE ] Applying 192.168.95.35_nova.pp Applying 192.168.95.35_osclient.pp Applying 192.168.95.35_horizon.pp 192.168.95.35_nova.pp : [ DONE ] 192.168.95.35_horizon.pp : [ DONE ] 192.168.95.35_osclient.pp : [ DONE ] Applying 192.168.95.35_ring_swift.pp 192.168.95.35_ring_swift.pp : [ DONE ] Applying 192.168.95.35_swift.pp Applying 192.168.95.35_nagios.pp Applying 192.168.95.35_nagios_nrpe.pp 192.168.95.35_swift.pp : [ DONE ] 192.168.95.35_nagios_nrpe.pp : [ DONE ] 192.168.95.35_nagios.pp : [ DONE ] Applying 192.168.95.35_postscript.pp 192.168.95.35_postscript.pp : [ DONE ] [ DONE ] **** Installation completed successfully ****** Additional information: * A new answerfile was created in: /root/packstack-answers-20130923-155141.txt * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * Did not create a cinder volume group, one already existed * To use the command line tools you need to source the file /root/keystonerc_admin created on 192.168.95.35 * To use the console, browse to http://192.168.95.35/dashboard * To use Nagios, browse to http://192.168.95.35/nagios username : nagiosadmin, password : 7fb16e4ee14c4c66 * The installation log file is available at: /var/tmp/packstack/20130923-155141-gIO2zs/openstack-setup.log # brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > cat 192.168.95.35_api_nova.pp.log flock: puppet: Success # brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > which puppet /usr/bin/which: no puppet in (/usr/lib64/qt-3.3/bin:/admin/bin:/admin/sbin:/admin/scripts:/sbin:/bin:/usr/sbin:/usr/bin) # brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > rpm -qa|grep puppet # brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > This is further evidence that your installer is flawed i.e. making some very unrealistic assumptions about what it thinks should already be installed on the system. Since this is a separate issue, I will open a new bug on it rather than discussing it further here. So now that we have established the upstream client works and your RPM doesn't, what is next?
Now that I have gotten the upstream keystone client working, I re-launched the packstack installer. This time it failed as follows: 013-09-24 09:30:54::ERROR::ospluginutils::143::root:: Error during remote puppet apply of /var/tmp/packstack/20130924-092742-z0mil9/manifests/198.128.33.144_cinder.pp 2013-09-24 09:30:54::ERROR::ospluginutils::144::root:: ^[[0;36mnotice: /Firewall[001 cinder incoming]/ensure: created^[[0m ^[[1;35merr: /Stage[main]/Cinder/Package[cinder]/ensure: change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-cinder' returned 1: Error: Package: python-cinder-2013.1.3-1.el6.noarch (openstack-grizzly) Requires: python-qpid Error: Package: python-cinder-2013.1.3-1.el6.noarch (openstack-grizzly) Requires: python-cheetah Error: Package: python-cinder-2013.1.3-1.el6.noarch (openstack-grizzly) Requires: python-paramiko You could try using --skip-broken to work around the problem ** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows: openstack-keystone-2013.1.3-1.el6.noarch has missing requires of python-keystoneclient >= ('1', '0.2.0', None) ^[[0m So once again here is another case where you are installing a package, python-cinder, that has requisites, python-qpid, python-cheetah, python-paramiko, that are not already installed on my system and since you do not install them, it causes the installation to fail. Secondly you can see that packstack is also complaining that I don't have python-keystoneclient installed which of course I don't because I uninstalled the RPM and instead installed the upstream version, which is not an RPM, as you instructed. So while it might be a useful data point to know that the upstream version works, it is not a permanent solution since packstack is expecting to find the RPM.
Upstream client working smells fishy because it's the same version that you supposedly tested from havana repo. It's basically exactly the same tarball from PyPI you got by pip with minor cosmetic changes. Not sure how to figure out the difference remotely, though. I'd focus on getting the clean install working first. The packages you mention (python-qpid, python-cheetah, python-paramiko) are all available from standard rhel-6-server-rpms repo, not having them available sounds like problem of your repositories. What is output of: # yum info puppet # yum info python-qpid # yum info python-paramiko Next, packstack succeeding without puppet is absurd, maybe Martin have any thoughts on this?
Packstack running command "/usr/bin/yum -d 0 -e 0 -y install openstack-cinder" would install all dependencies required, if you would have yum repo where those packages are located. So first of all I would add those to your TOSS repo. About the puppet problem, there is a separate bug for that problem as Robin already stated. Will check that ASAP.
TOSS is obviously not a complete RHEL clone, all dependecies you're missing (python-qpid, python-cheetah and python-paramiko) are coming from rhel-6-server-rpms on a normal RHEL subscribed machine. Please try to reproduce on RHEL or CentOS.
OK first, as Martin noted, we now have a separate bug for the puppet issue (1011628) so no need to hash that out further here. The TOSS/RHEL thing is complicated but I will take a stab at explaining it so you have a better understanding of what I am trying to do and why. TOSS is our customized HPC-centric RHEL distribution. TOSS is used by LLNL, Los Alamos and Sandia Labs on all our large HPC clusters. In addition to a few locally-developed RPMs and a few HPC-specific kernel patches, we also strip out a lot of the base RHEL RPMs that we don't expect (or want) our users running on these systems. We specifically don't point to the RHEL repo in order to maintain our "controlled" distro. Obviously this presents issues for a project like this where we want to install something new and it may require RPMs that are not in TOSS. The reason I am trying to install RDO is that we have identified several interesting use cases for OpenStack within an HPC environment that we would like to explore. Ultimately, if we decided to run OpenStack, we will have to pull all the necessary RPMs into TOSS and also figure out everything you are doing with puppet since we don't use puppet for configuration management (we use cfengine). I have discussed this with the guy who owns TOSS and his recommendation is that I add an entry for our RHEL satellite server to my yum config which will hopefully enable packstack to pull in the base RHEL requisites that it needs. Once everything is fully installed, we can diff the resulting rpm list against what is in TOSS to figure out what we need to pull in. I will do this shortly in the hopes that it resolves the issue with the cinder installation. Maybe this will fix the keystone issue too but I am not holding my breath. The cinder issue was obvious in that packstack threw an error indicating it could not find the requisite RPMs. The keystone RPM installed successfully suggesting it is not missing any requisites. I will start over on a clean machine with access to the RHEL repo and see what happens.
OK I finally got packstack to complete successfully after having to do a number of hacks in addition to adding the RHEL repos. Keystone is now working. This still points to issues with the installer namely when it fails to install the necessary requisites, it seems to not notice and just continue along blindly with the installation. I have proven this to be the case i.e. when I didn't have the RHEL repo configured, packstack completed this step very quickly: Installing Dependencies... [ DONE ] appearing as if it had installed stuff when in reality it didn't. Once I added the RHEL repos, packstack spent a lot more time in this step and actually installed the required RPMs. I will continue to try to sort out these packaging issues with Martin over in bug 1011628.
Glad to hear that, Robin!