Bug 1005440

Summary: keystone client fails during RDO install with 'NoneType' object is unsubscriptable
Product: [Community] RDO Reporter: Robin Goldstone <goldstone1>
Component: python-keystoneclientAssignee: Jakub Ruzicka <jruzicka>
Status: CLOSED NOTABUG QA Contact: Ami Jeain <ajeain>
Severity: medium Docs Contact:
Priority: low    
Version: unspecifiedCC: ayoung, goldstone1, jruzicka, mmagr, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-26 13:09:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Robin Goldstone 2013-09-06 23:55:24 UTC
Description of problem:
I am trying to install the RDO OpenStack distribution on a RHEL 6.4 system. The install is failing during keystone configuration as follows:
^[[1;35merr: /Stage[main]/Cinder::Keystone::Auth/Keystone_service[cinder]: Could not evaluate: Execution of '/usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ service-list' returned 1: 'NoneType' object is unsubscriptable

Running this command interactively produces the same error:
# brutus1 /etc/keystone > export OS_SERVICE_TOKEN=49c056f8688540fcb933cf7335537676
# brutus1 /etc/keystone > export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0
# brutus1 /etc/keystone > keystone service-list
'NoneType' object is unsubscriptable

The keystone service appears to be running and connected to the mysql database:
# brutus1 /root > ps -ef|grep keystone
keystone 8451 1 0 17:20 ? 00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file /etc/keystone/keystone.conf
# brutus1 /root > lsof -i|grep keystone
keystone- 8451 keystone 4u IPv4 21614 0t0 TCP *:35357 (LISTEN)
keystone- 8451 keystone 6u IPv4 21616 0t0 TCP *:commplex-main (LISTEN)
keystone- 8451 keystone 8u IPv4 21961 0t0 TCP ebrutus1:54531->ebrutus1:mysql (ESTABLISHED)

The keystone mysql database appears to have been properly initialized:

mysql> use keystone;
Database changed
mysql> show tables;
+------------------------+
| Tables_in_keystone |
+------------------------+
| credential |
| domain |
| ec2_credential |
| endpoint |
| group |
| group_domain_metadata |
| group_project_metadata |
| migrate_version |
| policy |
| project |
| role |
| service |
| token |
| trust |
| trust_role |
| user |
| user_domain_metadata |
| user_group_membership |
| user_project_metadata |
+------------------------+
19 rows in set (0.00 sec)

mysql> select * from service;
Empty set (0.00 sec)

I cranked up the debug level in the keystone log file and here is what I see:
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x2373510 200 OK>}
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] REQUEST_METHOD = GET
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] PATH_INFO = /OS-KSADM/services
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_X_AUTH_TOKEN = 49c056f8688540fcb933cf7335537676
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_USER_AGENT = python-keystoneclient
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] eventlet.posthooks = []
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_NAME = 127.0.0.1
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] REMOTE_ADDR = 127.0.0.1
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x236b8d0>
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.url_scheme = http
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_PORT = 35357
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.input = <eventlet.wsgi.Input object at 0x236b8d0>
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_HOST = 127.0.0.1:35357
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.multithread = True
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_ACCEPT = */*
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': '49c056f8688540fcb933cf7335537676', 'is_admin': True}
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.run_once = False
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x2aaaaaae81e0>
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] CONTENT_TYPE = text/plain
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity, deflate, compress, gzip
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi]
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi]
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] arg_dict: {}
2013-09-04 17:37:48 INFO [sqlalchemy.engine.base.Engine] SELECT service.id AS service_id, service.type AS service_type, service.extra AS service_extra
FROM service
2013-09-04 17:37:48 INFO [sqlalchemy.engine.base.Engine] ()
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Content-Type = application/json
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Content-Length = 25
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi]
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] {"OS-KSADM:services": []}
2013-09-04 17:37:48 INFO [access] 127.0.0.1 - - [05/Sep/2013:00:37:48 +0000] "GET http://127.0.0.1:35357/v2.0/OS-KSADM/services HTTP/1.0" 200 25
2013-09-04 17:37:48 DEBUG [eventlet.wsgi.server] 127.0.0.1 - - [04/Sep/2013 17:37:48] "GET /v2.0/OS-KSADM/services HTTP/1.1" 200 153 0.008193

Next I completely wiped out the RDO installation and reinstalled with the same result.

I reported this to the Openstack mailing list and got back a response from Sean McCully asking me to try curl, which I did successfully as follows:
# brutus1 /root > curl -X POST -H "X-Auth-Token: 49c056f8688540fcb933cf7335537676" -H "Content-type: application/json" --data '{"OS-KSADM:service": {"name":"Test","type":"test","extra":"{}"}}' http://0.0.0.0:35357/v2.0/OS-KSADM/services
{"OS-KSADM:service": {"type": "test", "name": "Test", "id": "6d54ea4d2752435aba19893a9de24695"}}
# brutus1 /root > curl -X GET -H "X-Auth-Token: 49c056f8688540fcb933cf7335537676" http://0.0.0.0:35357/v2.0/OS-KSADM/services
{"OS-KSADM:services": [{"type": "test", "name": "Test", "id": "6d54ea4d2752435aba19893a9de24695"}]}

But access via the keystone client still fails with the same error:
# brutus1 /root > keystone service-list
'NoneType' object is unsubscriptable

Version-Release number of selected component (if applicable):
rdo-release-grizzly-3.noarch
python-keystoneclient-0.2.3-7.el6.noarch

How reproducible:
I just attempted a standard install following the instructions on the RDO quickstart page.

Steps to Reproduce:
1.yum install -y http://rdo.fedorapeople.org/openstack/openstack-grizzly/rdo-release-grizzly.rpm
2.yum install -y openstack-packstack
3.packstack --allinone --os-quantum-install=n

Actual results:


Expected results:


Additional info:
I have reported this issue on the RDO Forum but no one has offered any assistance.  I opened a bug upstream:
https://bugs.launchpad.net/keystone/+bug/1221331

But was told this is not an issue with the keystone client itself but rather a problem with the RDO packaging, likely an out-of-sync version of the keystone client.

Comment 1 Robin Goldstone 2013-09-10 18:16:42 UTC
Is anyone from RedHat going to look at this?

Comment 2 Jakub Ruzicka 2013-09-11 20:47:08 UTC
I'm unable to reproduce this on RHEL 6.4 VM.

Can you please try latest python-keystoneclient (bellow) and let me know if it works?

yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/python-keystoneclient-0.3.2-3.el6.noarch.rpm

Comment 3 Robin Goldstone 2013-09-11 22:19:13 UTC
I did try this the other day but it complained about missing dependencies.  Are you saying I should be able to drop this RPM in on top of the RDO Grizzly release and it should install cleanly and work?  Did you try to update to this version of the RPM on your RDO Grizzly system?

Comment 4 Jakub Ruzicka 2013-09-12 15:19:38 UTC
Yes, it works for me on RHEL 6.4 with RDO Grizzly, EPEL 6 and RHEL 'optional' repos enabled. All these are required for RDO Grizzly installation, so it's a standard setup.

Also, please paste here output of `yum repolist`.

Comment 5 Robin Goldstone 2013-09-12 15:26:01 UTC
I do not have the "optional" channel enabled.  I see where this is mentioned in the RDO documentation: http://openstack.redhat.com/Repositories but it doesn't explain how to do this.  My cluster is located behind a firewall and we typically don't access external repos directly so if I need to set up the optional channel, I will need more specific instructions on how this is done.

Comment 6 Jakub Ruzicka 2013-09-12 16:23:11 UTC
As of enabling the optional repo, I must redirect you to Red Hat support.

However, I just tried to install Grizzly without the optional repo and it worked for me, including the installation of the havana keystoneclient. There seems to be something strange about your setup.

What is the error when installing the newer keystone rpm?

What is the `yum repolist` output?

Comment 7 Robin Goldstone 2013-09-13 15:42:14 UTC
Here is 'yum repolist' output as requested. The TOSS repo is our internal RHEL 6.4 repository.  I do not know how to set up the 'optional' channel but per your previous message, this should not be required.
# brutus1 /root > yum repolist
Loaded plugins: changelog, priorities, ps, refresh-packagekit, security
73 packages excluded due to repository priority protections
repo id                                   repo name                                                              status
TOSS                                      TOSS                                                                   2754+11
epel                                      Extra Packages for Enterprise Linux 6 - x86_64                         9519+62
openstack-grizzly                         OpenStack Grizzly Repository                                            286+74
repolist: 12559

Here is output showing my reported 'NoneType' object is unsubscriptable error after the initial Grizzly install.  I then installed the newer python-keystone client per your instructions and attempted to run the same keystone command.  As you can see, this produces a bunch of errors that seem to indicate this version of keystone client is incompatible with the RDO Grizzly installation.  

# brutus1 /root > export OS_SERVICE_TOKEN=dcda2d4ceb6d44ef8f7e46abe37cae63
# brutus1 /root > export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0
# brutus1 /root > keystone service-list
'NoneType' object is unsubscriptable
# brutus1 /root > 
# brutus1 /root > yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/python-keystoneclient-0.3.2-3.el6.noarch.rpm
Loaded plugins: changelog, priorities, ps, refresh-packagekit, security
73 packages excluded due to repository priority protections
Setting up Install Process
python-keystoneclient-0.3.2-3.el6.noarch.rpm                                                     | 151 kB     00:00     
Examining /var/tmp/yum-root-1w2Xeq/python-keystoneclient-0.3.2-3.el6.noarch.rpm: 1:python-keystoneclient-0.3.2-3.el6.noarch
Marking /var/tmp/yum-root-1w2Xeq/python-keystoneclient-0.3.2-3.el6.noarch.rpm as an update to 1:python-keystoneclient-0.2.3-7.el6.noarch
Resolving Dependencies
--> Running transaction check
---> Package python-keystoneclient.noarch 1:0.2.3-7.el6 will be updated
---> Package python-keystoneclient.noarch 1:0.3.2-3.el6 will be an update
--> Processing Dependency: python-httplib2 for package: 1:python-keystoneclient-0.3.2-3.el6.noarch
--> Running transaction check
---> Package python-httplib2.noarch 0:0.7.7-1.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package                      Arch          Version              Repository                                        Size
========================================================================================================================
Updating:
 python-keystoneclient        noarch        1:0.3.2-3.el6        /python-keystoneclient-0.3.2-3.el6.noarch        592 k
Installing for dependencies:
 python-httplib2              noarch        0.7.7-1.el6          epel                                              70 k

Transaction Summary
========================================================================================================================
Install       1 Package(s)
Upgrade       1 Package(s)

Total size: 662 k
Total download size: 70 k
Is this ok [y/N]: y
Downloading Packages:
python-httplib2-0.7.7-1.el6.noarch.rpm                                                           |  70 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : python-httplib2-0.7.7-1.el6.noarch                                                                   1/3 
  Updating   : 1:python-keystoneclient-0.3.2-3.el6.noarch                                                           2/3 
  Cleanup    : 1:python-keystoneclient-0.2.3-7.el6.noarch                                                           3/3 
  Verifying  : python-httplib2-0.7.7-1.el6.noarch                                                                   1/3 
  Verifying  : 1:python-keystoneclient-0.3.2-3.el6.noarch                                                           2/3 
  Verifying  : 1:python-keystoneclient-0.2.3-7.el6.noarch                                                           3/3 

Dependency Installed:
  python-httplib2.noarch 0:0.7.7-1.el6                                                                                  

Updated:
  python-keystoneclient.noarch 1:0.3.2-3.el6                                                                            

Complete!
# brutus1 /root > keystone service-list
Traceback (most recent call last):
  File "/usr/bin/keystone", line 6, in <module>
    from keystoneclient.shell import main
  File "/usr/lib/python2.6/site-packages/keystoneclient/shell.py", line 30, in <module>
    from keystoneclient import access
  File "/usr/lib/python2.6/site-packages/keystoneclient/access.py", line 21, in <module>
    from keystoneclient import service_catalog
  File "/usr/lib/python2.6/site-packages/keystoneclient/service_catalog.py", line 20, in <module>
    from keystoneclient import exceptions
  File "/usr/lib/python2.6/site-packages/keystoneclient/exceptions.py", line 7, in <module>
    from keystoneclient.openstack.common import jsonutils
  File "/usr/lib/python2.6/site-packages/keystoneclient/openstack/common/jsonutils.py", line 44, in <module>
    import netaddr
ImportError: No module named netaddr
# brutus1 /root >

Comment 8 Jakub Ruzicka 2013-09-16 15:01:05 UTC
This was just a missing dependency on python-nettaddr. You can either `yum install python-nettaddr` or use fixed build:

yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/python-keystoneclient-0.3.2-4.el6.noarch.rpm

Comment 9 Robin Goldstone 2013-09-17 14:26:07 UTC
I updated per your instructions and am now back to the original error:
# brutus1 /root/robing > rpm -q python-keystoneclient
python-keystoneclient-0.3.2-4.el6.noarch
# brutus1 /root/robing > keystone service-list
'NoneType' object is unsubscriptable

Comment 10 Jakub Ruzicka 2013-09-18 13:11:26 UTC
OK, so you get the same problem with both versions that were working for people on test day so I suspect there's something weird with your setup.

Please provide output of

# keystone --debug service-list

Can you reproduce this after reinstalling the machine, doing `yum update` and fresh install of RDO Grizzly?

Also, how up to date are these TOSS repos?

Comment 11 Robin Goldstone 2013-09-18 21:54:31 UTC
Here is the output you requested:
# brutus1 /root > keystone --debug service-list
REQ: curl -i -X GET http://127.0.0.1:35357/v2.0/OS-KSADM/services -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: dcda2d4ceb6d44ef8f7e46abe37cae63"
RESP: [200] {'date': 'Wed, 18 Sep 2013 21:43:50 GMT', 'content-type': 'application/json', 'content-length': '25', 'vary': 'X-Auth-Token'}
RESP BODY: None

No body was returned.

I have done a full uninstall/reinstall of RDO twice with the same results. I cannot easily do an OS reinstall of the machine.  I would need to have a sysadmin wipe it and reinstall it.  

Our TOSS distro is derived from RHEL 6.4.  We track the base RHEL 6.4 releases  closely though my test system is one minor rev back.  I can give you an RPM list if you'd like but the documentation for RDO says it should work with RHEL 6.4 which is effectively what I am running.  
# brutus1 /root > cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.4 (Santiago)
And I'm running the Openstack kernel:
# brutus1 /root > uname -a
Linux brutus1 2.6.32-358.118.1.openstack.el6.x86_64 #1 SMP Wed Aug 14 13:18:08 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

Comment 12 Robin Goldstone 2013-09-20 20:02:00 UTC
Any more ideas?

Comment 13 Jakub Ruzicka 2013-09-23 14:30:59 UTC
So the problem is that keystoneclient thinks it received no response body.

I can't really help you debug that as I can't reproduce this at all.

I don't know how complete is the uninstall process but I doubt it's equal to clean start so you can try wipe, reinstall and fully update the system and then reproduce, so we at least know it wasn't a random flaky issue.

As the Grizzly install on RHEL was tested by many folks during recent test day and none of them hit this, I still find your setup/repos the most likely culprit.

Comment 14 Jakub Ruzicka 2013-09-23 15:17:31 UTC
Also, before wiping the system, I would definitely try using the upstream python-keystoneclient as suggested in the upstream bug.

Comment 15 Robin Goldstone 2013-09-23 15:54:19 UTC
I am not the only person who has hit this bug.  I initially reported it on the RDO forum and referenced two other postings on that forum reporting the same issue:
http://openstack.redhat.com/forum/discussion/113#Item_4
http://openstack.redhat.com/forum/discussion/247#Item_3

You will see that no one has responded to these other reports.

Also the original issue that I reported on the RDO forum was another problem where packstack failed to install puppet yet proceeded to complete "successfully" despite the fact that none of the puppet commands were actually executed.  

This was again the case above where you pointed me to a newer keystone client that in turn required another RPM, python-netaddr, that you hadn't included as a requisite in your packaging.  

This suggests to me a more general class of packaging issues namely you are making certain assumptions about RPMs that you expect to be installed as part of RHEL without explicitly requiring them as requisites in your packaging.  I originally tried to find someplace on the RDO documentation that actually listed all of the required RPMs and could not find anything.  

Re: using the upstream python-keystone client, you can see from my upstream bug report that I tried to install it but it is not packaged as an RPM and requires a tool called pip plus a level of python expertise that I do not possess.

Comment 16 Jakub Ruzicka 2013-09-24 09:35:21 UTC
I said that none of RDO testers hit this.

I'm aware of the thread as I even commented on it. Since noone answered you, noone knows what might be the problem.

Havana isn't even released and requirements are changing rapidly, that was the case of missing netaddr, which produced clear error indicating you should install the module.

Although certain assumptions are neccessary, requirements are specified explicitly in RPMs. If you find that is not the case, you are welcome to report a bug.

You have been given fairly simple debugging hints

 * try upstram client
 * try on a clean updated machine

Upstream even provided you with nice instructions how to install the upstream client which fails for you in an alarming way again suggesting something is wrong with your setup. But you refuse to do a clean install because it "would need a sysadmin".

The only other option I'm aware of is python debugging.

Comment 17 Robin Goldstone 2013-09-24 16:13:16 UTC
OK I fixed the issue I was having getting the upstream client installed.
# brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > pip install python-keystoneclient
Downloading/unpacking python-keystoneclient
  Downloading python-keystoneclient-0.3.2.tar.gz (231kB): 231kB downloaded
  Running setup.py egg_info for package python-keystoneclient
    
    Installed /var/tmp/root/pip-build-root/python-keystoneclient/pbr-0.5.21-py2.6.egg
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Downloading/unpacking pbr>=0.5.21,<1.0 (from python-keystoneclient)
  Downloading pbr-0.5.21.tar.gz (123kB): 123kB downloaded
  Running setup.py egg_info for package pbr
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.6/site-packages (from python-keystoneclient)
Requirement already satisfied (use --upgrade to upgrade): iso8601>=0.1.4 in /usr/lib/python2.6/site-packages (from python-keystoneclient)
Downloading/unpacking PrettyTable>=0.6,<0.8 (from python-keystoneclient)
  Downloading prettytable-0.7.2.tar.bz2
  Running setup.py egg_info for package PrettyTable
Downloading/unpacking requests>=1.1 (from python-keystoneclient)
  Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded
  Running setup.py egg_info for package requests
Requirement already satisfied (use --upgrade to upgrade): simplejson>=2.0.9 in /usr/lib64/python2.6/site-packages (from python-keystoneclient)
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.6/site-packages (from python-keystoneclient)
Requirement already satisfied (use --upgrade to upgrade): oslo.config>=1.1.0 in /usr/lib/python2.6/site-packages (from python-keystoneclient)
Requirement already satisfied (use --upgrade to upgrade): netaddr in /usr/lib/python2.6/site-packages (from python-keystoneclient)
Requirement already satisfied (use --upgrade to upgrade): pip>=1.0 in /usr/lib/python2.6/site-packages (from pbr>=0.5.21,<1.0->python-keystoneclient)
Installing collected packages: python-keystoneclient, pbr, PrettyTable, requests
  Running setup.py install for python-keystoneclient
    [pbr] Reusing existing SOURCES.txt
    Installing keystone script to /usr/bin
  Running setup.py install for pbr
    [pbr] Reusing existing SOURCES.txt
  Found existing installation: prettytable 0.5
    Uninstalling prettytable:
      Successfully uninstalled prettytable
  Running setup.py install for PrettyTable
  Found existing installation: requests 0.11.1
    Uninstalling requests:
      Successfully uninstalled requests
  Running setup.py install for requests
Successfully installed python-keystoneclient pbr PrettyTable requests
Cleaning up...
# brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 >

The upstream client seems to work correctly:
 brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > export OS_SERVICE_TOKEN=dcda2d4ceb6d44ef8f7e46abe37cae63
# brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0 
# brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > keystone service-list
# brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > keystone role-list
+----------------------------------+----------+
|                id                |   name   |
+----------------------------------+----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
+----------------------------------+----------+
# brutus1 /home/robing/rdo/python-keystoneclient-0.3.2 > 

OK, so what does this mean?  If the upstream client works and your RPM version of the same client doesn't, is this an RDO issue?

Also, I did try to reinstall on a clean updated machine.  And on that system I am back to the first error I reported on the RDO forum namely packstack appears to install everything correctly but in reality it does absolutely nothing because puppet is not installed on my system.  On the RDO forum they told me that packstack is supposed to install puppet if it is not already installed but as you can see, it does not do that.  Not only does it fail to install puppet, but it apparently does not even check return codes when it runs the puppet commands.  If it did, you would think it would notice that puppet is not even installed.  Instead it reports "installation completed successfully" when in fact nothing is installed.

# brutus2 /home/robing/rdo > yum install -y http://rdo.fedorapeople.org/openstack/openstack-grizzly/rdo-release-grizzly.rpm
Loaded plugins: changelog, ps, refresh-packagekit, security
file:///repo/toss/current/x86_64/repodata/repomd.xml: [Errno 14] Could not open/read file:///repo/toss/current/x86_64/repodata/repomd.xml
Trying other mirror.
TOSS                                                                                             | 3.6 kB     00:00     
TOSS/primary_db                                                                                  | 2.4 MB     00:00     
epel/metalink                                                                                    |  13 kB     00:00     
epel                                                                                             | 4.2 kB     00:00     
epel/primary_db                                                                                  | 5.6 MB     00:00     
Setting up Install Process
rdo-release-grizzly.rpm                                                                          | 7.4 kB     00:00     
Examining /var/tmp/yum-root-Km2rlg/rdo-release-grizzly.rpm: rdo-release-grizzly-3.noarch
Marking /var/tmp/yum-root-Km2rlg/rdo-release-grizzly.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package rdo-release.noarch 0:grizzly-3 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package                    Arch                  Version                     Repository                           Size
========================================================================================================================
Installing:
 rdo-release                noarch                grizzly-3                   /rdo-release-grizzly                3.3 k

Transaction Summary
========================================================================================================================
Install       1 Package(s)

Total size: 3.3 k
Installed size: 3.3 k
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : rdo-release-grizzly-3.noarch                                                                         1/1 
  Verifying  : rdo-release-grizzly-3.noarch                                                                         1/1 

Installed:
  rdo-release.noarch 0:grizzly-3                                                                                        

Complete!
# brutus2 /home/robing/rdo > packstack --allinone --os-quantum-install=n
Welcome to Installer setup utility

Installing:
Clean Up...                                            [ DONE ]
Setting up ssh keys...                                 [ DONE ]
Adding pre install manifest entries...                 [ DONE ]
Adding MySQL manifest entries...                       [ DONE ]
Adding QPID manifest entries...                        [ DONE ]
Adding Keystone manifest entries...                    [ DONE ]
Adding Glance Keystone manifest entries...             [ DONE ]
Adding Glance manifest entries...                      [ DONE ]
Adding Cinder Keystone manifest entries...             [ DONE ]
Installing dependencies for Cinder...                  [ DONE ]
Checking if the Cinder server has a cinder-volumes vg...[ DONE ]
Adding Cinder manifest entries...                      [ DONE ]
Adding Nova API manifest entries...                    [ DONE ]
Adding Nova Keystone manifest entries...               [ DONE ]
Adding Nova Cert manifest entries...                   [ DONE ]
Adding Nova Conductor manifest entries...              [ DONE ]
Adding Nova Compute manifest entries...                [ DONE ]
Adding Nova Scheduler manifest entries...              [ DONE ]
Adding Nova VNC Proxy manifest entries...              [ DONE ]
Adding Nova Common manifest entries...                 [ DONE ]
Adding Nova Network manifest entries...                [ DONE ]
Adding OpenStack Client manifest entries...            [ DONE ]
Adding Horizon manifest entries...                     [ DONE ]
Adding Swift Keystone manifest entries...              [ DONE ]
Adding Swift builder manifest entries...               [ DONE ]
Adding Swift proxy manifest entries...                 [ DONE ]
Adding Swift storage manifest entries...               [ DONE ]
Adding Swift common manifest entries...                [ DONE ]
Preparing servers...                                   [ DONE ]
Adding Nagios server manifest entries...               [ DONE ]
Adding Nagios host manifest entries...                 [ DONE ]
Adding post install manifest entries...                [ DONE ]
Installing Dependencies...                             [ DONE ]
Copying Puppet modules and manifests...                [ DONE ]
Applying Puppet manifests...
Applying 192.168.95.35_prescript.pp
192.168.95.35_prescript.pp :                                         [ DONE ]
Applying 192.168.95.35_mysql.pp
Applying 192.168.95.35_qpid.pp
192.168.95.35_mysql.pp :                                             [ DONE ]
192.168.95.35_qpid.pp :                                              [ DONE ]
Applying 192.168.95.35_keystone.pp
Applying 192.168.95.35_glance.pp
Applying 192.168.95.35_cinder.pp
192.168.95.35_keystone.pp :                                          [ DONE ]
192.168.95.35_cinder.pp :                                            [ DONE ]
192.168.95.35_glance.pp :                                            [ DONE ]
Applying 192.168.95.35_api_nova.pp
192.168.95.35_api_nova.pp :                                          [ DONE ]
Applying 192.168.95.35_nova.pp
Applying 192.168.95.35_osclient.pp
Applying 192.168.95.35_horizon.pp
192.168.95.35_nova.pp :                                              [ DONE ]
192.168.95.35_horizon.pp :                                           [ DONE ]
192.168.95.35_osclient.pp :                                          [ DONE ]
Applying 192.168.95.35_ring_swift.pp
192.168.95.35_ring_swift.pp :                                        [ DONE ]
Applying 192.168.95.35_swift.pp
Applying 192.168.95.35_nagios.pp
Applying 192.168.95.35_nagios_nrpe.pp
192.168.95.35_swift.pp :                                             [ DONE ]
192.168.95.35_nagios_nrpe.pp :                                       [ DONE ]
192.168.95.35_nagios.pp :                                            [ DONE ]
Applying 192.168.95.35_postscript.pp
192.168.95.35_postscript.pp :                                        [ DONE ]
                           [ DONE ]

 **** Installation completed successfully ******


Additional information:
 * A new answerfile was created in: /root/packstack-answers-20130923-155141.txt
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * Did not create a cinder volume group, one already existed
 * To use the command line tools you need to source the file /root/keystonerc_admin created on 192.168.95.35
 * To use the console, browse to http://192.168.95.35/dashboard
 * To use Nagios, browse to http://192.168.95.35/nagios username : nagiosadmin, password : 7fb16e4ee14c4c66
 * The installation log file is available at: /var/tmp/packstack/20130923-155141-gIO2zs/openstack-setup.log

# brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > cat 192.168.95.35_api_nova.pp.log 
flock: puppet: Success
# brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > which puppet
/usr/bin/which: no puppet in (/usr/lib64/qt-3.3/bin:/admin/bin:/admin/sbin:/admin/scripts:/sbin:/bin:/usr/sbin:/usr/bin)
# brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > rpm -qa|grep puppet
# brutus2 /var/tmp/packstack/20130923-155141-gIO2zs/manifests > 

This is further evidence that your installer is flawed i.e. making some very unrealistic assumptions about what it thinks should already be installed on the system.  Since this is a separate issue, I will open a new bug on it rather than discussing it further here.  

So now that we have established the upstream client works and your RPM doesn't, what is next?

Comment 18 Robin Goldstone 2013-09-24 19:31:07 UTC
Now that I have gotten the upstream keystone client working, I re-launched the packstack installer.  This time it failed as follows:
013-09-24 09:30:54::ERROR::ospluginutils::143::root:: Error  during remote puppet apply of /var/tmp/packstack/20130924-092742-z0mil9/manifests/198.128.33.144_cinder.pp
2013-09-24 09:30:54::ERROR::ospluginutils::144::root:: ^[[0;36mnotice: /Firewall[001 cinder incoming]/ensure: created^[[0m
^[[1;35merr: /Stage[main]/Cinder/Package[cinder]/ensure: change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-cinder' returned 1: Error: Package: python-cinder-2013.1.3-1.el6.noarch (openstack-grizzly)
           Requires: python-qpid
Error: Package: python-cinder-2013.1.3-1.el6.noarch (openstack-grizzly)
           Requires: python-cheetah
Error: Package: python-cinder-2013.1.3-1.el6.noarch (openstack-grizzly)
           Requires: python-paramiko
 You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
openstack-keystone-2013.1.3-1.el6.noarch has missing requires of python-keystoneclient >= ('1', '0.2.0', None)
^[[0m


So once again here is another case where you are installing a package, python-cinder, that has requisites, python-qpid, python-cheetah, python-paramiko, that are not already installed on my system and since you do not install them, it causes the installation to fail.

Secondly you can see that packstack is also complaining that I don't have python-keystoneclient installed which of course I don't because I uninstalled the RPM and instead installed the upstream version, which is not an RPM, as you instructed.  So while it might be a useful data point to know that the upstream version works, it is not a permanent solution since packstack is expecting to find the RPM.

Comment 19 Jakub Ruzicka 2013-09-25 13:43:57 UTC
Upstream client working smells fishy because it's the same version that you supposedly tested from havana repo. It's basically exactly the same tarball from PyPI you got by pip with minor cosmetic changes. Not sure how to figure out the difference remotely, though.

I'd focus on getting the clean install working first.

The packages you mention (python-qpid, python-cheetah, python-paramiko) are all available from standard rhel-6-server-rpms repo, not having them available sounds like problem of your repositories.

What is output of:

# yum info puppet
# yum info python-qpid
# yum info python-paramiko


Next, packstack succeeding without puppet is absurd, maybe Martin have any thoughts on this?

Comment 20 Martin Magr 2013-09-25 13:49:08 UTC
Packstack running command "/usr/bin/yum -d 0 -e 0 -y install openstack-cinder" would install all dependencies required, if you would have yum repo where those packages are located.

So first of all I would add those to your TOSS repo.

About the puppet problem, there is a separate bug for that problem as Robin already stated. Will check that ASAP.

Comment 21 Alan Pevec 2013-09-25 13:52:54 UTC
TOSS is obviously not a complete RHEL clone,
all dependecies you're missing (python-qpid, python-cheetah and python-paramiko) are coming from rhel-6-server-rpms on a normal RHEL subscribed machine.
Please try to reproduce on RHEL or CentOS.

Comment 22 Robin Goldstone 2013-09-25 16:24:59 UTC
OK first, as Martin noted, we now have a separate bug for the puppet issue (1011628) so no need to hash that out further here.

The TOSS/RHEL thing is complicated but I will take a stab at explaining it so you have a better understanding of what I am trying to do and why.  TOSS is our customized HPC-centric RHEL distribution.  TOSS is used by LLNL, Los Alamos and Sandia Labs on all our large HPC clusters.  In addition to a few locally-developed RPMs and a few HPC-specific kernel patches, we also strip out a lot of the base RHEL RPMs that we don't expect (or want) our users running on these systems.  We specifically don't point to the RHEL repo in order to maintain our "controlled" distro.

Obviously this presents issues for a project like this where we want to install something new and it may require RPMs that are not in TOSS.  The reason I am trying to install RDO is that we have identified several interesting use cases for OpenStack within an HPC environment that we would like to explore.  Ultimately, if we decided to run OpenStack, we will have to pull all the necessary RPMs into TOSS and also figure out everything you are doing with puppet since we don't use puppet for configuration management (we use cfengine).  

I have discussed this with the guy who owns TOSS and his recommendation is that I add an entry for our RHEL satellite server to my yum config which will hopefully enable packstack to pull in the base RHEL requisites that it needs.  Once everything is fully installed, we can diff the resulting rpm list against what is in TOSS to figure out what we need to pull in.  I will do this shortly in the hopes that it resolves the issue with the cinder installation.

Maybe this will fix the keystone issue too but I am not holding my breath.  The cinder issue was obvious in that packstack threw an error indicating it could not find the requisite RPMs.  The keystone RPM installed successfully suggesting it is not missing any requisites.  I will start over on a clean machine with access to the RHEL repo and see what happens.

Comment 23 Robin Goldstone 2013-09-25 22:21:37 UTC
OK I finally got packstack to complete successfully after having to do a number of hacks in addition to adding the RHEL repos.  Keystone is now working.  

This still points to issues with the installer namely when it fails to install the necessary requisites, it seems to not notice and just continue along blindly with the installation.  I have proven this to be the case i.e. when I didn't have the RHEL repo configured, packstack completed this step very quickly:
Installing Dependencies...                             [ DONE ]
appearing as if it had installed stuff when in reality it didn't.  Once I added the RHEL repos, packstack spent a lot more time in this step and actually installed the required RPMs.  I will continue to try to sort out these packaging issues with Martin over in bug 1011628.

Comment 24 Jakub Ruzicka 2013-09-26 13:09:22 UTC
Glad to hear that, Robin!