Bug 1151227

(In reply to Prasanth Anbalagan from comment #0)
> 2. Create EC2 credentials
> 
> [root@........]# keystone ec2-credentials-create --user-id
> 4169f08e010a48e2a985d975e38bede8 --tenant-id 21d99917df7544379af88233612f987f
> 
> +-----------+----------------------------------+
> |  Property |              Value               |
> +-----------+----------------------------------+
> |   access  | 3e19884c2a42498c8a64dbce0f5d5a3f |
> |   secret  | 1ee80a30ac214ad681de55013007da3f |
> | tenant_id | 21d99917df7544379af88233612f987f |
> |  trust_id |                                  |
> |  user_id  | 4169f08e010a48e2a985d975e38bede8 |
> +-----------+----------------------------------+
> 
> 3. List the credentials.
> 
> [root@stone-vds1 s3-curl(keystone_admin)]# keystone ec2-credentials-list
> 
> [root@stone-vds1 s3-curl(keystone_admin)]# keystone ec2-credentials-list
> 
> [root@stone-vds1 s3-curl(keystone_admin)]# 

Are you creating the credentials for the same user you are logged in as?  You are specifying the '--user-id' option to create the credentials for a specific user (as opposed to the current user), though you are not using the '--user-id' option to state which user's ec2-credentials you want to list.  See this example from my RHOS 4.0 installation:

[rhosuser@rhos ~(keystone_admin)]$ keystone user-get 7d102cf9ecd14b218504fe623fb6fe5d
+----------+----------------------------------+
| Property |              Value               |
+----------+----------------------------------+
|  email   |                                  |
| enabled  |               True               |
|    id    | 7d102cf9ecd14b218504fe623fb6fe5d |
|   name   |               demo               |
| tenantId | a9f0ea377e854bce91be0c4caf63427c |
+----------+----------------------------------+
[rhosuser@rhos ~(keystone_admin)]$ keystone ec2-credentials-create --user-id 7d102cf9ecd14b218504fe623fb6fe5d
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|   access  | 289086f7af814aa092347407141a065d |
|   secret  | d3de23db4d744e6dbcbe1d90849fc1c6 |
| tenant_id | 0f3bbf873c1a462fa2045a33407bc763 |
|  trust_id |                                  |
|  user_id  | 7d102cf9ecd14b218504fe623fb6fe5d |
+-----------+----------------------------------+
[rhosuser@rhos ~(keystone_admin)]$ keystone ec2-credentials-list

[rhosuser@rhos ~(keystone_admin)]$ keystone ec2-credentials-list --user-id 7d102cf9ecd14b218504fe623fb6fe5d
+--------+----------------------------------+----------------------------------+
| tenant |              access              |              secret              |
+--------+----------------------------------+----------------------------------+
| admin  | 289086f7af814aa092347407141a065d | d3de23db4d744e6dbcbe1d90849fc1c6 |
+--------+----------------------------------+----------------------------------+

If you enable debug logging in keystone.conf, do you see messages similar to this in /etc/keystone/keystone.log when you attempt to use s3curl?

2014-10-09 23:08:26.843 6458 INFO sqlalchemy.engine.base.Engine [-] SELECT credential.id AS credential_id, credential.user_id AS credential_user_id, credential.project_id AS credential_project_id, credential.`blob` AS credential_blob, credential.type AS credential_type, credential.extra AS credential_extra
FROM credential
WHERE credential.id = %s
2014-10-09 23:08:26.843 6458 INFO sqlalchemy.engine.base.Engine [-] ('7c62a8fe087bc165edc4d3e90b99245f0e91a5b43c61047a4ee336ad5112f62e',)
2014-10-09 23:08:26.844 6458 WARNING keystone.common.wsgi [-] Authorization failed. Credential signature mismatch from 127.0.0.1
2014-10-09 23:08:26.845 6458 INFO access [-] 127.0.0.1 - - [10/Oct/2014:03:08:26 +0000] "POST http://127.0.0.1:35357/v2.0/s3tokens HTTP/1.0" 401 93

It looks like there is a mismatch in the message that s3curl and keystone sign and compare.  This results in a signature mismatch.  The key is being looked up properly by Keystone.  I can see this with some extra logging and debugging I performed on each side:

[rhosuser@rhos s3-curl(keystone_admin)]$ ./s3curl.pl --debug --id=personal --createBucket -- http://192.168.128.102:8080/stuff
s3curl: Found the url: host=192.168.128.102; port=8080; uri=/stuff; query=;
s3curl: cname endpoint signing case
s3curl: StringToSign='PUT\n\n\nFri, 10 Oct 2014 03:39:57 +0000\n/192.168.128.102/stuff'
s3curl: SecretKey='412e0286427247ada8a6d8253d3f18cf'
s3curl: Signature='Rf4mzce9EwWKH0Cc1U58FdD6IkI='
...


(Pdb) msg
'PUT\n\n\nFri, 10 Oct 2014 03:39:57 +0000\n/stuff'
(Pdb) key
'412e0286427247ada8a6d8253d3f18cf'
(Pdb) credentials
{u'access': u'937bf81899014af0a45e5274285c32f8', u'token': u'UFVUCgoKRnJpLCAxMCBPY3QgMjAxNCAwMzozOTo1NyArMDAwMAovc3R1ZmY=', u'signature': u'Rf4mzce9EwWKH0Cc1U58FdD6IkI='}


The difference is that s3curl is adding in the IP address of the Keystone server just before the bucket name that I'm attempting to create.  This is missing from the message on the Keystone side.  I still need to dig into what the correct message should be.

It turns out that s3curl.pl is only designed to work with AWS, which isn't a big surprise.  If the host you are connecting to is not one of the real AWS endpoints, it prepends the host to the message that is signed.  You can see this in this extra debug logging I added to s3curl.pl:

---------------------------------------------------------------------------
[rhosuser@rhos s3-curl(keystone_admin)]$ ./s3curl.pl --debug --id=personal --createBucket -- http://192.168.128.102:8080/stuff
s3curl: Found the url: host=192.168.128.102; port=8080; uri=/stuff; query=;
s3curl: NGK - before getResourceToSign... /stuff
s3curl: NGK - host is 192.168.128.102
s3curl: NGK - processing endpoint s3.amazonaws.com
s3curl: NGK - processing endpoint s3-us-west-1.amazonaws.com
s3curl: NGK - processing endpoint s3-us-west-2.amazonaws.com
s3curl: NGK - processing endpoint s3-us-gov-west-1.amazonaws.com
s3curl: NGK - processing endpoint s3-eu-west-1.amazonaws.com
s3curl: NGK - processing endpoint s3-ap-southeast-1.amazonaws.com
s3curl: NGK - processing endpoint s3-ap-northeast-1.amazonaws.com
s3curl: NGK - processing endpoint s3-sa-east-1.amazonaws.com
s3curl: cname endpoint signing case
s3curl: NGK - after getResourceToSign... /192.168.128.102/stuff
s3curl: Resource is '/192.168.128.102/stuff'
s3curl: StringToSign='PUT\n\n\nFri, 10 Oct 2014 03:59:00 +0000\n/192.168.128.102/stuff'
---------------------------------------------------------------------------

If I modify s3curl.pl to not call the getResourceToSign() subroutine, everything works fine:

-----------------------------------------------------------------------------
[rhosuser@rhos s3-curl(keystone_admin)]$ ./s3curl.pl --id=personal --createBucket -- http://192.168.128.102:8080/stuff -v
* About to connect() to 192.168.128.102 port 8080 (#0)
*   Trying 192.168.128.102... connected
* Connected to 192.168.128.102 (192.168.128.102) port 8080 (#0)
> PUT /stuff HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 192.168.128.102:8080
> Accept: */*
> Date: Fri, 10 Oct 2014 04:04:12 +0000
> Authorization: AWS 937bf81899014af0a45e5274285c32f8:rNCG3F7j5WOOK+r1vpBjxET7dPg=
> Content-Length: 0
> 
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
< Location: stuff
< Content-Length: 0
< Date: Fri, 10 Oct 2014 04:04:13 GMT
< 
* Connection #0 to host 192.168.128.102 left intact
* Closing connection #0
-----------------------------------------------------------------------------

This seems like it's working as designed.  I'm not sure if it's correct for us to be adding the host to the message that is signed on the Keystone side or not.

Based off of the following specification for AWS S3, it appears that keystoneclient is doing the right thing:

  http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#ConstructingTheCanonicalizedResourceElement

There is nothing in here that mentions including the host as a part of tha canonicalized resource in the string that is signed.  This appears to be a deviation from the specification in s3curl.pl for vanity subdomains in AWS.  I believe that s3curl.pl is designed to be customized based off of this comment where the endpoints are defined:

--------------------------------------------------------
# begin customizing here
my @endpoints = ( 's3.amazonaws.com',
                  's3-us-west-1.amazonaws.com',
                  's3-us-west-2.amazonaws.com',
                  's3-us-gov-west-1.amazonaws.com',
                  's3-eu-west-1.amazonaws.com',
                  's3-ap-southeast-1.amazonaws.com',
                  's3-ap-northeast-1.amazonaws.com',
                  's3-sa-east-1.amazonaws.com', );
--------------------------------------------------------

If the domain being used by keytone is set in this list, then s3curl would work fine.  It is unforunate that a command-line option isn't allows to extend the endpoints.  I don't see anything that we can do in keystoneclient for this, as we are following the published specification.

Closing this as NOTABUG.