Description of problem: ======================= The issue was observed when trying to verify bug#1141929 - enabling swift3/s3token with swift. After enabling swift3/s3token, restarting proxy-server is successful but accessing swift object store using S3 API returns 'access denied' error. This is because ec2-credentials are not present in the database - 'keystone ec2-credentials-list' is empty even when the credentials are being created using 'keystone ec2-credentials-create' Version-Release number of selected component (if applicable): ============================================================= RHOS 4.0 How reproducible: ================= 100% Steps to Reproduce: ================= 1. Enable swift3/s3token based on, http://docs.openstack.org/grizzly/openstack-compute/admin/content/configuring-swift-with-s3-emulation-to-use-keystone.html Note: use keystoneclient instead of keystone.middleware 2. Create EC2 credentials [root@........]# keystone ec2-credentials-create --user-id 4169f08e010a48e2a985d975e38bede8 --tenant-id 21d99917df7544379af88233612f987f +-----------+----------------------------------+ | Property | Value | +-----------+----------------------------------+ | access | 3e19884c2a42498c8a64dbce0f5d5a3f | | secret | 1ee80a30ac214ad681de55013007da3f | | tenant_id | 21d99917df7544379af88233612f987f | | trust_id | | | user_id | 4169f08e010a48e2a985d975e38bede8 | +-----------+----------------------------------+ 3. List the credentials. [root@stone-vds1 s3-curl(keystone_admin)]# keystone ec2-credentials-list [root@stone-vds1 s3-curl(keystone_admin)]# keystone ec2-credentials-list [root@stone-vds1 s3-curl(keystone_admin)]# P.S: If this does not reproduce the problem, try deleting the first set of credentials and then recreate them. I did this because during the first attempt, I created the credentials for the wrong user (a copy paste error from documentation). S3CURL OUTPUT ============== Note that I have a .s3curl file with the above ec2 credentials. [root@stone-vds1 s3-curl]# ./s3curl.pl --id=spersonal -- http://10.35.115.14:8080 -v * About to connect() to 10.35.115.14 port 8080 (#0) * Trying 10.35.115.14... connected * Connected to 10.35.115.14 (10.35.115.14) port 8080 (#0) GET / HTTP/1.1 User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 Host: 10.35.115.14:8080 Accept: */* Date: Wed, 08 Oct 2014 19:47:02 +0000 Authorization: AWS 4169f08e010a48e2a985d975e38bede8:ev7ex0VZOYKGN+umUC59vZXOQjU= HTTP/1.1 403 Forbidden Content-Length: 124 Content-Type: text/xml Date: Wed, 08 Oct 2014 19:47:02 GMT ?xml version="1.0" encoding="UTF-8"?> <Error> <Code>AccessDenied</Code> <Message>Access denied</Message> </Error> * Connection #0 to host 10.35.115.14 left intact * Closing connection #0 [root@stone-vds1 s3-curl]#
(In reply to Prasanth Anbalagan from comment #0) > 2. Create EC2 credentials > > [root@........]# keystone ec2-credentials-create --user-id > 4169f08e010a48e2a985d975e38bede8 --tenant-id 21d99917df7544379af88233612f987f > > +-----------+----------------------------------+ > | Property | Value | > +-----------+----------------------------------+ > | access | 3e19884c2a42498c8a64dbce0f5d5a3f | > | secret | 1ee80a30ac214ad681de55013007da3f | > | tenant_id | 21d99917df7544379af88233612f987f | > | trust_id | | > | user_id | 4169f08e010a48e2a985d975e38bede8 | > +-----------+----------------------------------+ > > 3. List the credentials. > > [root@stone-vds1 s3-curl(keystone_admin)]# keystone ec2-credentials-list > > [root@stone-vds1 s3-curl(keystone_admin)]# keystone ec2-credentials-list > > [root@stone-vds1 s3-curl(keystone_admin)]# Are you creating the credentials for the same user you are logged in as? You are specifying the '--user-id' option to create the credentials for a specific user (as opposed to the current user), though you are not using the '--user-id' option to state which user's ec2-credentials you want to list. See this example from my RHOS 4.0 installation: [rhosuser@rhos ~(keystone_admin)]$ keystone user-get 7d102cf9ecd14b218504fe623fb6fe5d +----------+----------------------------------+ | Property | Value | +----------+----------------------------------+ | email | | | enabled | True | | id | 7d102cf9ecd14b218504fe623fb6fe5d | | name | demo | | tenantId | a9f0ea377e854bce91be0c4caf63427c | +----------+----------------------------------+ [rhosuser@rhos ~(keystone_admin)]$ keystone ec2-credentials-create --user-id 7d102cf9ecd14b218504fe623fb6fe5d +-----------+----------------------------------+ | Property | Value | +-----------+----------------------------------+ | access | 289086f7af814aa092347407141a065d | | secret | d3de23db4d744e6dbcbe1d90849fc1c6 | | tenant_id | 0f3bbf873c1a462fa2045a33407bc763 | | trust_id | | | user_id | 7d102cf9ecd14b218504fe623fb6fe5d | +-----------+----------------------------------+ [rhosuser@rhos ~(keystone_admin)]$ keystone ec2-credentials-list [rhosuser@rhos ~(keystone_admin)]$ keystone ec2-credentials-list --user-id 7d102cf9ecd14b218504fe623fb6fe5d +--------+----------------------------------+----------------------------------+ | tenant | access | secret | +--------+----------------------------------+----------------------------------+ | admin | 289086f7af814aa092347407141a065d | d3de23db4d744e6dbcbe1d90849fc1c6 | +--------+----------------------------------+----------------------------------+
If you enable debug logging in keystone.conf, do you see messages similar to this in /etc/keystone/keystone.log when you attempt to use s3curl? 2014-10-09 23:08:26.843 6458 INFO sqlalchemy.engine.base.Engine [-] SELECT credential.id AS credential_id, credential.user_id AS credential_user_id, credential.project_id AS credential_project_id, credential.`blob` AS credential_blob, credential.type AS credential_type, credential.extra AS credential_extra FROM credential WHERE credential.id = %s 2014-10-09 23:08:26.843 6458 INFO sqlalchemy.engine.base.Engine [-] ('7c62a8fe087bc165edc4d3e90b99245f0e91a5b43c61047a4ee336ad5112f62e',) 2014-10-09 23:08:26.844 6458 WARNING keystone.common.wsgi [-] Authorization failed. Credential signature mismatch from 127.0.0.1 2014-10-09 23:08:26.845 6458 INFO access [-] 127.0.0.1 - - [10/Oct/2014:03:08:26 +0000] "POST http://127.0.0.1:35357/v2.0/s3tokens HTTP/1.0" 401 93
It looks like there is a mismatch in the message that s3curl and keystone sign and compare. This results in a signature mismatch. The key is being looked up properly by Keystone. I can see this with some extra logging and debugging I performed on each side: [rhosuser@rhos s3-curl(keystone_admin)]$ ./s3curl.pl --debug --id=personal --createBucket -- http://192.168.128.102:8080/stuff s3curl: Found the url: host=192.168.128.102; port=8080; uri=/stuff; query=; s3curl: cname endpoint signing case s3curl: StringToSign='PUT\n\n\nFri, 10 Oct 2014 03:39:57 +0000\n/192.168.128.102/stuff' s3curl: SecretKey='412e0286427247ada8a6d8253d3f18cf' s3curl: Signature='Rf4mzce9EwWKH0Cc1U58FdD6IkI=' ... (Pdb) msg 'PUT\n\n\nFri, 10 Oct 2014 03:39:57 +0000\n/stuff' (Pdb) key '412e0286427247ada8a6d8253d3f18cf' (Pdb) credentials {u'access': u'937bf81899014af0a45e5274285c32f8', u'token': u'UFVUCgoKRnJpLCAxMCBPY3QgMjAxNCAwMzozOTo1NyArMDAwMAovc3R1ZmY=', u'signature': u'Rf4mzce9EwWKH0Cc1U58FdD6IkI='} The difference is that s3curl is adding in the IP address of the Keystone server just before the bucket name that I'm attempting to create. This is missing from the message on the Keystone side. I still need to dig into what the correct message should be.
It turns out that s3curl.pl is only designed to work with AWS, which isn't a big surprise. If the host you are connecting to is not one of the real AWS endpoints, it prepends the host to the message that is signed. You can see this in this extra debug logging I added to s3curl.pl: --------------------------------------------------------------------------- [rhosuser@rhos s3-curl(keystone_admin)]$ ./s3curl.pl --debug --id=personal --createBucket -- http://192.168.128.102:8080/stuff s3curl: Found the url: host=192.168.128.102; port=8080; uri=/stuff; query=; s3curl: NGK - before getResourceToSign... /stuff s3curl: NGK - host is 192.168.128.102 s3curl: NGK - processing endpoint s3.amazonaws.com s3curl: NGK - processing endpoint s3-us-west-1.amazonaws.com s3curl: NGK - processing endpoint s3-us-west-2.amazonaws.com s3curl: NGK - processing endpoint s3-us-gov-west-1.amazonaws.com s3curl: NGK - processing endpoint s3-eu-west-1.amazonaws.com s3curl: NGK - processing endpoint s3-ap-southeast-1.amazonaws.com s3curl: NGK - processing endpoint s3-ap-northeast-1.amazonaws.com s3curl: NGK - processing endpoint s3-sa-east-1.amazonaws.com s3curl: cname endpoint signing case s3curl: NGK - after getResourceToSign... /192.168.128.102/stuff s3curl: Resource is '/192.168.128.102/stuff' s3curl: StringToSign='PUT\n\n\nFri, 10 Oct 2014 03:59:00 +0000\n/192.168.128.102/stuff' --------------------------------------------------------------------------- If I modify s3curl.pl to not call the getResourceToSign() subroutine, everything works fine: ----------------------------------------------------------------------------- [rhosuser@rhos s3-curl(keystone_admin)]$ ./s3curl.pl --id=personal --createBucket -- http://192.168.128.102:8080/stuff -v * About to connect() to 192.168.128.102 port 8080 (#0) * Trying 192.168.128.102... connected * Connected to 192.168.128.102 (192.168.128.102) port 8080 (#0) > PUT /stuff HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: 192.168.128.102:8080 > Accept: */* > Date: Fri, 10 Oct 2014 04:04:12 +0000 > Authorization: AWS 937bf81899014af0a45e5274285c32f8:rNCG3F7j5WOOK+r1vpBjxET7dPg= > Content-Length: 0 > < HTTP/1.1 200 OK < Content-Type: text/html; charset=UTF-8 < Location: stuff < Content-Length: 0 < Date: Fri, 10 Oct 2014 04:04:13 GMT < * Connection #0 to host 192.168.128.102 left intact * Closing connection #0 ----------------------------------------------------------------------------- This seems like it's working as designed. I'm not sure if it's correct for us to be adding the host to the message that is signed on the Keystone side or not.
Based off of the following specification for AWS S3, it appears that keystoneclient is doing the right thing: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#ConstructingTheCanonicalizedResourceElement There is nothing in here that mentions including the host as a part of tha canonicalized resource in the string that is signed. This appears to be a deviation from the specification in s3curl.pl for vanity subdomains in AWS. I believe that s3curl.pl is designed to be customized based off of this comment where the endpoints are defined: -------------------------------------------------------- # begin customizing here my @endpoints = ( 's3.amazonaws.com', 's3-us-west-1.amazonaws.com', 's3-us-west-2.amazonaws.com', 's3-us-gov-west-1.amazonaws.com', 's3-eu-west-1.amazonaws.com', 's3-ap-southeast-1.amazonaws.com', 's3-ap-northeast-1.amazonaws.com', 's3-sa-east-1.amazonaws.com', ); -------------------------------------------------------- If the domain being used by keytone is set in this list, then s3curl would work fine. It is unforunate that a command-line option isn't allows to extend the endpoints. I don't see anything that we can do in keystoneclient for this, as we are following the published specification. Closing this as NOTABUG.