Bug 1471626 - [RFE] Enable AWS4 signature support in S3 auth for RHCS RGW when using Keystone
[RFE] Enable AWS4 signature support in S3 auth for RHCS RGW when using Keystone
Status: NEW
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
3.0
x86_64 Linux
medium Severity medium
: rc
: 4.0
Assigned To: Radoslaw Zarzynski
ceph-qe-bugs
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-17 02:14 EDT by Vimal Kumar
Modified: 2017-09-21 19:14 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vimal Kumar 2017-07-17 02:14:48 EDT
a) Description of problem:

S3 user-authentication using the new AWS4 signatures fail while the authentication in RGW is done via OpenStack Keystone. 

The same authentication method (S3 using AWS4 signatures) works fine if going directly to RGW. Openstack keystone doesn't seem to have the S3/AWS4 authentication method implemented.

Else, the authentication works fine with Openstack Keystone if AWS2 signatures are used. 

b) Version-Release number of selected component (if applicable):

RHCS2.x

c) How reproducible:

Always

d) Steps to Reproduce:

* Integrate OpenStack Keystone with RHCS2.x RGW. 
* Try authenticating an S3 user with AWS4 signatures through Keystone, it should fail.
* Try authenticating an S3 user with AWS2 signatures, it should succeed.

e) Actual results:

S3 user authentication through Keystone fails while using AWS4 signatures.

f) Expected results:

S3 user auth via Keystone should work for both AWS2 and AWS4 signatures.

f) Additional info:

An upstream tracker with a similar request can be seen at http://tracker.ceph.com/issues/19056.
Comment 2 Radoslaw Zarzynski 2017-07-18 19:18:50 EDT
Support for AWSv4, in conjunction with Keystone as the auth backend, has been introduced to RadosGW very recently -- no longer than several months ago. At the moment it's available only in the master branch. That's it, the feature isn't in Jewel but will be released as a part of the upcoming Luminous LTS.

Also the Keystone's EC2/S3 compatibility WSGI middleware must be recent enough to offer the AWSv4 support. The patch has been merged with Keystone's code base in December 2015 [1]. It's worth to note that Keystone doesn't provide reasonable support for all parts of the AWSv4 (e.g. Browser Uploads and Chunked Transfers). Though, these missing items haven't been mentioned in the bug description.

The basic support for AWSv4/Keystone has been verified today on master branches of RadosGW. A testcase from the ceph/s3tests has been used in the procedure. The traffic between s3tests and RadosGW, as well between RadosGW and Keystone, has been sniffed out. Some parts of it are posted below:

    [rzarzynski@rzarzynski s3-tests]$ S3_USE_SIGV4=1 S3TEST_CONF=s3-keystone.conf ./virtualenv/bin/nosetests -v -a '!fails_on_rgw' s3tests.functional.test_headers:test_bucket_create_bad_contentlength_none
    s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok
    
    ----------------------------------------------------------------------
    Ran 1 test in 7.999s
    
    OK
    [rzarzynski@rzarzynski s3-tests]$

Traffic between s3tests and RadosGW:

    [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 8000 -d lo
    [sudo] hasło użytkownika rzarzynski: 
    interface: lo (127.0.0.0/255.0.0.0)
    filter: ( port 8000 ) and ((ip || ip6) || (vlan && (ip || ip6)))
    ######

    [...]

    T 127.0.0.1:44764 -> 127.0.0.1:8000 [AP]
    PUT /rzarzynski-bf4npcxghpxqd9hcbn-1/ HTTP/1.1.
    Accept-Encoding: identity.
    Content-Length: 0.
    x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
    Host: localhost:8000:8000.
    Authorization: AWS4-HMAC-SHA256 Credential=6a19cd673509465793efb4a24e93ab22/20170718//s3/aws4_request,SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date,Signature=6a3cb1e6f61122815ff0b4284000e4639de5dfaebfde9dcc2a704eb809234cfa.
    X-Amz-Date: 20170718T131558Z.
    User-Agent: Boto/2.47.0 Python/2.7.13 Linux/4.11.5-200.fc25.x86_64.
    .
    
    ##
    T 127.0.0.1:8000 -> 127.0.0.1:44764 [AP]
    HTTP/1.1 200 OK.
    x-amz-request-id: tx000000000000000000011-00596e0a0e-106c-default.
    Content-Length: 0.
    Date: Tue, 18 Jul 2017 13:15:59 GMT.
    .

A part of the exchange between RadosGW and Keystone: 
    
    [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 35357 -d lo
    interface: lo (127.0.0.0/255.0.0.0)
    filter: ( port 35357 ) and ((ip || ip6) || (vlan && (ip || ip6)))
    ######
    T 127.0.0.1:42138 -> 127.0.0.1:35357 [AP]
    POST /v2.0/s3tokens HTTP/1.1.
    Host: localhost:35357.
    Accept: */*.
    X-Auth-Token: ADMIN.
    Content-Type: application/json.
    Content-Length: 319.
    Expect: 100-continue.
    .
    
    ##
    T 127.0.0.1:42138 -> 127.0.0.1:35357 [AP]
    {"credentials":{"access":"1b2038eb999041d697c5582e06f5b0af","token":"QVdTNC1ITUFDLVNIQTI1NgoyMDE3MDcxOFQxMzE2NDZaCjIwMTcwNzE4Ly9zMy9hd3M0X3JlcXVlc3QKZTRmMmYwNTRmNWY1YTIwMDkzNTc0MTc1NjY3Y2U0OTExNzIwODMyMTM5ZGU2NDIxZTFjZmY3NzRiODlkMzZjZA==","signature":"215fb6d1a2bda77df4554b7e7b215a4efc6f98fd801226c7fda94ae3fe52729c"}}
    ##
    T 127.0.0.1:35357 -> 127.0.0.1:42138 [AP]
    HTTP/1.0 200 OK.
    
    [...]

The "token" is just AWS's string-to-sign encoded in BASE64. Let's decode it to prove AWSv4 has been truly used:

    $ echo QVdTNC1ITUFDLVNIQTI1NgoyMDE3MDcxOFQxMzE2NDZaCjIwMTcwNzE4Ly9zMy9hd3M0X3JlcXVlc3QKZTRmMmYwNTRmNWY1YTIwMDkzNTc0MTc1NjY3Y2U0OTExNzIwODMyMTM5ZGU2NDIxZTFjZmY3NzRiODlkMzZjZA== | base64 -d
    AWS4-HMAC-SHA256
    20170718T131646Z
    20170718//s3/aws4_request
    e4f2f054f5f5a20093574175667ce4911720832139de6421e1cff774b89d36cd
Comment 3 Radoslaw Zarzynski 2017-07-18 19:26:57 EDT
[1] "s3 token authentication doesn't support v4 protocol", https://bugs.launchpad.net/keystone/+bug/1473042
Comment 4 Radoslaw Zarzynski 2017-07-19 04:40:38 EDT
It's worth to note that the EC2/S3 compatibility middleware of Keystone has limitations in the matter of AWSv4 support. At the moment we know about:

 * Browser Uploads. They need special methods [2] of string-to-sign handling. Keystone lacks them.
* Chunked uploads. A raw key is necessary to validate each chunk locally but Keystone doesn't reveal it. The alternative is to engage Keystone for each chunk separately but this would overload it very easily.

[2] http://docs.aws.amazon.com/AmazonS3/latest/API/images/sigV4-post.png

Note You need to log in before you can comment on or make changes to this bug.