Bug 1471626 - [RFE] Enable AWS4 signature support in S3 auth for RHCS RGW when using Keystone
Summary: [RFE] Enable AWS4 signature support in S3 auth for RHCS RGW when using Keystone
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 3.0
Hardware: x86_64
OS: Linux
Target Milestone: rc
: 3.2
Assignee: Radoslaw Zarzynski
QA Contact: ceph-qe-bugs
Bara Ancincova
Depends On:
Blocks: 1629656
TreeView+ depends on / blocked
Reported: 2017-07-17 06:14 UTC by Vimal Kumar
Modified: 2019-01-03 19:01 UTC (History)
15 users (show)

Fixed In Version: RHEL: ceph-12.2.8-39.el7cp Ubuntu: ceph_12.2.8-36redhat1
Doc Type: Technology Preview
Doc Text:
.AWS4 signature support in S3 authentication for Ceph Object Gateway when using Keystone With this update, S3 user authentication using the new AWS4 signatures as a part of the Keystone service is supported as a Technology Preview. The Ceph Object Gateway supports a subset of the Amazon Secure Token Service (STS) REST APIs. STS Lite is one supported API. It provides access to a set of temporary credentials for identity and access management. For more information, see link:{developer-guide}#authentication-using-the-sts-lite-api-dev[Authentication using the STS Lite API (Technology Preview)] in the link:{developer-guide}[Developer Guide].
Clone Of:
Last Closed: 2019-01-03 19:01:20 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0020 None None None 2019-01-03 19:01:49 UTC

Description Vimal Kumar 2017-07-17 06:14:48 UTC
a) Description of problem:

S3 user-authentication using the new AWS4 signatures fail while the authentication in RGW is done via OpenStack Keystone. 

The same authentication method (S3 using AWS4 signatures) works fine if going directly to RGW. Openstack keystone doesn't seem to have the S3/AWS4 authentication method implemented.

Else, the authentication works fine with Openstack Keystone if AWS2 signatures are used. 

b) Version-Release number of selected component (if applicable):


c) How reproducible:


d) Steps to Reproduce:

* Integrate OpenStack Keystone with RHCS2.x RGW. 
* Try authenticating an S3 user with AWS4 signatures through Keystone, it should fail.
* Try authenticating an S3 user with AWS2 signatures, it should succeed.

e) Actual results:

S3 user authentication through Keystone fails while using AWS4 signatures.

f) Expected results:

S3 user auth via Keystone should work for both AWS2 and AWS4 signatures.

f) Additional info:

An upstream tracker with a similar request can be seen at http://tracker.ceph.com/issues/19056.

Comment 2 Radoslaw Zarzynski 2017-07-18 23:18:50 UTC
Support for AWSv4, in conjunction with Keystone as the auth backend, has been introduced to RadosGW very recently -- no longer than several months ago. At the moment it's available only in the master branch. That's it, the feature isn't in Jewel but will be released as a part of the upcoming Luminous LTS.

Also the Keystone's EC2/S3 compatibility WSGI middleware must be recent enough to offer the AWSv4 support. The patch has been merged with Keystone's code base in December 2015 [1]. It's worth to note that Keystone doesn't provide reasonable support for all parts of the AWSv4 (e.g. Browser Uploads and Chunked Transfers). Though, these missing items haven't been mentioned in the bug description.

The basic support for AWSv4/Keystone has been verified today on master branches of RadosGW. A testcase from the ceph/s3tests has been used in the procedure. The traffic between s3tests and RadosGW, as well between RadosGW and Keystone, has been sniffed out. Some parts of it are posted below:

    [rzarzynski@rzarzynski s3-tests]$ S3_USE_SIGV4=1 S3TEST_CONF=s3-keystone.conf ./virtualenv/bin/nosetests -v -a '!fails_on_rgw' s3tests.functional.test_headers:test_bucket_create_bad_contentlength_none
    s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok
    Ran 1 test in 7.999s
    [rzarzynski@rzarzynski s3-tests]$

Traffic between s3tests and RadosGW:

    [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 8000 -d lo
    [sudo] hasĹo uĹźytkownika rzarzynski: 
    interface: lo (
    filter: ( port 8000 ) and ((ip || ip6) || (vlan && (ip || ip6)))


    T -> [AP]
    PUT /rzarzynski-bf4npcxghpxqd9hcbn-1/ HTTP/1.1.
    Accept-Encoding: identity.
    Content-Length: 0.
    x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
    Host: localhost:8000:8000.
    Authorization: AWS4-HMAC-SHA256 Credential=6a19cd673509465793efb4a24e93ab22/20170718//s3/aws4_request,SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date,Signature=6a3cb1e6f61122815ff0b4284000e4639de5dfaebfde9dcc2a704eb809234cfa.
    X-Amz-Date: 20170718T131558Z.
    User-Agent: Boto/2.47.0 Python/2.7.13 Linux/4.11.5-200.fc25.x86_64.
    T -> [AP]
    HTTP/1.1 200 OK.
    x-amz-request-id: tx000000000000000000011-00596e0a0e-106c-default.
    Content-Length: 0.
    Date: Tue, 18 Jul 2017 13:15:59 GMT.

A part of the exchange between RadosGW and Keystone: 
    [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 35357 -d lo
    interface: lo (
    filter: ( port 35357 ) and ((ip || ip6) || (vlan && (ip || ip6)))
    T -> [AP]
    POST /v2.0/s3tokens HTTP/1.1.
    Host: localhost:35357.
    Accept: */*.
    X-Auth-Token: ADMIN.
    Content-Type: application/json.
    Content-Length: 319.
    Expect: 100-continue.
    T -> [AP]
    T -> [AP]
    HTTP/1.0 200 OK.

The "token" is just AWS's string-to-sign encoded in BASE64. Let's decode it to prove AWSv4 has been truly used:


Comment 3 Radoslaw Zarzynski 2017-07-18 23:26:57 UTC
[1] "s3 token authentication doesn't support v4 protocol", https://bugs.launchpad.net/keystone/+bug/1473042

Comment 4 Radoslaw Zarzynski 2017-07-19 08:40:38 UTC
It's worth to note that the EC2/S3 compatibility middleware of Keystone has limitations in the matter of AWSv4 support. At the moment we know about:

 * Browser Uploads. They need special methods [2] of string-to-sign handling. Keystone lacks them.
* Chunked uploads. A raw key is necessary to validate each chunk locally but Keystone doesn't reveal it. The alternative is to engage Keystone for each chunk separately but this would overload it very easily.

[2] http://docs.aws.amazon.com/AmazonS3/latest/API/images/sigV4-post.png

Comment 27 errata-xmlrpc 2019-01-03 19:01:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.