Bug 1471626

Support for AWSv4, in conjunction with Keystone as the auth backend, has been introduced to RadosGW very recently -- no longer than several months ago. At the moment it's available only in the master branch. That's it, the feature isn't in Jewel but will be released as a part of the upcoming Luminous LTS.

Also the Keystone's EC2/S3 compatibility WSGI middleware must be recent enough to offer the AWSv4 support. The patch has been merged with Keystone's code base in December 2015 [1]. It's worth to note that Keystone doesn't provide reasonable support for all parts of the AWSv4 (e.g. Browser Uploads and Chunked Transfers). Though, these missing items haven't been mentioned in the bug description.

The basic support for AWSv4/Keystone has been verified today on master branches of RadosGW. A testcase from the ceph/s3tests has been used in the procedure. The traffic between s3tests and RadosGW, as well between RadosGW and Keystone, has been sniffed out. Some parts of it are posted below:

    [rzarzynski@rzarzynski s3-tests]$ S3_USE_SIGV4=1 S3TEST_CONF=s3-keystone.conf ./virtualenv/bin/nosetests -v -a '!fails_on_rgw' s3tests.functional.test_headers:test_bucket_create_bad_contentlength_none
    s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok
    
    ----------------------------------------------------------------------
    Ran 1 test in 7.999s
    
    OK
    [rzarzynski@rzarzynski s3-tests]$

Traffic between s3tests and RadosGW:

    [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 8000 -d lo
    [sudo] hasĹo uĹźytkownika rzarzynski: 
    interface: lo (127.0.0.0/255.0.0.0)
    filter: ( port 8000 ) and ((ip || ip6) || (vlan && (ip || ip6)))
    ######

    [...]

    T 127.0.0.1:44764 -> 127.0.0.1:8000 [AP]
    PUT /rzarzynski-bf4npcxghpxqd9hcbn-1/ HTTP/1.1.
    Accept-Encoding: identity.
    Content-Length: 0.
    x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
    Host: localhost:8000:8000.
    Authorization: AWS4-HMAC-SHA256 Credential=6a19cd673509465793efb4a24e93ab22/20170718//s3/aws4_request,SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date,Signature=6a3cb1e6f61122815ff0b4284000e4639de5dfaebfde9dcc2a704eb809234cfa.
    X-Amz-Date: 20170718T131558Z.
    User-Agent: Boto/2.47.0 Python/2.7.13 Linux/4.11.5-200.fc25.x86_64.
    .
    
    ##
    T 127.0.0.1:8000 -> 127.0.0.1:44764 [AP]
    HTTP/1.1 200 OK.
    x-amz-request-id: tx000000000000000000011-00596e0a0e-106c-default.
    Content-Length: 0.
    Date: Tue, 18 Jul 2017 13:15:59 GMT.
    .

A part of the exchange between RadosGW and Keystone: 
    
    [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 35357 -d lo
    interface: lo (127.0.0.0/255.0.0.0)
    filter: ( port 35357 ) and ((ip || ip6) || (vlan && (ip || ip6)))
    ######
    T 127.0.0.1:42138 -> 127.0.0.1:35357 [AP]
    POST /v2.0/s3tokens HTTP/1.1.
    Host: localhost:35357.
    Accept: */*.
    X-Auth-Token: ADMIN.
    Content-Type: application/json.
    Content-Length: 319.
    Expect: 100-continue.
    .
    
    ##
    T 127.0.0.1:42138 -> 127.0.0.1:35357 [AP]
    {"credentials":{"access":"1b2038eb999041d697c5582e06f5b0af","token":"QVdTNC1ITUFDLVNIQTI1NgoyMDE3MDcxOFQxMzE2NDZaCjIwMTcwNzE4Ly9zMy9hd3M0X3JlcXVlc3QKZTRmMmYwNTRmNWY1YTIwMDkzNTc0MTc1NjY3Y2U0OTExNzIwODMyMTM5ZGU2NDIxZTFjZmY3NzRiODlkMzZjZA==","signature":"215fb6d1a2bda77df4554b7e7b215a4efc6f98fd801226c7fda94ae3fe52729c"}}
    ##
    T 127.0.0.1:35357 -> 127.0.0.1:42138 [AP]
    HTTP/1.0 200 OK.
    
    [...]

The "token" is just AWS's string-to-sign encoded in BASE64. Let's decode it to prove AWSv4 has been truly used:

    $ echo QVdTNC1ITUFDLVNIQTI1NgoyMDE3MDcxOFQxMzE2NDZaCjIwMTcwNzE4Ly9zMy9hd3M0X3JlcXVlc3QKZTRmMmYwNTRmNWY1YTIwMDkzNTc0MTc1NjY3Y2U0OTExNzIwODMyMTM5ZGU2NDIxZTFjZmY3NzRiODlkMzZjZA== | base64 -d
    AWS4-HMAC-SHA256
    20170718T131646Z
    20170718//s3/aws4_request
    e4f2f054f5f5a20093574175667ce4911720832139de6421e1cff774b89d36cd

[1] "s3 token authentication doesn't support v4 protocol", https://bugs.launchpad.net/keystone/+bug/1473042

It's worth to note that the EC2/S3 compatibility middleware of Keystone has limitations in the matter of AWSv4 support. At the moment we know about:

 * Browser Uploads. They need special methods [2] of string-to-sign handling. Keystone lacks them.
* Chunked uploads. A raw key is necessary to validate each chunk locally but Keystone doesn't reveal it. The alternative is to engage Keystone for each chunk separately but this would overload it very easily.

[2] http://docs.aws.amazon.com/AmazonS3/latest/API/images/sigV4-post.png

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0020