a) Description of problem: S3 user-authentication using the new AWS4 signatures fail while the authentication in RGW is done via OpenStack Keystone. The same authentication method (S3 using AWS4 signatures) works fine if going directly to RGW. Openstack keystone doesn't seem to have the S3/AWS4 authentication method implemented. Else, the authentication works fine with Openstack Keystone if AWS2 signatures are used. b) Version-Release number of selected component (if applicable): RHCS2.x c) How reproducible: Always d) Steps to Reproduce: * Integrate OpenStack Keystone with RHCS2.x RGW. * Try authenticating an S3 user with AWS4 signatures through Keystone, it should fail. * Try authenticating an S3 user with AWS2 signatures, it should succeed. e) Actual results: S3 user authentication through Keystone fails while using AWS4 signatures. f) Expected results: S3 user auth via Keystone should work for both AWS2 and AWS4 signatures. f) Additional info: An upstream tracker with a similar request can be seen at http://tracker.ceph.com/issues/19056.
Support for AWSv4, in conjunction with Keystone as the auth backend, has been introduced to RadosGW very recently -- no longer than several months ago. At the moment it's available only in the master branch. That's it, the feature isn't in Jewel but will be released as a part of the upcoming Luminous LTS. Also the Keystone's EC2/S3 compatibility WSGI middleware must be recent enough to offer the AWSv4 support. The patch has been merged with Keystone's code base in December 2015 [1]. It's worth to note that Keystone doesn't provide reasonable support for all parts of the AWSv4 (e.g. Browser Uploads and Chunked Transfers). Though, these missing items haven't been mentioned in the bug description. The basic support for AWSv4/Keystone has been verified today on master branches of RadosGW. A testcase from the ceph/s3tests has been used in the procedure. The traffic between s3tests and RadosGW, as well between RadosGW and Keystone, has been sniffed out. Some parts of it are posted below: [rzarzynski@rzarzynski s3-tests]$ S3_USE_SIGV4=1 S3TEST_CONF=s3-keystone.conf ./virtualenv/bin/nosetests -v -a '!fails_on_rgw' s3tests.functional.test_headers:test_bucket_create_bad_contentlength_none s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok ---------------------------------------------------------------------- Ran 1 test in 7.999s OK [rzarzynski@rzarzynski s3-tests]$ Traffic between s3tests and RadosGW: [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 8000 -d lo [sudo] hasĹo uĹźytkownika rzarzynski: interface: lo (127.0.0.0/255.0.0.0) filter: ( port 8000 ) and ((ip || ip6) || (vlan && (ip || ip6))) ###### [...] T 127.0.0.1:44764 -> 127.0.0.1:8000 [AP] PUT /rzarzynski-bf4npcxghpxqd9hcbn-1/ HTTP/1.1. Accept-Encoding: identity. Content-Length: 0. x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855. Host: localhost:8000:8000. Authorization: AWS4-HMAC-SHA256 Credential=6a19cd673509465793efb4a24e93ab22/20170718//s3/aws4_request,SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date,Signature=6a3cb1e6f61122815ff0b4284000e4639de5dfaebfde9dcc2a704eb809234cfa. X-Amz-Date: 20170718T131558Z. User-Agent: Boto/2.47.0 Python/2.7.13 Linux/4.11.5-200.fc25.x86_64. . ## T 127.0.0.1:8000 -> 127.0.0.1:44764 [AP] HTTP/1.1 200 OK. x-amz-request-id: tx000000000000000000011-00596e0a0e-106c-default. Content-Length: 0. Date: Tue, 18 Jul 2017 13:15:59 GMT. . A part of the exchange between RadosGW and Keystone: [rzarzynski@rzarzynski ~]$ sudo ngrep -W byline port 35357 -d lo interface: lo (127.0.0.0/255.0.0.0) filter: ( port 35357 ) and ((ip || ip6) || (vlan && (ip || ip6))) ###### T 127.0.0.1:42138 -> 127.0.0.1:35357 [AP] POST /v2.0/s3tokens HTTP/1.1. Host: localhost:35357. Accept: */*. X-Auth-Token: ADMIN. Content-Type: application/json. Content-Length: 319. Expect: 100-continue. . ## T 127.0.0.1:42138 -> 127.0.0.1:35357 [AP] {"credentials":{"access":"1b2038eb999041d697c5582e06f5b0af","token":"QVdTNC1ITUFDLVNIQTI1NgoyMDE3MDcxOFQxMzE2NDZaCjIwMTcwNzE4Ly9zMy9hd3M0X3JlcXVlc3QKZTRmMmYwNTRmNWY1YTIwMDkzNTc0MTc1NjY3Y2U0OTExNzIwODMyMTM5ZGU2NDIxZTFjZmY3NzRiODlkMzZjZA==","signature":"215fb6d1a2bda77df4554b7e7b215a4efc6f98fd801226c7fda94ae3fe52729c"}} ## T 127.0.0.1:35357 -> 127.0.0.1:42138 [AP] HTTP/1.0 200 OK. [...] The "token" is just AWS's string-to-sign encoded in BASE64. Let's decode it to prove AWSv4 has been truly used: $ echo QVdTNC1ITUFDLVNIQTI1NgoyMDE3MDcxOFQxMzE2NDZaCjIwMTcwNzE4Ly9zMy9hd3M0X3JlcXVlc3QKZTRmMmYwNTRmNWY1YTIwMDkzNTc0MTc1NjY3Y2U0OTExNzIwODMyMTM5ZGU2NDIxZTFjZmY3NzRiODlkMzZjZA== | base64 -d AWS4-HMAC-SHA256 20170718T131646Z 20170718//s3/aws4_request e4f2f054f5f5a20093574175667ce4911720832139de6421e1cff774b89d36cd
[1] "s3 token authentication doesn't support v4 protocol", https://bugs.launchpad.net/keystone/+bug/1473042
It's worth to note that the EC2/S3 compatibility middleware of Keystone has limitations in the matter of AWSv4 support. At the moment we know about: * Browser Uploads. They need special methods [2] of string-to-sign handling. Keystone lacks them. * Chunked uploads. A raw key is necessary to validate each chunk locally but Keystone doesn't reveal it. The alternative is to engage Keystone for each chunk separately but this would overload it very easily. [2] http://docs.aws.amazon.com/AmazonS3/latest/API/images/sigV4-post.png
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0020