Description of problem: When I try to access S3 API with signature v4, request always fails because of SignatureDoesNotMatch. However request succeeds when I enforce signature v4. ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack ec2 credentials list +----------------------------------+----------------------------------+----------------------------------+----------------------------------+ | Access | Secret | Project ID | User ID | +----------------------------------+----------------------------------+----------------------------------+----------------------------------+ | 9ab0fd679ddc45dd902b808b80152b73 | e96207b812134bae9c25140ad5b37967 | f9669c61b3894287942ae6dce8d6a1da | 6399c2d698c245c9889e3fe8714f480f | +----------------------------------+----------------------------------+----------------------------------+----------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cat s3.cfg [default] access_key=9ab0fd679ddc45dd902b808b80152b73 host_base=https://10.0.0.101:13808/ host_bucket=https://10.0.0.101:13808/ secret_key=e96207b812134bae9c25140ad5b37967 [stack@undercloud-0 ~]$ s3cmd -c s3.cfg ls s3://testbucket ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method. [stack@undercloud-0 ~]$ s3cmd -c s3.cfg ls s3://testbucket --signature-v2 2020-12-22 13:48 173 s3://testbucket/s3.cfg ~~~ I noticed that [filter:s3api] location in proxy-server.conf is not properly set, but even after I update the parameter to the correct keystone region any request with signature v4 still fails. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create ec2 credentials $ openstack ec2 credentials create --user <user> --project <project> 2. Create s3.cfg file. [default] access_key=<access key> host_base=<endpoint url> host_bucket=<endpoint url> secret_key=<secret key> 3. Send request via S3 api $ s3cmd -c s3.cfg mb s3://testbucket Actual results: The request to S3 API fails because of SignatureDoesNotMatch Expected results: The request to S3 API succeeds Additional info:
Please find debug outputs of s3cmd below. ~~~ [stack@undercloud-0 ~]$ s3cmd -c s3.cfg ls s3://testbucket --debug DEBUG: s3cmd version 2.0.2 DEBUG: ConfigParser: Reading file 's3.cfg' DEBUG: ConfigParser: access_key->9a...30_chars... DEBUG: ConfigParser: host_base->https://10.0.0.101:13808/ DEBUG: ConfigParser: host_bucket->https://10.0.0.101:13808/ DEBUG: ConfigParser: secret_key->e9...29_chars...7 DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Command: ls DEBUG: Bucket 's3://testbucket': DEBUG: CreateRequest: resource[uri]=/ DEBUG: ===== Send_request inner request to determine the bucket region ===== DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: canonical_headers = host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z DEBUG: Canonical Request: GET /testbucket/ location= host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=c9203dbeee940eba4e5cbeaac1c6fc03ef4236adb270bd7cb3d19ff9b260fd95', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: ConnMan.get(): creating new connection: https://10.0.0.101:13808/ DEBUG: Using ca_certs_file None DEBUG: httplib.HTTPSConnection() has both context and check_hostname DEBUG: non-proxied HTTPSConnection(10.0.0.101, 13808) DEBUG: format_uri(): /testbucket/?location DEBUG: Sending request method_string='GET', uri='/testbucket/?location', headers={'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=c9203dbeee940eba4e5cbeaac1c6fc03ef4236adb270bd7cb3d19ff9b260fd95', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://10.0.0.101:13808/#1) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>Authorizatio" b'nHeaderMalformed</Code><Message>The authorization header is malforme' b"d; the region 'us-east-1' is wrong; expecting 'regionOne'</Message><" b'RequestId>tx11f6b198718f4230b8d54-005fe286a6</RequestId><Region>regi' b'onOne</Region></Error>', 'headers': {'content-type': 'application/xml', 'date': 'Tue, 22 Dec 2020 23:52:06 GMT', 'transfer-encoding': 'chunked', 'x-amz-id-2': 'tx11f6b198718f4230b8d54-005fe286a6', 'x-amz-request-id': 'tx11f6b198718f4230b8d54-005fe286a6', 'x-openstack-request-id': 'tx11f6b198718f4230b8d54-005fe286a6', 'x-trans-id': 'tx11f6b198718f4230b8d54-005fe286a6'}, 'reason': 'Bad Request', 'status': 400} INFO: Forwarding request to regionOne DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: canonical_headers = host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z DEBUG: Canonical Request: GET /testbucket/ location= host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/regionOne/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=89cc936333828386d360cbbd63b81de26e211c22b3e4679fc0afcb1bb13c5493', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: ConnMan.get(): re-using connection: https://10.0.0.101:13808/#1 DEBUG: format_uri(): /testbucket/?location DEBUG: Sending request method_string='GET', uri='/testbucket/?location', headers={'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/regionOne/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=89cc936333828386d360cbbd63b81de26e211c22b3e4679fc0afcb1bb13c5493', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://10.0.0.101:13808/#2) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>SignatureDoe" b'sNotMatch</Code><Message>The request signature we calculated does no' b't match the signature you provided. Check your key and signing metho' b'd.</Message><RequestId>tx327abf7a103c43febc74c-005fe286a6</RequestId' b'><AWSAccessKeyId>9ab0fd679ddc45dd902b808b80152b73</AWSAccessKeyId><S' b'tringToSign>AWS4-HMAC-SHA256\n20201222T235206Z\n20201222/regionOne' b'/s3/aws4_request\n7a901355f7910feed997deaeba38127af99addcf4c3e7e8' b'bd3de424564254250</StringToSign><SignatureProvided>89cc936333828386d' b'360cbbd63b81de26e211c22b3e4679fc0afcb1bb13c5493</SignatureProvided><' b'StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a' b' 32 30 32 30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 32 30 32 30 31 3' b'2 32 32 2f 72 65 67 69 6f 6e 4f 6e 65 2f 73 33 2f 61 77 73 34 5f 72 ' b'65 71 75 65 73 74 0a 37 61 39 30 31 33 35 35 66 37 39 31 30 66 65 65' b' 64 39 39 37 64 65 61 65 62 61 33 38 31 32 37 61 66 39 39 61 64 64 6' b'3 66 34 63 33 65 37 65 38 62 64 33 64 65 34 32 34 35 36 34 32 35 34 ' b'32 35 30</StringToSignBytes><CanonicalRequest>GET\n/testbucket/\nl' b'ocation=\nhost:10.0.0.101:13808\nx-amz-content-sha256:e3b0c44298fc' b'1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:' b'20201222T235206Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c4429' b'8fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRe' b'quest><CanonicalRequestBytes>47 45 54 0a 2f 74 65 73 74 62 75 63 6b ' b'65 74 2f 0a 6c 6f 63 61 74 69 6f 6e 3d 0a 68 6f 73 74 3a 31 30 2e 30' b' 2e 30 2e 31 30 31 3a 31 33 38 30 38 0a 78 2d 61 6d 7a 2d 63 6f 6e 7' b'4 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 ' b'63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37' b' 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 3' b'7 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 ' b'30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 0a 68 6f 73 74 3b 78 2d 61' b' 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7' b'a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 ' b'39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65' b' 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 3' b'8 35 35</CanonicalRequestBytes></Error>', 'headers': {'content-type': 'application/xml', 'date': 'Tue, 22 Dec 2020 23:52:06 GMT', 'transfer-encoding': 'chunked', 'x-amz-id-2': 'tx327abf7a103c43febc74c-005fe286a6', 'x-amz-request-id': 'tx327abf7a103c43febc74c-005fe286a6', 'x-openstack-request-id': 'tx327abf7a103c43febc74c-005fe286a6', 'x-trans-id': 'tx327abf7a103c43febc74c-005fe286a6'}, 'reason': 'Forbidden', 'status': 403} DEBUG: S3Error: 403 (Forbidden) DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: x-amz-id-2: tx327abf7a103c43febc74c-005fe286a6 DEBUG: HttpHeader: x-amz-request-id: tx327abf7a103c43febc74c-005fe286a6 DEBUG: HttpHeader: x-trans-id: tx327abf7a103c43febc74c-005fe286a6 DEBUG: HttpHeader: x-openstack-request-id: tx327abf7a103c43febc74c-005fe286a6 DEBUG: HttpHeader: date: Tue, 22 Dec 2020 23:52:06 GMT DEBUG: HttpHeader: transfer-encoding: chunked DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch' DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.' DEBUG: ErrorXML: RequestId: 'tx327abf7a103c43febc74c-005fe286a6' DEBUG: ErrorXML: AWSAccessKeyId: '9ab0fd679ddc45dd902b808b80152b73' DEBUG: ErrorXML: StringToSign: 'AWS4-HMAC-SHA256\n20201222T235206Z\n20201222/regionOne/s3/aws4_request\n7a901355f7910feed997deaeba38127af99addcf4c3e7e8bd3de424564254250' DEBUG: ErrorXML: SignatureProvided: '89cc936333828386d360cbbd63b81de26e211c22b3e4679fc0afcb1bb13c5493' DEBUG: ErrorXML: StringToSignBytes: '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 32 30 32 30 31 32 32 32 2f 72 65 67 69 6f 6e 4f 6e 65 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 37 61 39 30 31 33 35 35 66 37 39 31 30 66 65 65 64 39 39 37 64 65 61 65 62 61 33 38 31 32 37 61 66 39 39 61 64 64 63 66 34 63 33 65 37 65 38 62 64 33 64 65 34 32 34 35 36 34 32 35 34 32 35 30' DEBUG: ErrorXML: CanonicalRequest: 'GET\n/testbucket/\nlocation=\nhost:10.0.0.101:13808\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20201222T235206Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' DEBUG: ErrorXML: CanonicalRequestBytes: '47 45 54 0a 2f 74 65 73 74 62 75 63 6b 65 74 2f 0a 6c 6f 63 61 74 69 6f 6e 3d 0a 68 6f 73 74 3a 31 30 2e 30 2e 30 2e 31 30 31 3a 31 33 38 30 38 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35' DEBUG: Error getlocation inner request: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method. DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: canonical_headers = host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z DEBUG: Canonical Request: GET /testbucket/ delimiter=%2F host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=638958b4c2a68a373e5b435095c91ba0ed2d3f1a3423c466ebb925e64fa4d9d8', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: ConnMan.get(): re-using connection: https://10.0.0.101:13808/#2 DEBUG: format_uri(): /testbucket/?delimiter=%2F DEBUG: Sending request method_string='GET', uri='/testbucket/?delimiter=%2F', headers={'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=638958b4c2a68a373e5b435095c91ba0ed2d3f1a3423c466ebb925e64fa4d9d8', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://10.0.0.101:13808/#3) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>Authorizatio" b'nHeaderMalformed</Code><Message>The authorization header is malforme' b"d; the region 'US' is wrong; expecting 'regionOne'</Message><Request" b'Id>txa74d6ce97fc648d4a282d-005fe286a6</RequestId><Region>regionOne</' b'Region></Error>', 'headers': {'content-type': 'application/xml', 'date': 'Tue, 22 Dec 2020 23:52:06 GMT', 'transfer-encoding': 'chunked', 'x-amz-id-2': 'txa74d6ce97fc648d4a282d-005fe286a6', 'x-amz-request-id': 'txa74d6ce97fc648d4a282d-005fe286a6', 'x-openstack-request-id': 'txa74d6ce97fc648d4a282d-005fe286a6', 'x-trans-id': 'txa74d6ce97fc648d4a282d-005fe286a6'}, 'reason': 'Bad Request', 'status': 400} INFO: Forwarding request to regionOne DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: canonical_headers = host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z DEBUG: Canonical Request: GET /testbucket/ delimiter=%2F host:10.0.0.101:13808/ x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20201222T235206Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/regionOne/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=dc823af2f694bd0d9d14ec6e731364d4ced2df8e17c463eef79d3764dfdac1b2', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): 10.0.0.101:13808/ DEBUG: ConnMan.get(): re-using connection: https://10.0.0.101:13808/#3 DEBUG: format_uri(): /testbucket/?delimiter=%2F DEBUG: Sending request method_string='GET', uri='/testbucket/?delimiter=%2F', headers={'x-amz-date': '20201222T235206Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=9ab0fd679ddc45dd902b808b80152b73/20201222/regionOne/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=dc823af2f694bd0d9d14ec6e731364d4ced2df8e17c463eef79d3764dfdac1b2', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://10.0.0.101:13808/#4) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>SignatureDoe" b'sNotMatch</Code><Message>The request signature we calculated does no' b't match the signature you provided. Check your key and signing metho' b'd.</Message><RequestId>tx9847df41fad349c088148-005fe286a6</RequestId' b'><AWSAccessKeyId>9ab0fd679ddc45dd902b808b80152b73</AWSAccessKeyId><S' b'tringToSign>AWS4-HMAC-SHA256\n20201222T235206Z\n20201222/regionOne' b'/s3/aws4_request\n262e814262a86685a47bfe70646a1c3647b6afef3af778d' b'21e3fc23d120cdd91</StringToSign><SignatureProvided>dc823af2f694bd0d9' b'd14ec6e731364d4ced2df8e17c463eef79d3764dfdac1b2</SignatureProvided><' b'StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a' b' 32 30 32 30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 32 30 32 30 31 3' b'2 32 32 2f 72 65 67 69 6f 6e 4f 6e 65 2f 73 33 2f 61 77 73 34 5f 72 ' b'65 71 75 65 73 74 0a 32 36 32 65 38 31 34 32 36 32 61 38 36 36 38 35' b' 61 34 37 62 66 65 37 30 36 34 36 61 31 63 33 36 34 37 62 36 61 66 6' b'5 66 33 61 66 37 37 38 64 32 31 65 33 66 63 32 33 64 31 32 30 63 64 ' b'64 39 31</StringToSignBytes><CanonicalRequest>GET\n/testbucket/\nd' b'elimiter=%2F\nhost:10.0.0.101:13808\nx-amz-content-sha256:e3b0c442' b'98fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-d' b'ate:20201222T235206Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c' b'44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</Canonic' b'alRequest><CanonicalRequestBytes>47 45 54 0a 2f 74 65 73 74 62 75 63' b' 6b 65 74 2f 0a 64 65 6c 69 6d 69 74 65 72 3d 25 32 46 0a 68 6f 73 7' b'4 3a 31 30 2e 30 2e 30 2e 31 30 31 3a 31 33 38 30 38 0a 78 2d 61 6d ' b'7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34' b' 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 6' b'2 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 ' b'35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74' b' 65 3a 32 30 32 30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 0a 68 6f 7' b'3 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 ' b'3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66' b' 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 3' b'7 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 ' b'37 38 35 32 62 38 35 35</CanonicalRequestBytes></Error>', 'headers': {'content-type': 'application/xml', 'date': 'Tue, 22 Dec 2020 23:52:06 GMT', 'transfer-encoding': 'chunked', 'x-amz-id-2': 'tx9847df41fad349c088148-005fe286a6', 'x-amz-request-id': 'tx9847df41fad349c088148-005fe286a6', 'x-openstack-request-id': 'tx9847df41fad349c088148-005fe286a6', 'x-trans-id': 'tx9847df41fad349c088148-005fe286a6'}, 'reason': 'Forbidden', 'status': 403} DEBUG: S3Error: 403 (Forbidden) DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: x-amz-id-2: tx9847df41fad349c088148-005fe286a6 DEBUG: HttpHeader: x-amz-request-id: tx9847df41fad349c088148-005fe286a6 DEBUG: HttpHeader: x-trans-id: tx9847df41fad349c088148-005fe286a6 DEBUG: HttpHeader: x-openstack-request-id: tx9847df41fad349c088148-005fe286a6 DEBUG: HttpHeader: date: Tue, 22 Dec 2020 23:52:06 GMT DEBUG: HttpHeader: transfer-encoding: chunked DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch' DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.' DEBUG: ErrorXML: RequestId: 'tx9847df41fad349c088148-005fe286a6' DEBUG: ErrorXML: AWSAccessKeyId: '9ab0fd679ddc45dd902b808b80152b73' DEBUG: ErrorXML: StringToSign: 'AWS4-HMAC-SHA256\n20201222T235206Z\n20201222/regionOne/s3/aws4_request\n262e814262a86685a47bfe70646a1c3647b6afef3af778d21e3fc23d120cdd91' DEBUG: ErrorXML: SignatureProvided: 'dc823af2f694bd0d9d14ec6e731364d4ced2df8e17c463eef79d3764dfdac1b2' DEBUG: ErrorXML: StringToSignBytes: '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 32 30 32 30 31 32 32 32 2f 72 65 67 69 6f 6e 4f 6e 65 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 32 36 32 65 38 31 34 32 36 32 61 38 36 36 38 35 61 34 37 62 66 65 37 30 36 34 36 61 31 63 33 36 34 37 62 36 61 66 65 66 33 61 66 37 37 38 64 32 31 65 33 66 63 32 33 64 31 32 30 63 64 64 39 31' DEBUG: ErrorXML: CanonicalRequest: 'GET\n/testbucket/\ndelimiter=%2F\nhost:10.0.0.101:13808\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20201222T235206Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' DEBUG: ErrorXML: CanonicalRequestBytes: '47 45 54 0a 2f 74 65 73 74 62 75 63 6b 65 74 2f 0a 64 65 6c 69 6d 69 74 65 72 3d 25 32 46 0a 68 6f 73 74 3a 31 30 2e 30 2e 30 2e 31 30 31 3a 31 33 38 30 38 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 30 31 32 32 32 54 32 33 35 32 30 36 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35' ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method. ~~~ I'm now using admin/admin in these tests. ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack user list +----------------------------------+-------------------------+ | ID | Name | +----------------------------------+-------------------------+ | 6399c2d698c245c9889e3fe8714f480f | admin | | 0673a23379ad4b41a8e8974e3869536e | barbican | | 322db37a5c5c4c0caf8377b911e4ca77 | cinder | | efb7a5fc97c84cdea105f2e275045c70 | cinderv2 | | 1a8c44f3964f466480988b723f7f2070 | cinderv3 | | 64165631554743a688df88d5c65f11d7 | glance | | 2f681124227947df85d07394b3583b13 | heat | | cb6a83e1323646f9a74f428f2b679492 | heat_stack_domain_admin | | a8188906352b4aa1b0606903ba11e85d | heat-cfn | | 5889c50c65004c5aa346c74306c18884 | neutron | | 99a92ad0642e4d76baa4f8c67c778022 | nova | | 182fc9f4681f41d9be70af897d24d148 | octavia | | a6300260cc184cf19b4edc15cc4de898 | placement | | 9709c01b20304eef90ee22acc3c2c5ce | swift | | c60afb2d6d274d3684dc7853e039dc0c | demo_tempestconf | | 36d125f1a7a841b5afdf7522eeb8f315 | alt_demo_tempestconf | +----------------------------------+-------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack project list +----------------------------------+------------------------------------------------------------------+ | ID | Name | +----------------------------------+------------------------------------------------------------------+ | 0e125cf9fb2e49f2aba55b0e653b4d68 | f9669c61b3894287942ae6dce8d6a1da-2da09dc8-5703-4c7f-8293-e6ffb83 | | 4358ae205fd044cf80432694504b6505 | demo | | d3b27b23b8a64911882acb2d80040499 | alt_demo | | f66e4fc669b344fd9a8f2e280888e1ed | service | | f9669c61b3894287942ae6dce8d6a1da | admin | +----------------------------------+------------------------------------------------------------------+ ~~~
It does not reproduce for me as-is. Here's what happens: [zaitcev@suzdal ~]$ s3cmd -v -d ls s3://testbucket >run.run 2>&1 DEBUG: s3cmd version 2.1.0 DEBUG: ConfigParser: Reading file '/q/zaitcev/.s3cfg' DEBUG: ConfigParser: access_key->73...29_chars...f DEBUG: ConfigParser: access_token-> DEBUG: ConfigParser: add_encoding_exts-> DEBUG: ConfigParser: add_headers-> DEBUG: ConfigParser: bucket_location->US DEBUG: ConfigParser: ca_certs_file-> DEBUG: ConfigParser: cache_file-> DEBUG: ConfigParser: check_ssl_certificate->True DEBUG: ConfigParser: check_ssl_hostname->True DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com DEBUG: ConfigParser: connection_pooling->True DEBUG: ConfigParser: content_disposition-> DEBUG: ConfigParser: content_type-> DEBUG: ConfigParser: default_mime_type->binary/octet-stream DEBUG: ConfigParser: delay_updates->False DEBUG: ConfigParser: delete_after->False DEBUG: ConfigParser: delete_after_fetch->False DEBUG: ConfigParser: delete_removed->False DEBUG: ConfigParser: dry_run->False DEBUG: ConfigParser: enable_multipart->True DEBUG: ConfigParser: encoding->UTF-8 DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: expiry_date-> DEBUG: ConfigParser: expiry_days-> DEBUG: ConfigParser: expiry_prefix-> DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: force->False DEBUG: ConfigParser: get_continue->False DEBUG: ConfigParser: gpg_command->/usr/bin/gpg DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_passphrase->...-3_chars... DEBUG: ConfigParser: guess_mime_type->True DEBUG: ConfigParser: host_base->rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: host_bucket->rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: invalidate_default_index_on_cf->False DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True DEBUG: ConfigParser: invalidate_on_cf->False DEBUG: ConfigParser: kms_key-> DEBUG: ConfigParser: limit->-1 DEBUG: ConfigParser: limitrate->0 DEBUG: ConfigParser: list_md5->False DEBUG: ConfigParser: log_target_prefix-> DEBUG: ConfigParser: long_listing->False DEBUG: ConfigParser: max_delete->-1 DEBUG: ConfigParser: mime_type-> DEBUG: ConfigParser: multipart_chunk_size_mb->15 DEBUG: ConfigParser: multipart_max_chunks->10000 DEBUG: ConfigParser: preserve_attrs->True DEBUG: ConfigParser: progress_meter->True DEBUG: ConfigParser: proxy_host-> DEBUG: ConfigParser: proxy_port->0 DEBUG: ConfigParser: public_url_use_https->False DEBUG: ConfigParser: put_continue->False DEBUG: ConfigParser: recursive->False DEBUG: ConfigParser: recv_chunk->65536 DEBUG: ConfigParser: reduced_redundancy->False DEBUG: ConfigParser: requester_pays->False DEBUG: ConfigParser: restore_days->1 DEBUG: ConfigParser: restore_priority->Standard DEBUG: ConfigParser: secret_key->0f...29_chars...f DEBUG: ConfigParser: send_chunk->65536 DEBUG: ConfigParser: server_side_encryption->False DEBUG: ConfigParser: signature_v2->False DEBUG: ConfigParser: signurl_use_https->False DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->300 DEBUG: ConfigParser: stats->False DEBUG: ConfigParser: stop_on_error->False DEBUG: ConfigParser: storage_class-> DEBUG: ConfigParser: throttle_max->100 DEBUG: ConfigParser: upload_id-> DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_http_expect->False DEBUG: ConfigParser: use_https->True DEBUG: ConfigParser: use_mime_magic->True DEBUG: ConfigParser: verbosity->WARNING DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ DEBUG: ConfigParser: website_error-> DEBUG: ConfigParser: website_index->index.html DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Command: ls DEBUG: Bucket 's3://testbucket': DEBUG: CreateRequest: resource[uri]=/ DEBUG: ===== SEND Inner request to determine the bucket region ===== DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: canonical_headers = host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034042Z DEBUG: Canonical Request: GET /testbucket/ location= host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034042Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20210114T034042Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=9a0d5aae2923a36e9e6a0243c47dfdd9aacc6e5da8b5a0bd1b3ceaee07555e8a', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConnMan.get(): creating new connection: https://rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: Using ca_certs_file None DEBUG: httplib.HTTPSConnection() has both context and check_hostname DEBUG: non-proxied HTTPSConnection(rhev-a24c-01.os.lab.eng.bos.redhat.com, None) DEBUG: format_uri(): /testbucket/?location DEBUG: Sending request method_string='GET', uri='/testbucket/?location', headers={'x-amz-date': '20210114T034042Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=9a0d5aae2923a36e9e6a0243c47dfdd9aacc6e5da8b5a0bd1b3ceaee07555e8a', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://rhev-a24c-01.os.lab.eng.bos.redhat.com#1) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<LocationConstraint xmlns" b'="http://s3.amazonaws.com/doc/2006-03-01/"/>', 'headers': {'content-length': '108', 'content-type': 'application/xml', 'date': 'Thu, 14 Jan 2021 03:40:43 GMT', 'x-amz-id-2': 'tx9e862c825f394d028bc49-005fffbd3a', 'x-amz-request-id': 'tx9e862c825f394d028bc49-005fffbd3a', 'x-openstack-request-id': 'tx9e862c825f394d028bc49-005fffbd3a', 'x-trans-id': 'tx9e862c825f394d028bc49-005fffbd3a'}, 'reason': 'OK', 'status': 200} DEBUG: ===== SUCCESS Inner request to determine the bucket region ('us-east-1') ===== DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: canonical_headers = host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034043Z DEBUG: Canonical Request: GET /testbucket/ delimiter=%2F host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034043Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20210114T034043Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=6ab40a404f37366b38f66b201548c4f13e6a7dcaab45e8f4a76aa5ebc81c3717', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConnMan.get(): re-using connection: https://rhev-a24c-01.os.lab.eng.bos.redhat.com#1 DEBUG: format_uri(): /testbucket/?delimiter=%2F DEBUG: Sending request method_string='GET', uri='/testbucket/?delimiter=%2F', headers={'x-amz-date': '20210114T034043Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=6ab40a404f37366b38f66b201548c4f13e6a7dcaab45e8f4a76aa5ebc81c3717', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://rhev-a24c-01.os.lab.eng.bos.redhat.com#2) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<ListBucketResult xml" b'ns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>testbucket</Name>' b'<Prefix/><Marker/><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><Is' b'Truncated>false</IsTruncated></ListBucketResult>', 'headers': {'content-length': '244', 'content-type': 'application/xml', 'date': 'Thu, 14 Jan 2021 03:40:43 GMT', 'x-amz-id-2': 'tx18946827fbe444079ef74-005fffbd3b', 'x-amz-request-id': 'tx18946827fbe444079ef74-005fffbd3b', 'x-openstack-request-id': 'tx18946827fbe444079ef74-005fffbd3b', 'x-trans-id': 'tx18946827fbe444079ef74-005fffbd3b'}, 'reason': 'OK', 'status': 200} However! If I use host bucket, then nothing works at all, not even v2! DEBUG: s3cmd version 2.1.0 DEBUG: ConfigParser: Reading file '/q/zaitcev/.s3cfg' DEBUG: ConfigParser: access_key->73...29_chars...f DEBUG: ConfigParser: access_token-> DEBUG: ConfigParser: add_encoding_exts-> DEBUG: ConfigParser: add_headers-> DEBUG: ConfigParser: bucket_location->US DEBUG: ConfigParser: ca_certs_file-> DEBUG: ConfigParser: cache_file-> DEBUG: ConfigParser: check_ssl_certificate->True DEBUG: ConfigParser: check_ssl_hostname->True DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com DEBUG: ConfigParser: connection_pooling->True DEBUG: ConfigParser: content_disposition-> DEBUG: ConfigParser: content_type-> DEBUG: ConfigParser: default_mime_type->binary/octet-stream DEBUG: ConfigParser: delay_updates->False DEBUG: ConfigParser: delete_after->False DEBUG: ConfigParser: delete_after_fetch->False DEBUG: ConfigParser: delete_removed->False DEBUG: ConfigParser: dry_run->False DEBUG: ConfigParser: enable_multipart->True DEBUG: ConfigParser: encoding->UTF-8 DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: expiry_date-> DEBUG: ConfigParser: expiry_days-> DEBUG: ConfigParser: expiry_prefix-> DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: force->False DEBUG: ConfigParser: get_continue->False DEBUG: ConfigParser: gpg_command->/usr/bin/gpg DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_passphrase->...-3_chars... DEBUG: ConfigParser: guess_mime_type->True DEBUG: ConfigParser: host_base->rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: host_bucket->%(bucket)s.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: invalidate_default_index_on_cf->False DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True DEBUG: ConfigParser: invalidate_on_cf->False DEBUG: ConfigParser: kms_key-> DEBUG: ConfigParser: limit->-1 DEBUG: ConfigParser: limitrate->0 DEBUG: ConfigParser: list_md5->False DEBUG: ConfigParser: log_target_prefix-> DEBUG: ConfigParser: long_listing->False DEBUG: ConfigParser: max_delete->-1 DEBUG: ConfigParser: mime_type-> DEBUG: ConfigParser: multipart_chunk_size_mb->15 DEBUG: ConfigParser: multipart_max_chunks->10000 DEBUG: ConfigParser: preserve_attrs->True DEBUG: ConfigParser: progress_meter->True DEBUG: ConfigParser: proxy_host-> DEBUG: ConfigParser: proxy_port->0 DEBUG: ConfigParser: public_url_use_https->False DEBUG: ConfigParser: put_continue->False DEBUG: ConfigParser: recursive->False DEBUG: ConfigParser: recv_chunk->65536 DEBUG: ConfigParser: reduced_redundancy->False DEBUG: ConfigParser: requester_pays->False DEBUG: ConfigParser: restore_days->1 DEBUG: ConfigParser: restore_priority->Standard DEBUG: ConfigParser: secret_key->0f...29_chars...f DEBUG: ConfigParser: send_chunk->65536 DEBUG: ConfigParser: server_side_encryption->False DEBUG: ConfigParser: signature_v2->True DEBUG: ConfigParser: signurl_use_https->False DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->300 DEBUG: ConfigParser: stats->False DEBUG: ConfigParser: stop_on_error->False DEBUG: ConfigParser: storage_class-> DEBUG: ConfigParser: throttle_max->100 DEBUG: ConfigParser: upload_id-> DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_http_expect->False DEBUG: ConfigParser: use_https->True DEBUG: ConfigParser: use_mime_magic->True DEBUG: ConfigParser: verbosity->WARNING DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ DEBUG: ConfigParser: website_error-> DEBUG: ConfigParser: website_index->index.html DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Command: ls DEBUG: Bucket 's3://testbucket': DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v2 DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 14 Jan 2021 03:43:36 +0000\n/testbucket/' DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConnMan.get(): creating new connection: https://testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: Using ca_certs_file None DEBUG: httplib.HTTPSConnection() has both context and check_hostname DEBUG: non-proxied HTTPSConnection(testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com, None) DEBUG: format_uri(): /?delimiter=%2F DEBUG: Sending request method_string='GET', uri='/?delimiter=%2F', headers={'x-amz-date': 'Thu, 14 Jan 2021 03:43:36 +0000', 'Authorization': 'AWS 73302341691b406996ae08f68facdf4f:p874o40CqLlp0L5SVKAnts6VUNU='}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com#1) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>SignatureDoe" b'sNotMatch</Code><Message>The request signature we calculated does no' b't match the signature you provided. Check your key and signing metho' b'd.</Message><RequestId>tx6f1ac5ddd0f443b599efa-005fffbde8</RequestId' b'><AWSAccessKeyId>73302341691b406996ae08f68facdf4f</AWSAccessKeyId><S' b'tringToSign>GET\n\n\n\nx-amz-date:Thu, 14 Jan 2021 03:43:36 +000' b'0\n/</StringToSign><SignatureProvided>p874o40CqLlp0L5SVKAnts6VUNU' b'=</SignatureProvided><StringToSignBytes>47 45 54 0a 0a 0a 0a 78 2d 6' b'1 6d 7a 2d 64 61 74 65 3a 54 68 75 2c 20 31 34 20 4a 61 6e 20 32 30 ' b'32 31 20 30 33 3a 34 33 3a 33 36 20 2b 30 30 30 30 0a 2f</StringToSi' b'gnBytes></Error>', 'headers': {'content-type': 'application/xml', 'date': 'Thu, 14 Jan 2021 03:43:36 GMT', 'transfer-encoding': 'chunked', 'x-amz-id-2': 'tx6f1ac5ddd0f443b599efa-005fffbde8', 'x-amz-request-id': 'tx6f1ac5ddd0f443b599efa-005fffbde8', 'x-openstack-request-id': 'tx6f1ac5ddd0f443b599efa-005fffbde8', 'x-trans-id': 'tx6f1ac5ddd0f443b599efa-005fffbde8'}, 'reason': 'Forbidden', 'status': 403} DEBUG: S3Error: 403 (Forbidden) DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: x-amz-id-2: tx6f1ac5ddd0f443b599efa-005fffbde8 DEBUG: HttpHeader: x-amz-request-id: tx6f1ac5ddd0f443b599efa-005fffbde8 DEBUG: HttpHeader: x-trans-id: tx6f1ac5ddd0f443b599efa-005fffbde8 DEBUG: HttpHeader: x-openstack-request-id: tx6f1ac5ddd0f443b599efa-005fffbde8 DEBUG: HttpHeader: date: Thu, 14 Jan 2021 03:43:36 GMT DEBUG: HttpHeader: transfer-encoding: chunked DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch' DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.' DEBUG: ErrorXML: RequestId: 'tx6f1ac5ddd0f443b599efa-005fffbde8' DEBUG: ErrorXML: AWSAccessKeyId: '73302341691b406996ae08f68facdf4f' DEBUG: ErrorXML: StringToSign: 'GET\n\n\n\nx-amz-date:Thu, 14 Jan 2021 03:43:36 +0000\n/' DEBUG: ErrorXML: SignatureProvided: 'p874o40CqLlp0L5SVKAnts6VUNU=' DEBUG: ErrorXML: StringToSignBytes: '47 45 54 0a 0a 0a 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 54 68 75 2c 20 31 34 20 4a 61 6e 20 32 30 32 31 20 30 33 3a 34 33 3a 33 36 20 2b 30 30 30 30 0a 2f' ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method. But it works yet again if bucket is not specified (GET /): DEBUG: s3cmd version 2.1.0 DEBUG: ConfigParser: Reading file '/q/zaitcev/.s3cfg' DEBUG: ConfigParser: access_key->73...29_chars...f DEBUG: ConfigParser: access_token-> DEBUG: ConfigParser: add_encoding_exts-> DEBUG: ConfigParser: add_headers-> DEBUG: ConfigParser: bucket_location->US DEBUG: ConfigParser: ca_certs_file-> DEBUG: ConfigParser: cache_file-> DEBUG: ConfigParser: check_ssl_certificate->True DEBUG: ConfigParser: check_ssl_hostname->True DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com DEBUG: ConfigParser: connection_pooling->True DEBUG: ConfigParser: content_disposition-> DEBUG: ConfigParser: content_type-> DEBUG: ConfigParser: default_mime_type->binary/octet-stream DEBUG: ConfigParser: delay_updates->False DEBUG: ConfigParser: delete_after->False DEBUG: ConfigParser: delete_after_fetch->False DEBUG: ConfigParser: delete_removed->False DEBUG: ConfigParser: dry_run->False DEBUG: ConfigParser: enable_multipart->True DEBUG: ConfigParser: encoding->UTF-8 DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: expiry_date-> DEBUG: ConfigParser: expiry_days-> DEBUG: ConfigParser: expiry_prefix-> DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: force->False DEBUG: ConfigParser: get_continue->False DEBUG: ConfigParser: gpg_command->/usr/bin/gpg DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_passphrase->...-3_chars... DEBUG: ConfigParser: guess_mime_type->True DEBUG: ConfigParser: host_base->rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: host_bucket->%(bucket)s.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: invalidate_default_index_on_cf->False DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True DEBUG: ConfigParser: invalidate_on_cf->False DEBUG: ConfigParser: kms_key-> DEBUG: ConfigParser: limit->-1 DEBUG: ConfigParser: limitrate->0 DEBUG: ConfigParser: list_md5->False DEBUG: ConfigParser: log_target_prefix-> DEBUG: ConfigParser: long_listing->False DEBUG: ConfigParser: max_delete->-1 DEBUG: ConfigParser: mime_type-> DEBUG: ConfigParser: multipart_chunk_size_mb->15 DEBUG: ConfigParser: multipart_max_chunks->10000 DEBUG: ConfigParser: preserve_attrs->True DEBUG: ConfigParser: progress_meter->True DEBUG: ConfigParser: proxy_host-> DEBUG: ConfigParser: proxy_port->0 DEBUG: ConfigParser: public_url_use_https->False DEBUG: ConfigParser: put_continue->False DEBUG: ConfigParser: recursive->False DEBUG: ConfigParser: recv_chunk->65536 DEBUG: ConfigParser: reduced_redundancy->False DEBUG: ConfigParser: requester_pays->False DEBUG: ConfigParser: restore_days->1 DEBUG: ConfigParser: restore_priority->Standard DEBUG: ConfigParser: secret_key->0f...29_chars...f DEBUG: ConfigParser: send_chunk->65536 DEBUG: ConfigParser: server_side_encryption->False DEBUG: ConfigParser: signature_v2->False DEBUG: ConfigParser: signurl_use_https->False DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->300 DEBUG: ConfigParser: stats->False DEBUG: ConfigParser: stop_on_error->False DEBUG: ConfigParser: storage_class-> DEBUG: ConfigParser: throttle_max->100 DEBUG: ConfigParser: upload_id-> DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_http_expect->False DEBUG: ConfigParser: use_https->True DEBUG: ConfigParser: use_mime_magic->True DEBUG: ConfigParser: verbosity->WARNING DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ DEBUG: ConfigParser: website_error-> DEBUG: ConfigParser: website_index->index.html DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Command: ls DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(None): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: canonical_headers = host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034453Z DEBUG: Canonical Request: GET / host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034453Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20210114T034453Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=39c80b9f3a2c3273a040765dd1d8653c38c2539e4e8e91e543f6de69546b2378', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(None): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConnMan.get(): creating new connection: https://rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: Using ca_certs_file None DEBUG: httplib.HTTPSConnection() has both context and check_hostname DEBUG: non-proxied HTTPSConnection(rhev-a24c-01.os.lab.eng.bos.redhat.com, None) DEBUG: format_uri(): / DEBUG: Sending request method_string='GET', uri='/', headers={'x-amz-date': '20210114T034453Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=39c80b9f3a2c3273a040765dd1d8653c38c2539e4e8e91e543f6de69546b2378', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://rhev-a24c-01.os.lab.eng.bos.redhat.com#1) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>Authorizatio" b'nHeaderMalformed</Code><Message>The authorization header is malforme' b"d; the region 'US' is wrong; expecting 'us-east-1'</Message><Request" b'Id>txebb4dd463d6e440f9a59c-005fffbe36</RequestId><Region>us-east-1</' b'Region></Error>', 'headers': {'content-type': 'application/xml', 'date': 'Thu, 14 Jan 2021 03:44:54 GMT', 'transfer-encoding': 'chunked', 'x-amz-id-2': 'txebb4dd463d6e440f9a59c-005fffbe36', 'x-amz-request-id': 'txebb4dd463d6e440f9a59c-005fffbe36', 'x-openstack-request-id': 'txebb4dd463d6e440f9a59c-005fffbe36', 'x-trans-id': 'txebb4dd463d6e440f9a59c-005fffbe36'}, 'reason': 'Bad Request', 'status': 400} INFO: Forwarding request to us-east-1 DEBUG: Using signature v4 DEBUG: get_hostname(None): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: canonical_headers = host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034454Z DEBUG: Canonical Request: GET / host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210114T034454Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20210114T034454Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=686ec6d18c722293499cec41dd02c3faed4671fb5faa9aabbf8bb67c549dd453', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(None): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConnMan.get(): re-using connection: https://rhev-a24c-01.os.lab.eng.bos.redhat.com#1 DEBUG: format_uri(): / DEBUG: Sending request method_string='GET', uri='/', headers={'x-amz-date': '20210114T034454Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210114/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=686ec6d18c722293499cec41dd02c3faed4671fb5faa9aabbf8bb67c549dd453', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://rhev-a24c-01.os.lab.eng.bos.redhat.com#2) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<ListAllMyBucketsResult x" b'mlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>ksuserten:' b'ksuser</ID><DisplayName>ksuserten:ksuser</DisplayName></Owner><Bucke' b'ts><Bucket><Name>testbucket</Name><CreationDate>2009-02-03T16:45:09.' b'000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>', 'headers': {'content-length': '331', 'content-type': 'application/xml', 'date': 'Thu, 14 Jan 2021 03:44:54 GMT', 'x-amz-id-2': 'txc2e510b7e36f4370a761d-005fffbe36', 'x-amz-request-id': 'txc2e510b7e36f4370a761d-005fffbe36', 'x-openstack-request-id': 'txc2e510b7e36f4370a761d-005fffbe36', 'x-trans-id': 'txc2e510b7e36f4370a761d-005fffbe36'}, 'reason': 'OK', 'status': 200} 2009-02-03 16:45 s3://testbucket Something is seriously not right with the way we're dealing with signatures and HMAC. And, in the same time, this somehow changes between Takashi is seeing and I'm seeing.
Looks like I got it running. There was a regression with S3 v4 signatures right at the tip of the current tree, in 2.27.0-development. Here's how it looks, with V4 and host bucket: [zaitcev@suzdal ~]$ s3cmd -d ls s3://testbucket > x 2>&1 DEBUG: s3cmd version 2.1.0 DEBUG: ConfigParser: Reading file '/q/zaitcev/.s3cfg' DEBUG: ConfigParser: access_key->73...29_chars...f DEBUG: ConfigParser: access_token-> DEBUG: ConfigParser: add_headers-> DEBUG: ConfigParser: bucket_location->US DEBUG: ConfigParser: ca_certs_file-> DEBUG: ConfigParser: check_ssl_certificate->True DEBUG: ConfigParser: check_ssl_hostname->True DEBUG: ConfigParser: delay_updates->False DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: host_base->rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: host_bucket->%(bucket)s.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: long_listing->False DEBUG: ConfigParser: secret_key->0f...29_chars...f DEBUG: ConfigParser: signature_v2->False DEBUG: ConfigParser: signurl_use_https->False DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->300 DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_https->True DEBUG: ConfigParser: verbosity->WARNING DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Command: ls DEBUG: Bucket 's3://testbucket': DEBUG: CreateRequest: resource[uri]=/ DEBUG: ===== SEND Inner request to determine the bucket region ===== DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: canonical_headers = host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210126T055725Z DEBUG: Canonical Request: GET /testbucket/ location= host:rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210126T055725Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20210126T055725Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210126/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=3f26bfff72ae38b57d96d8d5e982359920136eef1b0b4ee1685154ac5115faf2', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConnMan.get(): creating new connection: https://rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: Using ca_certs_file None DEBUG: httplib.HTTPSConnection() has both context and check_hostname DEBUG: non-proxied HTTPSConnection(rhev-a24c-01.os.lab.eng.bos.redhat.com, None) DEBUG: format_uri(): /testbucket/?location DEBUG: Sending request method_string='GET', uri='/testbucket/?location', headers={'x-amz-date': '20210126T055725Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210126/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=3f26bfff72ae38b57d96d8d5e982359920136eef1b0b4ee1685154ac5115faf2', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://rhev-a24c-01.os.lab.eng.bos.redhat.com#1) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<LocationConstraint xmlns" b'="http://s3.amazonaws.com/doc/2006-03-01/"/>', 'headers': {'content-length': '108', 'content-type': 'application/xml', 'date': 'Tue, 26 Jan 2021 05:57:26 GMT', 'x-amz-id-2': 'tx9da4d790e69a4e92a359f-00600faf46', 'x-amz-request-id': 'tx9da4d790e69a4e92a359f-00600faf46', 'x-openstack-request-id': 'tx9da4d790e69a4e92a359f-00600faf46', 'x-trans-id': 'tx9da4d790e69a4e92a359f-00600faf46'}, 'reason': 'OK', 'status': 200} DEBUG: ===== SUCCESS Inner request to determine the bucket region ('us-east-1') ===== DEBUG: Using signature v4 DEBUG: get_hostname(testbucket): testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: canonical_headers = host:testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210126T055726Z DEBUG: Canonical Request: GET / delimiter=%2F host:testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210126T055726Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20210126T055726Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210126/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=9e3101d6582219cb704e5825694edf4cfb3a261b7a837984964a3a58c951bf64', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(testbucket): testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: ConnMan.get(): creating new connection: https://testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com DEBUG: httplib.HTTPSConnection() has both context and check_hostname DEBUG: non-proxied HTTPSConnection(testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com, None) DEBUG: format_uri(): /?delimiter=%2F DEBUG: Sending request method_string='GET', uri='/?delimiter=%2F', headers={'x-amz-date': '20210126T055726Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=73302341691b406996ae08f68facdf4f/20210126/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=9e3101d6582219cb704e5825694edf4cfb3a261b7a837984964a3a58c951bf64', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://testbucket.rhev-a24c-01.os.lab.eng.bos.redhat.com#1) DEBUG: Response: {'data': b"<?xml version='1.0' encoding='UTF-8'?>\n<ListAllMyBucketsResult x" b'mlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>ksuserten:' b'ksuser</ID><DisplayName>ksuserten:ksuser</DisplayName></Owner><Bucke' b'ts><Bucket><Name>testbucket</Name><CreationDate>2009-02-03T16:45:09.' b'000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>', 'headers': {'content-length': '331', 'content-type': 'application/xml', 'date': 'Tue, 26 Jan 2021 05:57:27 GMT', 'x-amz-id-2': 'txc923d8bfaebc4a95a0a67-00600faf47', 'x-amz-request-id': 'txc923d8bfaebc4a95a0a67-00600faf47', 'x-openstack-request-id': 'txc923d8bfaebc4a95a0a67-00600faf47', 'x-trans-id': 'txc923d8bfaebc4a95a0a67-00600faf47'}, 'reason': 'OK', 'status': 200}
What is the exact RPM version on the server where the error happens? Please let me know, I'll find the source and make an exact test patch.
Sorry I forgot to capture the version installed in the deployment. I'll try reproducing the same issue once I get a machine to set up a reproducer. I checked the same in 16.1.3 deployment but seems it is working well. One thing which is suspicious is that in the current deployment I don't have ssl enabled on public endpoints. However when I tested this previously and found the issue I had ssl enabled on public endpoints. I guess this might cause something wrong with signature verification but anyway I'll recheck that.
My cluster has SSL enabled and works with it (terminator is stunnel).