Description of problem: s3cmd info not working on Ceph 3.2 (cors policies) giving 500 Version-Release number of selected component (if applicable): Red Hat Ceph Storage 3.2 How reproducible: Always Works perfectly fine in RHCS 3.1 # ceph daemon client.rgw.`hostname -s` version {"version":"12.2.5-59.el7cp","release":"luminous","release_type":"stable"}[root@dell-per630-9 ~]# $ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1 s3://bucket1/ (bucket): Location: us-east-1 Payer: BucketOwner Expiration Rule: none Policy: none CORS: none ACL: Test User: FULL_CONTROL Not working in RHCS 3.2 - 12.2.8-52 or RHCS 3.2.z1 - 12.2.8-89 ===================================================================== From RHCS 3.2 it gives - 500 (Internal Server Error) $ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1 --debug DEBUG: s3cmd version 2.0.2 DEBUG: ConfigParser: Reading file '.s3cfg.rhcs.3' DEBUG: ConfigParser: access_key->1U...17_chars...B DEBUG: ConfigParser: access_token-> DEBUG: ConfigParser: add_encoding_exts-> DEBUG: ConfigParser: add_headers-> DEBUG: ConfigParser: bucket_location-> DEBUG: ConfigParser: ca_certs_file-> DEBUG: ConfigParser: cache_file-> DEBUG: ConfigParser: check_ssl_certificate->False DEBUG: ConfigParser: check_ssl_hostname->False DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com DEBUG: ConfigParser: default_mime_type->binary/octet-stream DEBUG: ConfigParser: delay_updates->False DEBUG: ConfigParser: delete_after->False DEBUG: ConfigParser: delete_after_fetch->False DEBUG: ConfigParser: delete_removed->False DEBUG: ConfigParser: dry_run->False DEBUG: ConfigParser: enable_multipart->True DEBUG: ConfigParser: encoding->UTF-8 DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: expiry_date-> DEBUG: ConfigParser: expiry_days-> DEBUG: ConfigParser: expiry_prefix-> DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: force->False DEBUG: ConfigParser: get_continue->False DEBUG: ConfigParser: gpg_command->/usr/bin/gpg DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_passphrase->...-3_chars... DEBUG: ConfigParser: guess_mime_type->True DEBUG: ConfigParser: host_base->dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: ConfigParser: host_bucket->%(bucket).dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: invalidate_default_index_on_cf->False DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True DEBUG: ConfigParser: invalidate_on_cf->False DEBUG: ConfigParser: kms_key-> DEBUG: ConfigParser: limitrate->0 DEBUG: ConfigParser: list_md5->False DEBUG: ConfigParser: log_target_prefix-> DEBUG: ConfigParser: long_listing->False DEBUG: ConfigParser: max_delete->-1 DEBUG: ConfigParser: mime_type-> DEBUG: ConfigParser: multipart_chunk_size_mb->5 DEBUG: ConfigParser: multipart_max_chunks->10000 DEBUG: ConfigParser: preserve_attrs->True DEBUG: ConfigParser: progress_meter->True DEBUG: ConfigParser: proxy_host-> DEBUG: ConfigParser: proxy_port->0 DEBUG: ConfigParser: put_continue->False DEBUG: ConfigParser: recursive->False DEBUG: ConfigParser: recv_chunk->65536 DEBUG: ConfigParser: reduced_redundancy->False DEBUG: ConfigParser: requester_pays->False DEBUG: ConfigParser: restore_days->1 DEBUG: ConfigParser: secret_key->rO...37_chars...x DEBUG: ConfigParser: send_chunk->65536 DEBUG: ConfigParser: server_side_encryption->False DEBUG: ConfigParser: signature_v2->False DEBUG: ConfigParser: simpledb_host->sdb.dell-per630-10.gsslab.pnq2.redhat.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->300 DEBUG: ConfigParser: stats->False DEBUG: ConfigParser: stop_on_error->False DEBUG: ConfigParser: storage_class-> DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_https->False DEBUG: ConfigParser: use_mime_magic->True DEBUG: ConfigParser: verbosity->WARNING DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.dell-per630-10.gsslab.pnq2.redhat.com/ DEBUG: ConfigParser: website_error-> DEBUG: ConfigParser: website_index->index.html DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Command: info DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232445Z DEBUG: Canonical Request: GET /bucket1/ location= host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232445Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232445Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45def18f2fafb55e98dcaa246802b1f52af98e17eb169dcf38903c79ca32f2d5', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: ConnMan.get(): creating new connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: non-proxied HTTPConnection(dell-per630-9.gsslab.pnq2.redhat.com, 8080) DEBUG: format_uri(): /bucket1/?location DEBUG: Sending request method_string='GET', uri='/bucket1/?location', headers={'x-amz-date': '20190314T232445Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45def18f2fafb55e98dcaa246802b1f52af98e17eb169dcf38903c79ca32f2d5', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#1) DEBUG: Response: {'data': b'<?xml version="1.0" encoding="UTF-8"?><LocationConstraint xmlns="htt' b'p://s3.amazonaws.com/doc/2006-03-01/"></LocationConstraint>', 'headers': {'content-length': '127', 'date': 'Thu, 14 Mar 2019 23:24:45 GMT', 'x-amz-request-id': 'tx000000000000000000001-005c8ae2bd-aebea-default'}, 'reason': 'OK', 'status': 200} DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232446Z DEBUG: Canonical Request: GET /bucket1/ requestPayment= host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232446Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232446Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=e6a64cb80d27376926de0cf829be15f39c608550cfa4238119e90e62fd785b80', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#1 DEBUG: format_uri(): /bucket1/?requestPayment DEBUG: Sending request method_string='GET', uri='/bucket1/?requestPayment', headers={'x-amz-date': '20190314T232446Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=e6a64cb80d27376926de0cf829be15f39c608550cfa4238119e90e62fd785b80', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#2) DEBUG: Response: {'data': b'<?xml version="1.0" encoding="UTF-8"?><RequestPaymentConfiguration x' b'mlns="http://s3.amazonaws.com/doc/2006-03-01/"><Payer>BucketOwner</P' b'ayer></RequestPaymentConfiguration>', 'headers': {'content-length': '171', 'content-type': 'application/xml', 'date': 'Thu, 14 Mar 2019 23:24:46 GMT', 'x-amz-request-id': 'tx000000000000000000002-005c8ae2be-aebea-default'}, 'reason': 'OK', 'status': 200} s3://bucket1/ (bucket): Location: us-east-1 Payer: BucketOwner DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232447Z DEBUG: Canonical Request: GET /bucket1/ lifecycle= host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232447Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232447Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=8dfc3aa583ff9b38a26a6e7d12a77dc5c49ac663c1f86df87519eac38ee882cf', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#2 DEBUG: format_uri(): /bucket1/?lifecycle DEBUG: Sending request method_string='GET', uri='/bucket1/?lifecycle', headers={'x-amz-date': '20190314T232447Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=8dfc3aa583ff9b38a26a6e7d12a77dc5c49ac663c1f86df87519eac38ee882cf', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#3) DEBUG: Response: {'data': b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchLifecycleCo' b'nfiguration</Code><BucketName>bucket1</BucketName><RequestId>tx00000' b'0000000000000003-005c8ae2bf-aebea-default</RequestId><HostId>aebea-d' b'efault-default</HostId></Error>', 'headers': {'accept-ranges': 'bytes', 'content-length': '235', 'content-type': 'application/xml', 'date': 'Thu, 14 Mar 2019 23:24:47 GMT', 'x-amz-request-id': 'tx000000000000000000003-005c8ae2bf-aebea-default'}, 'reason': 'Not Found', 'status': 404} DEBUG: S3Error: 404 (Not Found) DEBUG: HttpHeader: content-length: 235 DEBUG: HttpHeader: x-amz-request-id: tx000000000000000000003-005c8ae2bf-aebea-default DEBUG: HttpHeader: accept-ranges: bytes DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: date: Thu, 14 Mar 2019 23:24:47 GMT DEBUG: ErrorXML: Code: 'NoSuchLifecycleConfiguration' DEBUG: ErrorXML: BucketName: 'bucket1' DEBUG: ErrorXML: RequestId: 'tx000000000000000000003-005c8ae2bf-aebea-default' DEBUG: ErrorXML: HostId: 'aebea-default-default' DEBUG: Could not get /?lifecycle - lifecycle probably not configured for this bucket Expiration Rule: none DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232448Z DEBUG: Canonical Request: GET /bucket1/ policy= host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232448Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232448Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=b18e64560662680d2bc52875c3f7fb6c3084c51101e1ba5e24fc33d77e121274', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#3 DEBUG: format_uri(): /bucket1/?policy DEBUG: Sending request method_string='GET', uri='/bucket1/?policy', headers={'x-amz-date': '20190314T232448Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=b18e64560662680d2bc52875c3f7fb6c3084c51101e1ba5e24fc33d77e121274', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#4) DEBUG: Response: {'data': b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucketPolic' b'y</Code><Message>The bucket policy does not exist</Message><BucketNa' b'me>bucket1</BucketName><RequestId>tx000000000000000000004-005c8ae2c0' b'-aebea-default</RequestId><HostId>aebea-default-default</HostId></Er' b'ror>', 'headers': {'accept-ranges': 'bytes', 'content-length': '276', 'content-type': 'application/xml', 'date': 'Thu, 14 Mar 2019 23:24:48 GMT', 'x-amz-request-id': 'tx000000000000000000004-005c8ae2c0-aebea-default'}, 'reason': 'Not Found', 'status': 404} DEBUG: S3Error: 404 (Not Found) DEBUG: HttpHeader: content-length: 276 DEBUG: HttpHeader: x-amz-request-id: tx000000000000000000004-005c8ae2c0-aebea-default DEBUG: HttpHeader: accept-ranges: bytes DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: date: Thu, 14 Mar 2019 23:24:48 GMT DEBUG: ErrorXML: Code: 'NoSuchBucketPolicy' DEBUG: ErrorXML: Message: 'The bucket policy does not exist' DEBUG: ErrorXML: BucketName: 'bucket1' DEBUG: ErrorXML: RequestId: 'tx000000000000000000004-005c8ae2c0-aebea-default' DEBUG: ErrorXML: HostId: 'aebea-default-default' Policy: none DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v4 DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232449Z DEBUG: Canonical Request: GET /bucket1/ cors= host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232449Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ---------------------- DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232449Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'} DEBUG: Processing request, please wait... DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#4 DEBUG: format_uri(): /bucket1/?cors DEBUG: Sending request method_string='GET', uri='/bucket1/?cors', headers={'x-amz-date': '20190314T232449Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#5) DEBUG: Response: {'data': b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>UnknownError</Cod' b'e><BucketName>bucket1</BucketName><RequestId>tx000000000000000000005' b'-005c8ae2c1-aebea-default</RequestId><HostId>aebea-default-default</' b'HostId></Error>', 'headers': {'accept-ranges': 'bytes', 'content-length': '219', 'content-type': 'application/xml', 'date': 'Thu, 14 Mar 2019 23:24:50 GMT', 'x-amz-request-id': 'tx000000000000000000005-005c8ae2c1-aebea-default'}, 'reason': 'Internal Server Error', 'status': 500} DEBUG: S3Error: 500 (Internal Server Error) DEBUG: HttpHeader: content-length: 219 DEBUG: HttpHeader: x-amz-request-id: tx000000000000000000005-005c8ae2c1-aebea-default DEBUG: HttpHeader: accept-ranges: bytes DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: date: Thu, 14 Mar 2019 23:24:50 GMT DEBUG: ErrorXML: Code: 'UnknownError' DEBUG: ErrorXML: BucketName: 'bucket1' DEBUG: ErrorXML: RequestId: 'tx000000000000000000005-005c8ae2c1-aebea-default' DEBUG: ErrorXML: HostId: 'aebea-default-default' WARNING: Retrying failed request: /?cors (500 (UnknownError)) WARNING: Waiting 3 sec... DEBUG: Using signature v4 DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080 DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232453Z 2019-03-15 04:54:49.590850 7f4688e3e700 20 CONTENT_LENGTH=0 2019-03-15 04:54:49.590862 7f4688e3e700 20 HTTP_ACCEPT_ENCODING=identity 2019-03-15 04:54:49.590864 7f4688e3e700 20 HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee 2019-03-15 04:54:49.590868 7f4688e3e700 20 HTTP_HOST=dell-per630-9.gsslab.pnq2.redhat.com:8080 2019-03-15 04:54:49.590870 7f4688e3e700 20 HTTP_VERSION=1.1 2019-03-15 04:54:49.590872 7f4688e3e700 20 HTTP_X_AMZ_CONTENT_SHA256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2019-03-15 04:54:49.590875 7f4688e3e700 20 HTTP_X_AMZ_DATE=20190314T232449Z 2019-03-15 04:54:49.590877 7f4688e3e700 20 QUERY_STRING=cors 2019-03-15 04:54:49.590882 7f4688e3e700 20 REMOTE_ADDR=10.3.116.162 2019-03-15 04:54:49.590884 7f4688e3e700 20 REQUEST_METHOD=GET 2019-03-15 04:54:49.590885 7f4688e3e700 20 REQUEST_URI=/bucket1/ 2019-03-15 04:54:49.590886 7f4688e3e700 20 SCRIPT_URI=/bucket1/ 2019-03-15 04:54:49.590888 7f4688e3e700 20 SERVER_PORT=8080 2019-03-15 04:54:49.590890 7f4688e3e700 1 ====== starting new request req=0x7f4688e37f90 ===== 2019-03-15 04:54:49.590919 7f4688e3e700 2 req 5:0.000028::GET /bucket1/::initializing for trans_id = tx000000000000000000005-005c8ae2c1-aebea-default 2019-03-15 04:54:49.590929 7f4688e3e700 10 rgw api priority: s3=6 s3website=5 2019-03-15 04:54:49.590931 7f4688e3e700 10 host=dell-per630-9.gsslab.pnq2.redhat.com 2019-03-15 04:54:49.590933 7f4688e3e700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2019-03-15 04:54:49.590935 7f4688e3e700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bucket1/ 2019-03-15 04:54:49.590946 7f4688e3e700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 2019-03-15 04:54:49.590950 7f4688e3e700 10 meta>> HTTP_X_AMZ_DATE 2019-03-15 04:54:49.590955 7f4688e3e700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2019-03-15 04:54:49.590959 7f4688e3e700 10 x>> x-amz-date:20190314T232449Z 2019-03-15 04:54:49.590977 7f4688e3e700 20 get_handler handler=25RGWHandler_REST_Bucket_S3 2019-03-15 04:54:49.590981 7f4688e3e700 10 handler=25RGWHandler_REST_Bucket_S3 2019-03-15 04:54:49.590984 7f4688e3e700 2 req 5:0.000094:s3:GET /bucket1/::getting op 0 2019-03-15 04:54:49.590989 7f4688e3e700 10 op=22RGWGetCORS_ObjStore_S3 2019-03-15 04:54:49.590991 7f4688e3e700 2 req 5:0.000101:s3:GET /bucket1/:get_cors:verifying requester 2019-03-15 04:54:49.590993 7f4688e3e700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2019-03-15 04:54:49.590995 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2019-03-15 04:54:49.590998 7f4688e3e700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2019-03-15 04:54:49.590999 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy 2019-03-15 04:54:49.591001 7f4688e3e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine 2019-03-15 04:54:49.591012 7f4688e3e700 10 v4 signature format = 758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee 2019-03-15 04:54:49.591017 7f4688e3e700 10 v4 credential format = 1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request 2019-03-15 04:54:49.591019 7f4688e3e700 10 access key id = 1UOOQZ8213EP5UT4AXJB 2019-03-15 04:54:49.591021 7f4688e3e700 10 credential scope = 20190314//s3/aws4_request 2019-03-15 04:54:49.591043 7f4688e3e700 10 canonical headers format = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232449Z 2019-03-15 04:54:49.591049 7f4688e3e700 10 payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2019-03-15 04:54:49.591074 7f4688e3e700 10 canonical request = GET /bucket1/ cors= host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232449Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2019-03-15 04:54:49.591075 7f4688e3e700 10 canonical request hash = 65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e 2019-03-15 04:54:49.591115 7f4688e3e700 10 string to sign = AWS4-HMAC-SHA256 20190314T232449Z 20190314//s3/aws4_request 65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e 2019-03-15 04:54:49.591143 7f4688e3e700 20 sending request to http://10.74.130.128:5000/v3/auth/tokens 2019-03-15 04:54:50.062414 7f4688e3e700 20 sending request to http://10.74.130.128:5000/v3/s3tokens 2019-03-15 04:54:50.079774 7f4688e3e700 20 rgw::auth::keystone::EC2Engine denied with reason=-2028 2019-03-15 04:54:50.079787 7f4688e3e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028 2019-03-15 04:54:50.079790 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine 2019-03-15 04:54:50.079802 7f4688e3e700 10 v4 signature format = 758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee 2019-03-15 04:54:50.079808 7f4688e3e700 10 v4 credential format = 1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request 2019-03-15 04:54:50.079810 7f4688e3e700 10 access key id = 1UOOQZ8213EP5UT4AXJB 2019-03-15 04:54:50.079811 7f4688e3e700 10 credential scope = 20190314//s3/aws4_request 2019-03-15 04:54:50.079835 7f4688e3e700 10 canonical headers format = host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232449Z 2019-03-15 04:54:50.079842 7f4688e3e700 10 payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2019-03-15 04:54:50.079866 7f4688e3e700 10 canonical request = GET /bucket1/ cors= host:dell-per630-9.gsslab.pnq2.redhat.com:8080 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20190314T232449Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2019-03-15 04:54:50.079868 7f4688e3e700 10 canonical request hash = 65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e 2019-03-15 04:54:50.079926 7f4688e3e700 10 string to sign = AWS4-HMAC-SHA256 20190314T232449Z 20190314//s3/aws4_request 65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e 2019-03-15 04:54:50.080002 7f4688e3e700 10 date_k = 0b6d2c2533a022d37edb1e795d38732280e0e5cc9e702603ec65f441073e609f 2019-03-15 04:54:50.080013 7f4688e3e700 10 region_k = b986c8f42c32a79528fd4c677155515f3acf588cc8209409508a0a53b302ff6c 2019-03-15 04:54:50.080018 7f4688e3e700 10 service_k = eabd9a65d56b942d90090e34b53589db933f17dc11aba8f053efc8a78a3d5fe2 2019-03-15 04:54:50.080024 7f4688e3e700 10 signing_k = 692aaf5fa4a688f5320e1b019983da0b0f3c9a8ef50a2b758f7618f35478da9c 2019-03-15 04:54:50.080048 7f4688e3e700 10 generated signature = 758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee 2019-03-15 04:54:50.080051 7f4688e3e700 15 string_to_sign=AWS4-HMAC-SHA256 20190314T232449Z 20190314//s3/aws4_request 65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e 2019-03-15 04:54:50.080061 7f4688e3e700 15 server signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee 2019-03-15 04:54:50.080062 7f4688e3e700 15 client signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee 2019-03-15 04:54:50.080063 7f4688e3e700 15 compare=0 2019-03-15 04:54:50.080068 7f4688e3e700 20 rgw::auth::s3::LocalEngine granted access 2019-03-15 04:54:50.080070 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy granted access 2019-03-15 04:54:50.080073 7f4688e3e700 2 req 5:0.489183:s3:GET /bucket1/:get_cors:normalizing buckets and tenants 2019-03-15 04:54:50.080076 7f4688e3e700 10 s->object=<NULL> s->bucket=bucket1 2019-03-15 04:54:50.080079 7f4688e3e700 2 req 5:0.489189:s3:GET /bucket1/:get_cors:init permissions 2019-03-15 04:54:50.080096 7f4688e3e700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>testuser</ID><DisplayName>Test User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>testuser</ID><DisplayName>Test User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2019-03-15 04:54:50.080105 7f4688e3e700 20 get_system_obj_state: rctx=0x7f4688e365f0 obj=default.rgw.meta:users.uid:testuser state=0x5608431c4040 s->prefetch_data=0 2019-03-15 04:54:50.080111 7f4688e3e700 10 cache get: name=default.rgw.meta+users.uid+testuser : hit (requested=0x6, cached=0x17) 2019-03-15 04:54:50.080115 7f4688e3e700 20 get_system_obj_state: s->obj_tag was set empty 2019-03-15 04:54:50.080116 7f4688e3e700 20 Read xattr: user.rgw.idtag 2019-03-15 04:54:50.080131 7f4688e3e700 2 req 5:0.489241:s3:GET /bucket1/:get_cors:recalculating target 2019-03-15 04:54:50.080133 7f4688e3e700 2 req 5:0.489243:s3:GET /bucket1/:get_cors:reading permissions 2019-03-15 04:54:50.080134 7f4688e3e700 2 req 5:0.489245:s3:GET /bucket1/:get_cors:init op 2019-03-15 04:54:50.080136 7f4688e3e700 2 req 5:0.489246:s3:GET /bucket1/:get_cors:verifying op mask 2019-03-15 04:54:50.080137 7f4688e3e700 20 required_mask= 1 user.op_mask=7 2019-03-15 04:54:50.080139 7f4688e3e700 2 req 5:0.489248:s3:GET /bucket1/:get_cors:verifying op permissions 2019-03-15 04:54:50.080141 7f4688e3e700 2 req 5:0.489252:s3:GET /bucket1/:get_cors:verifying op params 2019-03-15 04:54:50.080142 7f4688e3e700 2 req 5:0.489253:s3:GET /bucket1/:get_cors:pre-executing 2019-03-15 04:54:50.080143 7f4688e3e700 2 req 5:0.489254:s3:GET /bucket1/:get_cors:executing 2019-03-15 04:54:50.080146 7f4688e3e700 20 no CORS configuration attr found 2019-03-15 04:54:50.080147 7f4688e3e700 2 No CORS configuration set yet for this bucket 2019-03-15 04:54:50.080147 7f4688e3e700 2 req 5:0.489258:s3:GET /bucket1/:get_cors:completing 2019-03-15 04:54:50.080150 7f4688e3e700 0 WARNING: set_req_state_err err_no=2044 resorting to 500 2019-03-15 04:54:50.080305 7f4688e3e700 2 req 5:0.489415:s3:GET /bucket1/:get_cors:op status=-2 2019-03-15 04:54:50.080307 7f4688e3e700 2 req 5:0.489417:s3:GET /bucket1/:get_cors:http status=500 2019-03-15 04:54:50.080309 7f4688e3e700 1 ====== req done req=0x7f4688e37f90 op status=-2 http_status=500 ====== 2019-03-15 04:54:50.080320 7f4688e3e700 20 process_request() returned -2044 2019-03-15 04:54:50.080340 7f4688e3e700 1 civetweb: 0x5608429b0000: 10.3.116.162 - - [15/Mar/2019:04:54:45 +0530] "GET /bucket1/?cors HTTP/1.1" 500 0 - -
I did look into what was changed in between 12.2.5 which is 3.1 and 12.2.8 which is 3.2 but I see only one change related to CORS which is - http://tracker.ceph.com/issues/24810 but looks like it is not related. As I went ahead and tested upstream 12.2.8 with vstart cluster and it is working fine at least I am 100% sure it came something in downstream custom backport we did for RGW on top on upstream v12.2.8. # cat build/out/rgw.j.log |less 2019-03-16 01:30:54.034491 7fa870a2bfc0 0 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process radosgw, pid 42623 2019-03-16 01:30:54.035183 7fa870a2bfc0 0 pidfile_write: ignore empty --pid-file $ s3cmd info s3://bucket1 s3://bucket1/ (bucket): Location: us-east-1 Payer: BucketOwner Expiration Rule: none Policy: none CORS: none ACL: M. Tester: FULL_CONTROL We need to find out which backports we did on top v12.2.8 and which one is impacting indirectly CORS output.
2019-03-15 04:54:50.080146 7f4688e3e700 20 no CORS configuration attr found 2019-03-15 04:54:50.080147 7f4688e3e700 2 No CORS configuration set yet for this bucket I was checking the code around these logs but no luck identify what changes in CORS code could be causing as only change I saw which I mentioned in comment#1 and it is present in upstream v12.2.8 and in this version it is working it means this gives +1 that this change is not causing the issue. From RHCS 3.1 - 12.2.5-59.el7cp $ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1 s3://bucket1/ (bucket): Location: us-east-1 Payer: BucketOwner Expiration Rule: none Policy: none CORS: none ACL: Test User: FULL_CONTROL From RHCS 3.2.z1 - 12.2.8-89.el7cp $ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1 s3://bucket1/ (bucket): Location: us-east-1 Payer: BucketOwner Expiration Rule: none Policy: none WARNING: Retrying failed request: /?cors (500 (UnknownError)) WARNING: Waiting 3 sec... WARNING: Retrying failed request: /?cors (500 (UnknownError)) WARNING: Waiting 6 sec...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:0911