Bug 1689410 - s3cmd info not working on Ceph 3.2 (cors policies) giving 500 (Internal Server Error)
Summary: s3cmd info not working on Ceph 3.2 (cors policies) giving 500 (Internal Serve...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 3.2
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: z2
: 3.2
Assignee: Matt Benjamin (redhat)
QA Contact: ceph-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks: 1629656
TreeView+ depends on / blocked
 
Reported: 2019-03-15 20:19 UTC by Vikhyat Umrao
Modified: 2019-11-11 09:48 UTC (History)
10 users (show)

Fixed In Version: RHEL: ceph-12.2.8-98.el7cp Ubuntu: ceph_12.2.8-83redhat1
Doc Type: Bug Fix
Doc Text:
.The RGW server no longer returns an incorrect S3 error code `NoSuchKey` when asked to return non-existent `CORS` rules Previously, the Ceph Object Gateway (RGW) sever would return an incorrect S3 error code `NoSuchKey` when asked to return non-existent `CORS` rules. This caused the `s3cmd` tool and other programs to misbehave. With this update, the RGW server now returns `NoSuchCORSConfiguration` for this case, and the `s3cmd` tool and other programs that expect this error behave correctly.
Clone Of:
Environment:
Last Closed: 2019-04-30 15:57:08 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 26964 0 None None None 2019-03-18 11:16:07 UTC
Red Hat Product Errata RHSA-2019:0911 0 None None None 2019-04-30 15:57:23 UTC

Description Vikhyat Umrao 2019-03-15 20:19:14 UTC 
Description of problem:
s3cmd info not working on Ceph 3.2 (cors policies) giving 500 


Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 3.2

How reproducible:
Always

Works perfectly fine in RHCS 3.1

# ceph daemon client.rgw.`hostname -s` version
{"version":"12.2.5-59.el7cp","release":"luminous","release_type":"stable"}[root@dell-per630-9 ~]# 


$ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1
s3://bucket1/ (bucket):
   Location:  us-east-1
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    none
   CORS:      none
   ACL:       Test User: FULL_CONTROL


Not working in RHCS 3.2 - 12.2.8-52 or RHCS 3.2.z1 - 12.2.8-89
=====================================================================

From RHCS 3.2 it gives - 500 (Internal Server Error)

$ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1 --debug
DEBUG: s3cmd version 2.0.2
DEBUG: ConfigParser: Reading file '.s3cfg.rhcs.3'
DEBUG: ConfigParser: access_key->1U...17_chars...B
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->
DEBUG: ConfigParser: ca_certs_file->
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: check_ssl_certificate->False
DEBUG: ConfigParser: check_ssl_hostname->False
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: expiry_date->
DEBUG: ConfigParser: expiry_days->
DEBUG: ConfigParser: expiry_prefix->
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: ConfigParser: host_bucket->%(bucket).dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: kms_key->
DEBUG: ConfigParser: limitrate->0
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: long_listing->False
DEBUG: ConfigParser: max_delete->-1
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->5
DEBUG: ConfigParser: multipart_max_chunks->10000
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: put_continue->False
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->65536
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: requester_pays->False
DEBUG: ConfigParser: restore_days->1
DEBUG: ConfigParser: secret_key->rO...37_chars...x
DEBUG: ConfigParser: send_chunk->65536
DEBUG: ConfigParser: server_side_encryption->False
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: simpledb_host->sdb.dell-per630-10.gsslab.pnq2.redhat.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: stats->False
DEBUG: ConfigParser: stop_on_error->False
DEBUG: ConfigParser: storage_class->
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: use_mime_magic->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.dell-per630-10.gsslab.pnq2.redhat.com/
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file -> 
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Command: info
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232445Z

DEBUG: Canonical Request:
GET
/bucket1/
location=
host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232445Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232445Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45def18f2fafb55e98dcaa246802b1f52af98e17eb169dcf38903c79ca32f2d5', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: ConnMan.get(): creating new connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: non-proxied HTTPConnection(dell-per630-9.gsslab.pnq2.redhat.com, 8080)
DEBUG: format_uri(): /bucket1/?location
DEBUG: Sending request method_string='GET', uri='/bucket1/?location', headers={'x-amz-date': '20190314T232445Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=45def18f2fafb55e98dcaa246802b1f52af98e17eb169dcf38903c79ca32f2d5', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#1)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><LocationConstraint xmlns="htt'
         b'p://s3.amazonaws.com/doc/2006-03-01/"></LocationConstraint>',
 'headers': {'content-length': '127',
             'date': 'Thu, 14 Mar 2019 23:24:45 GMT',
             'x-amz-request-id': 'tx000000000000000000001-005c8ae2bd-aebea-default'},
 'reason': 'OK',
 'status': 200}
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232446Z

DEBUG: Canonical Request:
GET
/bucket1/
requestPayment=
host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232446Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232446Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=e6a64cb80d27376926de0cf829be15f39c608550cfa4238119e90e62fd785b80', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#1
DEBUG: format_uri(): /bucket1/?requestPayment
DEBUG: Sending request method_string='GET', uri='/bucket1/?requestPayment', headers={'x-amz-date': '20190314T232446Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=e6a64cb80d27376926de0cf829be15f39c608550cfa4238119e90e62fd785b80', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#2)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><RequestPaymentConfiguration x'
         b'mlns="http://s3.amazonaws.com/doc/2006-03-01/"><Payer>BucketOwner</P'
         b'ayer></RequestPaymentConfiguration>',
 'headers': {'content-length': '171',
             'content-type': 'application/xml',
             'date': 'Thu, 14 Mar 2019 23:24:46 GMT',
             'x-amz-request-id': 'tx000000000000000000002-005c8ae2be-aebea-default'},
 'reason': 'OK',
 'status': 200}
s3://bucket1/ (bucket):
   Location:  us-east-1
   Payer:     BucketOwner
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232447Z

DEBUG: Canonical Request:
GET
/bucket1/
lifecycle=
host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232447Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232447Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=8dfc3aa583ff9b38a26a6e7d12a77dc5c49ac663c1f86df87519eac38ee882cf', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#2
DEBUG: format_uri(): /bucket1/?lifecycle
DEBUG: Sending request method_string='GET', uri='/bucket1/?lifecycle', headers={'x-amz-date': '20190314T232447Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=8dfc3aa583ff9b38a26a6e7d12a77dc5c49ac663c1f86df87519eac38ee882cf', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#3)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchLifecycleCo'
         b'nfiguration</Code><BucketName>bucket1</BucketName><RequestId>tx00000'
         b'0000000000000003-005c8ae2bf-aebea-default</RequestId><HostId>aebea-d'
         b'efault-default</HostId></Error>',
 'headers': {'accept-ranges': 'bytes',
             'content-length': '235',
             'content-type': 'application/xml',
             'date': 'Thu, 14 Mar 2019 23:24:47 GMT',
             'x-amz-request-id': 'tx000000000000000000003-005c8ae2bf-aebea-default'},
 'reason': 'Not Found',
 'status': 404}
DEBUG: S3Error: 404 (Not Found)
DEBUG: HttpHeader: content-length: 235
DEBUG: HttpHeader: x-amz-request-id: tx000000000000000000003-005c8ae2bf-aebea-default
DEBUG: HttpHeader: accept-ranges: bytes
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: date: Thu, 14 Mar 2019 23:24:47 GMT
DEBUG: ErrorXML: Code: 'NoSuchLifecycleConfiguration'
DEBUG: ErrorXML: BucketName: 'bucket1'
DEBUG: ErrorXML: RequestId: 'tx000000000000000000003-005c8ae2bf-aebea-default'
DEBUG: ErrorXML: HostId: 'aebea-default-default'
DEBUG: Could not get /?lifecycle - lifecycle probably not configured for this bucket
   Expiration Rule: none
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232448Z

DEBUG: Canonical Request:
GET
/bucket1/
policy=
host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232448Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232448Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=b18e64560662680d2bc52875c3f7fb6c3084c51101e1ba5e24fc33d77e121274', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#3
DEBUG: format_uri(): /bucket1/?policy
DEBUG: Sending request method_string='GET', uri='/bucket1/?policy', headers={'x-amz-date': '20190314T232448Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=b18e64560662680d2bc52875c3f7fb6c3084c51101e1ba5e24fc33d77e121274', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#4)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucketPolic'
         b'y</Code><Message>The bucket policy does not exist</Message><BucketNa'
         b'me>bucket1</BucketName><RequestId>tx000000000000000000004-005c8ae2c0'
         b'-aebea-default</RequestId><HostId>aebea-default-default</HostId></Er'
         b'ror>',
 'headers': {'accept-ranges': 'bytes',
             'content-length': '276',
             'content-type': 'application/xml',
             'date': 'Thu, 14 Mar 2019 23:24:48 GMT',
             'x-amz-request-id': 'tx000000000000000000004-005c8ae2c0-aebea-default'},
 'reason': 'Not Found',
 'status': 404}
DEBUG: S3Error: 404 (Not Found)
DEBUG: HttpHeader: content-length: 276
DEBUG: HttpHeader: x-amz-request-id: tx000000000000000000004-005c8ae2c0-aebea-default
DEBUG: HttpHeader: accept-ranges: bytes
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: date: Thu, 14 Mar 2019 23:24:48 GMT
DEBUG: ErrorXML: Code: 'NoSuchBucketPolicy'
DEBUG: ErrorXML: Message: 'The bucket policy does not exist'
DEBUG: ErrorXML: BucketName: 'bucket1'
DEBUG: ErrorXML: RequestId: 'tx000000000000000000004-005c8ae2c0-aebea-default'
DEBUG: ErrorXML: HostId: 'aebea-default-default'
   Policy:    none
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232449Z

DEBUG: Canonical Request:
GET
/bucket1/
cors=
host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232449Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-date': '20190314T232449Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: ConnMan.get(): re-using connection: http://dell-per630-9.gsslab.pnq2.redhat.com:8080#4
DEBUG: format_uri(): /bucket1/?cors
DEBUG: Sending request method_string='GET', uri='/bucket1/?cors', headers={'x-amz-date': '20190314T232449Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://dell-per630-9.gsslab.pnq2.redhat.com:8080#5)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>UnknownError</Cod'
         b'e><BucketName>bucket1</BucketName><RequestId>tx000000000000000000005'
         b'-005c8ae2c1-aebea-default</RequestId><HostId>aebea-default-default</'
         b'HostId></Error>',
 'headers': {'accept-ranges': 'bytes',
             'content-length': '219',
             'content-type': 'application/xml',
             'date': 'Thu, 14 Mar 2019 23:24:50 GMT',
             'x-amz-request-id': 'tx000000000000000000005-005c8ae2c1-aebea-default'},
 'reason': 'Internal Server Error',
 'status': 500}
DEBUG: S3Error: 500 (Internal Server Error)
DEBUG: HttpHeader: content-length: 219
DEBUG: HttpHeader: x-amz-request-id: tx000000000000000000005-005c8ae2c1-aebea-default
DEBUG: HttpHeader: accept-ranges: bytes
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: date: Thu, 14 Mar 2019 23:24:50 GMT
DEBUG: ErrorXML: Code: 'UnknownError'
DEBUG: ErrorXML: BucketName: 'bucket1'
DEBUG: ErrorXML: RequestId: 'tx000000000000000000005-005c8ae2c1-aebea-default'
DEBUG: ErrorXML: HostId: 'aebea-default-default'
WARNING: Retrying failed request: /?cors (500 (UnknownError))
WARNING: Waiting 3 sec...
DEBUG: Using signature v4
DEBUG: get_hostname(bucket1): dell-per630-9.gsslab.pnq2.redhat.com:8080
DEBUG: canonical_headers = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232453Z




2019-03-15 04:54:49.590850 7f4688e3e700 20 CONTENT_LENGTH=0
2019-03-15 04:54:49.590862 7f4688e3e700 20 HTTP_ACCEPT_ENCODING=identity
2019-03-15 04:54:49.590864 7f4688e3e700 20 HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential=1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee
2019-03-15 04:54:49.590868 7f4688e3e700 20 HTTP_HOST=dell-per630-9.gsslab.pnq2.redhat.com:8080
2019-03-15 04:54:49.590870 7f4688e3e700 20 HTTP_VERSION=1.1
2019-03-15 04:54:49.590872 7f4688e3e700 20 HTTP_X_AMZ_CONTENT_SHA256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2019-03-15 04:54:49.590875 7f4688e3e700 20 HTTP_X_AMZ_DATE=20190314T232449Z
2019-03-15 04:54:49.590877 7f4688e3e700 20 QUERY_STRING=cors
2019-03-15 04:54:49.590882 7f4688e3e700 20 REMOTE_ADDR=10.3.116.162
2019-03-15 04:54:49.590884 7f4688e3e700 20 REQUEST_METHOD=GET
2019-03-15 04:54:49.590885 7f4688e3e700 20 REQUEST_URI=/bucket1/
2019-03-15 04:54:49.590886 7f4688e3e700 20 SCRIPT_URI=/bucket1/
2019-03-15 04:54:49.590888 7f4688e3e700 20 SERVER_PORT=8080
2019-03-15 04:54:49.590890 7f4688e3e700  1 ====== starting new request req=0x7f4688e37f90 =====
2019-03-15 04:54:49.590919 7f4688e3e700  2 req 5:0.000028::GET /bucket1/::initializing for trans_id = tx000000000000000000005-005c8ae2c1-aebea-default
2019-03-15 04:54:49.590929 7f4688e3e700 10 rgw api priority: s3=6 s3website=5
2019-03-15 04:54:49.590931 7f4688e3e700 10 host=dell-per630-9.gsslab.pnq2.redhat.com
2019-03-15 04:54:49.590933 7f4688e3e700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2019-03-15 04:54:49.590935 7f4688e3e700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bucket1/
2019-03-15 04:54:49.590946 7f4688e3e700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2019-03-15 04:54:49.590950 7f4688e3e700 10 meta>> HTTP_X_AMZ_DATE
2019-03-15 04:54:49.590955 7f4688e3e700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2019-03-15 04:54:49.590959 7f4688e3e700 10 x>> x-amz-date:20190314T232449Z
2019-03-15 04:54:49.590977 7f4688e3e700 20 get_handler handler=25RGWHandler_REST_Bucket_S3
2019-03-15 04:54:49.590981 7f4688e3e700 10 handler=25RGWHandler_REST_Bucket_S3
2019-03-15 04:54:49.590984 7f4688e3e700  2 req 5:0.000094:s3:GET /bucket1/::getting op 0
2019-03-15 04:54:49.590989 7f4688e3e700 10 op=22RGWGetCORS_ObjStore_S3
2019-03-15 04:54:49.590991 7f4688e3e700  2 req 5:0.000101:s3:GET /bucket1/:get_cors:verifying requester
2019-03-15 04:54:49.590993 7f4688e3e700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2019-03-15 04:54:49.590995 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2019-03-15 04:54:49.590998 7f4688e3e700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2019-03-15 04:54:49.590999 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy
2019-03-15 04:54:49.591001 7f4688e3e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine
2019-03-15 04:54:49.591012 7f4688e3e700 10 v4 signature format = 758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee
2019-03-15 04:54:49.591017 7f4688e3e700 10 v4 credential format = 1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request
2019-03-15 04:54:49.591019 7f4688e3e700 10 access key id = 1UOOQZ8213EP5UT4AXJB
2019-03-15 04:54:49.591021 7f4688e3e700 10 credential scope = 20190314//s3/aws4_request
2019-03-15 04:54:49.591043 7f4688e3e700 10 canonical headers format = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232449Z

2019-03-15 04:54:49.591049 7f4688e3e700 10 payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2019-03-15 04:54:49.591074 7f4688e3e700 10 canonical request = GET
/bucket1/
cors=
host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232449Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2019-03-15 04:54:49.591075 7f4688e3e700 10 canonical request hash = 65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e
2019-03-15 04:54:49.591115 7f4688e3e700 10 string to sign = AWS4-HMAC-SHA256
20190314T232449Z
20190314//s3/aws4_request
65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e
2019-03-15 04:54:49.591143 7f4688e3e700 20 sending request to http://10.74.130.128:5000/v3/auth/tokens
2019-03-15 04:54:50.062414 7f4688e3e700 20 sending request to http://10.74.130.128:5000/v3/s3tokens
2019-03-15 04:54:50.079774 7f4688e3e700 20 rgw::auth::keystone::EC2Engine denied with reason=-2028
2019-03-15 04:54:50.079787 7f4688e3e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028
2019-03-15 04:54:50.079790 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2019-03-15 04:54:50.079802 7f4688e3e700 10 v4 signature format = 758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee
2019-03-15 04:54:50.079808 7f4688e3e700 10 v4 credential format = 1UOOQZ8213EP5UT4AXJB/20190314//s3/aws4_request
2019-03-15 04:54:50.079810 7f4688e3e700 10 access key id = 1UOOQZ8213EP5UT4AXJB
2019-03-15 04:54:50.079811 7f4688e3e700 10 credential scope = 20190314//s3/aws4_request
2019-03-15 04:54:50.079835 7f4688e3e700 10 canonical headers format = host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232449Z

2019-03-15 04:54:50.079842 7f4688e3e700 10 payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2019-03-15 04:54:50.079866 7f4688e3e700 10 canonical request = GET
/bucket1/
cors=
host:dell-per630-9.gsslab.pnq2.redhat.com:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20190314T232449Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2019-03-15 04:54:50.079868 7f4688e3e700 10 canonical request hash = 65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e
2019-03-15 04:54:50.079926 7f4688e3e700 10 string to sign = AWS4-HMAC-SHA256
20190314T232449Z
20190314//s3/aws4_request
65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e
2019-03-15 04:54:50.080002 7f4688e3e700 10 date_k    = 0b6d2c2533a022d37edb1e795d38732280e0e5cc9e702603ec65f441073e609f
2019-03-15 04:54:50.080013 7f4688e3e700 10 region_k  = b986c8f42c32a79528fd4c677155515f3acf588cc8209409508a0a53b302ff6c
2019-03-15 04:54:50.080018 7f4688e3e700 10 service_k = eabd9a65d56b942d90090e34b53589db933f17dc11aba8f053efc8a78a3d5fe2
2019-03-15 04:54:50.080024 7f4688e3e700 10 signing_k = 692aaf5fa4a688f5320e1b019983da0b0f3c9a8ef50a2b758f7618f35478da9c
2019-03-15 04:54:50.080048 7f4688e3e700 10 generated signature = 758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee
2019-03-15 04:54:50.080051 7f4688e3e700 15 string_to_sign=AWS4-HMAC-SHA256
20190314T232449Z
20190314//s3/aws4_request
65cc1010fc80193c0434af7a59c12d392f40c6e9763c93285e1bc81aedc7129e
2019-03-15 04:54:50.080061 7f4688e3e700 15 server signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee
2019-03-15 04:54:50.080062 7f4688e3e700 15 client signature=758ad0351151d54cb86bddec86bda6e477b0352ea25a44cf41224ba21c1b4fee
2019-03-15 04:54:50.080063 7f4688e3e700 15 compare=0
2019-03-15 04:54:50.080068 7f4688e3e700 20 rgw::auth::s3::LocalEngine granted access
2019-03-15 04:54:50.080070 7f4688e3e700 20 rgw::auth::s3::AWSAuthStrategy granted access
2019-03-15 04:54:50.080073 7f4688e3e700  2 req 5:0.489183:s3:GET /bucket1/:get_cors:normalizing buckets and tenants
2019-03-15 04:54:50.080076 7f4688e3e700 10 s->object=<NULL> s->bucket=bucket1
2019-03-15 04:54:50.080079 7f4688e3e700  2 req 5:0.489189:s3:GET /bucket1/:get_cors:init permissions
2019-03-15 04:54:50.080096 7f4688e3e700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>testuser</ID><DisplayName>Test User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>testuser</ID><DisplayName>Test User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2019-03-15 04:54:50.080105 7f4688e3e700 20 get_system_obj_state: rctx=0x7f4688e365f0 obj=default.rgw.meta:users.uid:testuser state=0x5608431c4040 s->prefetch_data=0
2019-03-15 04:54:50.080111 7f4688e3e700 10 cache get: name=default.rgw.meta+users.uid+testuser : hit (requested=0x6, cached=0x17)
2019-03-15 04:54:50.080115 7f4688e3e700 20 get_system_obj_state: s->obj_tag was set empty
2019-03-15 04:54:50.080116 7f4688e3e700 20 Read xattr: user.rgw.idtag
2019-03-15 04:54:50.080131 7f4688e3e700  2 req 5:0.489241:s3:GET /bucket1/:get_cors:recalculating target
2019-03-15 04:54:50.080133 7f4688e3e700  2 req 5:0.489243:s3:GET /bucket1/:get_cors:reading permissions
2019-03-15 04:54:50.080134 7f4688e3e700  2 req 5:0.489245:s3:GET /bucket1/:get_cors:init op
2019-03-15 04:54:50.080136 7f4688e3e700  2 req 5:0.489246:s3:GET /bucket1/:get_cors:verifying op mask
2019-03-15 04:54:50.080137 7f4688e3e700 20 required_mask= 1 user.op_mask=7
2019-03-15 04:54:50.080139 7f4688e3e700  2 req 5:0.489248:s3:GET /bucket1/:get_cors:verifying op permissions
2019-03-15 04:54:50.080141 7f4688e3e700  2 req 5:0.489252:s3:GET /bucket1/:get_cors:verifying op params
2019-03-15 04:54:50.080142 7f4688e3e700  2 req 5:0.489253:s3:GET /bucket1/:get_cors:pre-executing
2019-03-15 04:54:50.080143 7f4688e3e700  2 req 5:0.489254:s3:GET /bucket1/:get_cors:executing
2019-03-15 04:54:50.080146 7f4688e3e700 20 no CORS configuration attr found
2019-03-15 04:54:50.080147 7f4688e3e700  2 No CORS configuration set yet for this bucket
2019-03-15 04:54:50.080147 7f4688e3e700  2 req 5:0.489258:s3:GET /bucket1/:get_cors:completing
2019-03-15 04:54:50.080150 7f4688e3e700  0 WARNING: set_req_state_err err_no=2044 resorting to 500
2019-03-15 04:54:50.080305 7f4688e3e700  2 req 5:0.489415:s3:GET /bucket1/:get_cors:op status=-2
2019-03-15 04:54:50.080307 7f4688e3e700  2 req 5:0.489417:s3:GET /bucket1/:get_cors:http status=500
2019-03-15 04:54:50.080309 7f4688e3e700  1 ====== req done req=0x7f4688e37f90 op status=-2 http_status=500 ======
2019-03-15 04:54:50.080320 7f4688e3e700 20 process_request() returned -2044
2019-03-15 04:54:50.080340 7f4688e3e700  1 civetweb: 0x5608429b0000: 10.3.116.162 - - [15/Mar/2019:04:54:45 +0530] "GET /bucket1/?cors HTTP/1.1" 500 0 - -
Comment 1 Vikhyat Umrao 2019-03-15 20:23:54 UTC 
I did look into what was changed in between 12.2.5 which is 3.1 and 12.2.8 which is 3.2 but I see only one change related to CORS which is - http://tracker.ceph.com/issues/24810 but looks like it is not related.

As I went ahead and tested upstream 12.2.8 with vstart cluster and it is working fine at least I am 100% sure it came something in downstream custom backport we did for RGW on top on upstream v12.2.8.

# cat build/out/rgw.j.log |less

2019-03-16 01:30:54.034491 7fa870a2bfc0  0 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process radosgw, pid 42623
2019-03-16 01:30:54.035183 7fa870a2bfc0  0 pidfile_write: ignore empty --pid-file

$ s3cmd info s3://bucket1
s3://bucket1/ (bucket):
   Location:  us-east-1
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    none
   CORS:      none
   ACL:       M. Tester: FULL_CONTROL

We need to find out which backports we did on top v12.2.8 and which one is impacting indirectly CORS output.
Comment 2 Vikhyat Umrao 2019-03-15 20:33:43 UTC 
2019-03-15 04:54:50.080146 7f4688e3e700 20 no CORS configuration attr found
2019-03-15 04:54:50.080147 7f4688e3e700  2 No CORS configuration set yet for this bucket

I was checking the code around these logs but no luck identify what changes in CORS code could be causing as only change I saw which I mentioned in comment#1 and it is present in upstream v12.2.8 and in this version it is working it means this gives +1 that this change is not causing the issue.

From RHCS 3.1 - 12.2.5-59.el7cp

$ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1
s3://bucket1/ (bucket):
   Location:  us-east-1
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    none
   CORS:      none
   ACL:       Test User: FULL_CONTROL


From RHCS 3.2.z1 - 12.2.8-89.el7cp

$ s3cmd -c .s3cfg.rhcs.3 info s3://bucket1
s3://bucket1/ (bucket):
   Location:  us-east-1
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    none
WARNING: Retrying failed request: /?cors (500 (UnknownError))
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?cors (500 (UnknownError))
WARNING: Waiting 6 sec...
Comment 20 errata-xmlrpc 2019-04-30 15:57:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:0911
Note You need to log in before you can comment on or make changes to this bug.