Bug 2142941
| Summary: | RGW cloud Transition. HEAD/GET requests to MCG are failing with 403 error | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | daniel parkes <dparkes> |
| Component: | Multi-Cloud Object Gateway | Assignee: | Nobody <nobody> |
| Status: | CLOSED ERRATA | QA Contact: | Tejas <tchandra> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.11 | CC: | dparkes, ebenahar, kramdoss, muagarwa, nbecker, ocs-bugs, odf-bz-bot, skoduri, tchandra |
| Target Milestone: | --- | ||
| Target Release: | ODF 4.13.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | 4.13.0-34 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-06-21 15:22:18 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2139694 | ||
@nbecker @dparkes Can you please provide the steps how to verify this issue? Thanks! I believe this will be tested by RGW / Ceph QE Yes, I agree with Nimrod that it would be a good idea for this fix to be tested by RGW / Ceph QE Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:3742 |
Description of problem (please be detailed as possible and provide log snippets): Issue 1) HEAD/GET requests to MCG are failing with 403 error. But the same requests succeed if using AWS endpoint directly sample packets collected - Frame 370399: 655 bytes on wire (5240 bits), 655 bytes captured (5240 bits) on interface any, id 0 Linux cooked capture v1 Internet Protocol Version 4, Src: 192.168.122.57, Dst: 20.241.247.215 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 639 Identification: 0x3a5b (14939) Flags: 0x40, Don't fragment ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 64 Protocol: TCP (6) Header Checksum: 0xb673 [validation disabled] [Header checksum status: Unverified] Source Address: 192.168.122.57 Destination Address: 20.241.247.215 [Destination GeoIP: US] Transmission Control Protocol, Src Port: 42430, Dst Port: 80, Seq: 1019, Ack: 1888, Len: 587 Hypertext Transfer Protocol HEAD /awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1\r\n Host: s3-openshift-storage.apps.ocp410.077dazopenshift.com\r\n Accept: */*\r\n Authorization: AWS4-HMAC-SHA256 Credential=Ieip6XK4IsHzgAdcBzhl/20221108/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=70b81add012ca01063a3823b6637b4d84a99c43eaebdc80ea276d4badf0b13fb\r\n Date: Tue, 08 Nov 2022 05:01:38 +0000\r\n X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\n X-Amz-Date: 20221108T050138Z\r\n \r\n [Full request URI: http://s3-openshift-storage.apps.ocp410.077dazopenshift.com/awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt] [HTTP request 3/3] [Prev request in frame: 370332] [Response in frame: 370415] Frame 370414: 684 bytes on wire (5472 bits), 684 bytes captured (5472 bits) on interface any, id 0 Linux cooked capture v1 Internet Protocol Version 4, Src: 20.241.247.215, Dst: 192.168.86.167 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 668 Identification: 0x5e1d (24093) Flags: 0x40, Don't fragment ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 42 Protocol: TCP (6) Header Checksum: 0xcc26 [validation disabled] [Header checksum status: Unverified] Source Address: 20.241.247.215 Destination Address: 192.168.86.167 [Source GeoIP: US] Transmission Control Protocol, Src Port: 80, Dst Port: 42430, Seq: 1888, Ack: 1606, Len: 616 Hypertext Transfer Protocol HTTP/1.1 403 Forbidden\r\n x-amz-request-id: la7qw6lm-36mvas-p2n\r\n x-amz-id-2: la7qw6lm-36mvas-p2n\r\n access-control-allow-origin: *\r\n access-control-allow-credentials: true\r\n access-control-allow-methods: GET,POST,PUT,DELETE,OPTIONS\r\n access-control-allow-headers: Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256\r\n access-control-expose-headers: ETag,X-Amz-Version-Id\r\n content-type: application/xml\r\n content-length: 503\r\n [Content length: 503] date: Tue, 08 Nov 2022 05:01:44 GMT\r\n keep-alive: timeout=5\r\n set-cookie: 1a4aa612fe797ac8466d7ee00e5520d5=92e33383aa51b20f8ac617f7458ed772; path=/; HttpOnly\r\n \r\n [HTTP response 3/3] [Time since request: 0.282287889 seconds] [Prev request in frame: 370334] [Prev response in frame: 370345] [Request in frame: 370401] [Request URI: http://s3-openshift-storage.apps.ocp410.077dazopenshift.com/awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt] Error Logs from Noobaa pod - [root@extensa022 oc]# ./oc logs noobaa-endpoint-8bddcccb8-6m4z4 | grep HEAD | grep lc_t56 Nov-8 4:07:09.407 [Endpoint/13] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your AWS secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.</Message><Resource>/awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt</Resource><RequestId>la7oxzrx-72sbdy-2up</RequestId></Error> HEAD /awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt {"accept":"*/*","authorization":"AWS4-HMAC-SHA256 Credential=Ieip6XK4IsHzgAdcBzhl/20221108/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=68260237c682b9f11e09e70b44b5605f3ab5c8296721c68bff96ca51dddf173b","date":"Tue, 08 Nov 2022 04:07:09 +0000","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","x-amz-date":"20221108T040709Z","host":"s3-openshift-storage.apps.ocp410.077dazopenshift.com","x-forwarded-host":"s3-openshift-storage.apps.ocp410.077dazopenshift.com","x-forwarded-port":"80","x-forwarded-proto":"http","forwarded":"for=122.166.91.8;host=s3-openshift-storage.apps.ocp410.077dazopenshift.com;proto=http","x-forwarded-for":"122.166.91.8"} Error: Signature that was calculated did not match [root@extensa022 oc]# Note: The same request format succeeds on AWS endpoint Frame 14664: 627 bytes on wire (5016 bits), 627 bytes captured (5016 bits) on interface any, id 0 Linux cooked capture v1 Internet Protocol Version 4, Src: 192.168.122.57, Dst: 52.216.56.0 Transmission Control Protocol, Src Port: 59240, Dst Port: 80, Seq: 490, Ack: 250, Len: 571 Hypertext Transfer Protocol HEAD /awsnamespacestore/bucket-awsns%2Flc_t1?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1\r\n Host: s3.us-east-1.amazonaws.com\r\n Accept: */*\r\n Authorization: AWS4-HMAC-SHA256 Credential=AKIAV5CPJ352P4NDRWP2/20221108/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=f2717aef6356187c7e0a5b81e5aab3ff3cd62b7ceb7cdb15a510705520882679\r\n Date: Tue, 08 Nov 2022 05:22:17 +0000\r\n X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\n X-Amz-Date: 20221108T052217Z\r\n \r\n [Full request URI: http://s3.us-east-1.amazonaws.com/awsnamespacestore/bucket-awsns%2Flc_t1?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt] [HTTP request 2/4] [Prev request in frame: 14652] [Response in frame: 14681] 14681 522.571330488 52.216.56.0 192.168.122.57 HTTP 627 HTTP/1.1 200 OK Version of all relevant components (if applicable): Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: