Description of problem (please be detailed as possible and provide log
snippets):


Issue 1) HEAD/GET requests to MCG are failing with 403 error. But the same requests succeed if using AWS endpoint directly

sample packets collected -

Frame 370399: 655 bytes on wire (5240 bits), 655 bytes captured (5240 bits) on interface any, id 0
Linux cooked capture v1
Internet Protocol Version 4, Src: 192.168.122.57, Dst: 20.241.247.215
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 639
    Identification: 0x3a5b (14939)
    Flags: 0x40, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: TCP (6)
    Header Checksum: 0xb673 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.122.57
    Destination Address: 20.241.247.215
    [Destination GeoIP: US]
Transmission Control Protocol, Src Port: 42430, Dst Port: 80, Seq: 1019, Ack: 1888, Len: 587
Hypertext Transfer Protocol
    HEAD /awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1\r\n
    Host: s3-openshift-storage.apps.ocp410.077dazopenshift.com\r\n
    Accept: */*\r\n
    Authorization: AWS4-HMAC-SHA256 Credential=Ieip6XK4IsHzgAdcBzhl/20221108/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=70b81add012ca01063a3823b6637b4d84a99c43eaebdc80ea276d4badf0b13fb\r\n
    Date: Tue, 08 Nov 2022 05:01:38 +0000\r\n
    X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\n
    X-Amz-Date: 20221108T050138Z\r\n
    \r\n
    [Full request URI: http://s3-openshift-storage.apps.ocp410.077dazopenshift.com/awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt]
    [HTTP request 3/3]
    [Prev request in frame: 370332]
    [Response in frame: 370415]



Frame 370414: 684 bytes on wire (5472 bits), 684 bytes captured (5472 bits) on interface any, id 0
Linux cooked capture v1
Internet Protocol Version 4, Src: 20.241.247.215, Dst: 192.168.86.167
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 668
    Identification: 0x5e1d (24093)
    Flags: 0x40, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 42
    Protocol: TCP (6)
    Header Checksum: 0xcc26 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 20.241.247.215
    Destination Address: 192.168.86.167
    [Source GeoIP: US]
Transmission Control Protocol, Src Port: 80, Dst Port: 42430, Seq: 1888, Ack: 1606, Len: 616
Hypertext Transfer Protocol
    HTTP/1.1 403 Forbidden\r\n
    x-amz-request-id: la7qw6lm-36mvas-p2n\r\n
    x-amz-id-2: la7qw6lm-36mvas-p2n\r\n
    access-control-allow-origin: *\r\n
    access-control-allow-credentials: true\r\n
    access-control-allow-methods: GET,POST,PUT,DELETE,OPTIONS\r\n
    access-control-allow-headers: Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256\r\n
    access-control-expose-headers: ETag,X-Amz-Version-Id\r\n
    content-type: application/xml\r\n
    content-length: 503\r\n
        [Content length: 503]
    date: Tue, 08 Nov 2022 05:01:44 GMT\r\n
    keep-alive: timeout=5\r\n
    set-cookie: 1a4aa612fe797ac8466d7ee00e5520d5=92e33383aa51b20f8ac617f7458ed772; path=/; HttpOnly\r\n
    \r\n
    [HTTP response 3/3]
    [Time since request: 0.282287889 seconds]
    [Prev request in frame: 370334]
    [Prev response in frame: 370345]
    [Request in frame: 370401]
    [Request URI: http://s3-openshift-storage.apps.ocp410.077dazopenshift.com/awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt]


Error Logs from Noobaa pod -


[root@extensa022 oc]# ./oc logs noobaa-endpoint-8bddcccb8-6m4z4 | grep HEAD | grep lc_t56
Nov-8 4:07:09.407 [Endpoint/13] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your AWS secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.</Message><Resource>/awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&amp;rgwx-stat=true&amp;rgwx-sync-manifest&amp;rgwx-skip-decrypt</Resource><RequestId>la7oxzrx-72sbdy-2up</RequestId></Error> HEAD /awsnsbucket/bucket1%2Flc_t56?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt {"accept":"*/*","authorization":"AWS4-HMAC-SHA256 Credential=Ieip6XK4IsHzgAdcBzhl/20221108/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=68260237c682b9f11e09e70b44b5605f3ab5c8296721c68bff96ca51dddf173b","date":"Tue, 08 Nov 2022 04:07:09 +0000","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","x-amz-date":"20221108T040709Z","host":"s3-openshift-storage.apps.ocp410.077dazopenshift.com","x-forwarded-host":"s3-openshift-storage.apps.ocp410.077dazopenshift.com","x-forwarded-port":"80","x-forwarded-proto":"http","forwarded":"for=122.166.91.8;host=s3-openshift-storage.apps.ocp410.077dazopenshift.com;proto=http","x-forwarded-for":"122.166.91.8"} Error: Signature that was calculated did not match
[root@extensa022 oc]#



Note: The same request format succeeds on AWS endpoint 

Frame 14664: 627 bytes on wire (5016 bits), 627 bytes captured (5016 bits) on interface any, id 0
Linux cooked capture v1
Internet Protocol Version 4, Src: 192.168.122.57, Dst: 52.216.56.0
Transmission Control Protocol, Src Port: 59240, Dst Port: 80, Seq: 490, Ack: 250, Len: 571
Hypertext Transfer Protocol
    HEAD /awsnamespacestore/bucket-awsns%2Flc_t1?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1\r\n
    Host: s3.us-east-1.amazonaws.com\r\n
    Accept: */*\r\n
    Authorization: AWS4-HMAC-SHA256 Credential=AKIAV5CPJ352P4NDRWP2/20221108/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=f2717aef6356187c7e0a5b81e5aab3ff3cd62b7ceb7cdb15a510705520882679\r\n
    Date: Tue, 08 Nov 2022 05:22:17 +0000\r\n
    X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\n
    X-Amz-Date: 20221108T052217Z\r\n
    \r\n
    [Full request URI: http://s3.us-east-1.amazonaws.com/awsnamespacestore/bucket-awsns%2Flc_t1?rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt]
    [HTTP request 2/4]
    [Prev request in frame: 14652]
    [Response in frame: 14681]


14681	522.571330488	52.216.56.0	192.168.122.57	HTTP	627	HTTP/1.1 200 OK 

Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info: