Bug 1414994

Summary: wget and AWS authentication header ERROR 403: Forbidden.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vikhyat Umrao <vumrao>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED NOTABUG QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.1CC: cbodley, ceph-eng-bugs, kbader, linuxkidd, mbenjamin, mhackett, mwatts, owasserm, sweil
Target Milestone: rc   
Target Release: 2.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-09 14:55:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vikhyat Umrao 2017-01-19 22:24:36 UTC
Description of problem:

wget and AWS authentication header ERROR 403: Forbidden.

$python test.py http://radosgw1.redhat.com:80/test-bucket1/s3cmd-1.6.1.tar.gz

wget -S http://radosgw1.redhat.com:80/test-bucket1/s3cmd-1.6.1.tar.gz --header="Authorization:AWS 12I2IH52A5ALV0C0ME7D:543qXdOfRTpSheMfNDjDG+GTfHE="        --header="Date:Thu, 19 Jan 2017 20:39:00 GMT"

--2017-01-20 02:09:00--  http://radosgw1.redhat.com/test-bucket1/s3cmd-1.6.1.tar.gz
Resolving radosgw1.redhat.com (radosgw1.redhat.com)... 10.65.2.246
Connecting to radosgw1.redhat.com (radosgw1.redhat.com)|10.65.2.246|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 403 Forbidden
  x-amz-request-id: tx00000000000000000004c-00588123e4-54224-default
  Content-Length: 196
  Accept-Ranges: bytes
  Content-Type: application/xml
  Date: Thu, 19 Jan 2017 20:39:00 GMT
  Connection: Keep-Alive
2017-01-20 02:09:00 ERROR 403: Forbidden.

Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 2.1
ceph-radosgw-10.2.3-17.el7cp.x86_64
wget-1.14-13.el7.x86_64
RHEL 7.3

How reproducible:
Always


Steps to Reproduce:
1. Created a script using link[1] as test.py.
2. Run as $python test.py http://radosgw1.redhat.com:80/test-bucket1/s3cmd-1.6.1.tar.gz
3. Error : ERROR 403: Forbidden.

[1] https://skizhak.wordpress.com/2014/01/02/fetch-content-from-amazon-s3-using-wget/

Comment 1 Vikhyat Umrao 2017-01-19 22:30:02 UTC
I have added one more line after cmd in the above given script to print the final command.

+ print cmd

I tried a couple of method before going to the conclusion that in AWS Authentication header the second field is a signature, not the secret key.

1) # wget --verbose -S http://radosgw1.redhat.com/test-bucket/testfile --header "Authorization:AWS 12I2IH52A5ALV0C0ME7D:YVaEUltC7du1sTuudyK2TX5am7uuPc4NWnUYZYVQ" --header "Date:Fri, 20 Jan 2017 01:50:40 IST"

--2017-01-20 01:53:31-- http://radosgw1.redhat.com/test-bucket/testfile
Resolving radosgw1.redhat.com (radosgw1.redhat.com)... 10.65.2.246
Connecting to radosgw1.redhat.com (radosgw1.redhat.com)|10.65.2.246|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 403 Forbidden
  x-amz-request-id: tx000000000000000000046-0058812044-54224-default
  Content-Length: 187
  Accept-Ranges: bytes
  Content-Type: application/xml
  Date: Thu, 19 Jan 2017 20:23:32 GMT
  Connection: Keep-Alive
2017-01-20 01:53:32 ERROR 403: Forbidden.

- If I run the same command which you are running, I am getting below given error in logs with debug_rgw=20.

2017-01-20 01:53:32.166864 7f00307c8700 0 NOTICE: failed to parse date for auth header <============
2017-01-20 01:53:32.166866 7f00307c8700 10 failed to create auth header

2017-01-20 01:53:32.166868 7f00307c8700 10 failed to authorize request
2017-01-20 01:53:32.166869 7f00307c8700 20 handler->ERRORHANDLER: err_no=-1 new_err_no=-1

- It says failed to parse date command.

2) # wget --verbose -S http://radosgw1.redhat.com/test-bucket/testfile --header "Authorization:AWS 12I2IH52A5ALV0C0ME7D:YVaEUltC7du1sTuudyK2TX5am7uuPc4NWnUYZYVQ Date:Fri, 20 Jan 2017 01:50:40 IST"

--2017-01-20 01:58:18-- http://radosgw1.redhat.com/test-bucket/testfile
Resolving radosgw1.redhat.com (radosgw1.redhat.com)... 10.65.2.246
Connecting to radosgw1.redhat.com (radosgw1.redhat.com)|10.65.2.246|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 403 Forbidden
  x-amz-request-id: tx000000000000000000049-0058812162-54224-default
  Content-Length: 193
  Accept-Ranges: bytes
  Content-Type: application/xml
  Date: Thu, 19 Jan 2017 20:28:18 GMT
  Connection: Keep-Alive
2017-01-20 01:58:18 ERROR 403: Forbidden.

2017-01-20 01:58:18.430294 7f002f7c6700 2 req 73:0.000146:s3:GET /test-bucket/testfile:get_obj:authorizing
2017-01-20 01:58:18.430332 7f002f7c6700 20 get_system_obj_state: rctx=0x7f002f7bf1e0 obj=default.rgw.users.keys:12I2IH52A5ALV0C0ME7D:YVaEUltC7du1sTuudyK2TX5am7uuPc4NWnUYZYVQ Date:Fri, 20 Jan 2017 01:50 state=0x7f00e4015d98 s->prefetch_data=0
2017-01-20 01:58:18.430362 7f002f7c6700 10 cache get: name=default.rgw.users.keys+12I2IH52A5ALV0C0ME7D:YVaEUltC7du1sTuudyK2TX5am7uuPc4NWnUYZYVQ Date:Fri, 20 Jan 2017 01:50 : type miss (requested=6, cached=0)
2017-01-20 01:58:18.431754 7f002f7c6700 10 cache put: name=default.rgw.users.keys+12I2IH52A5ALV0C0ME7D:YVaEUltC7du1sTuudyK2TX5am7uuPc4NWnUYZYVQ Date:Fri, 20 Jan 2017 01:50 info.flags=0
2017-01-20 01:58:18.431776 7f002f7c6700 10 moving default.rgw.users.keys+12I2IH52A5ALV0C0ME7D:YVaEUltC7du1sTuudyK2TX5am7uuPc4NWnUYZYVQ Date:Fri, 20 Jan 2017 01:50 to cache LRU end
2017-01-20 01:58:18.431789 7f002f7c6700 5 error reading user info, uid=12I2IH52A5ALV0C0ME7D:YVaEUltC7du1sTuudyK2TX5am7uuPc4NWnUYZYVQ Date:Fri, 20 Jan 2017 01:50 can't authenticate

2017-01-20 01:58:18.431792 7f002f7c6700 10 failed to authorize request

2017-01-20 01:58:18.431794 7f002f7c6700 20 handler->ERRORHANDLER: err_no=-2028 new_err_no=-2028
2017-01-20 01:58:18.431912 7f002f7c6700 2 req 73:0.001763:s3:GET /test-bucket/testfile:get_obj:op status=0
2017-01-20 01:58:18.431928 7f002f7c6700 2 req 73:0.001780:s3:GET /test-bucket/testfile:get_obj:http status=403

- The header is complete now but still the same issue with the different error code.

- Then I did some search around AWS authentication header and came to know that Authorization: AWS second part is not a secret key it is a signature. Please check link [1] and [2] for more information.

- If S3 objects and buckets are public then we can easily download them without mentioning the credential because they are public.

$ s3cmd put --acl-public index.html s3://test-bucket/
upload: 'index.html' -> 's3://test-bucket/index.html' [1 of 1]
 0 of 0 0% in 0s 0.00 B/s done
Public URL of the object is: http://radosgw1.redhat.com/test-bucket/index.html

# wget http://radosgw1.redhat.com/test-bucket/index.html
--2017-01-20 02:04:16-- http://radosgw1.redhat.com/test-bucket/index.html
Resolving radosgw1.redhat.com (radosgw1.redhat.com)... 10.65.2.246
Connecting to radosgw1.redhat.com (radosgw1.redhat.com)|10.65.2.246|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [inode/x-empty]
Saving to: ‘index.html’

  [ <=> ] 0 --.-K/s in 0s 

2017-01-20 02:04:16 (0.00 B/s) - ‘index.html’ saved [0/0]

[1] https://forums.aws.amazon.com/thread.jspa?messageID=251088
[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html