Bug 1361760
| Summary: | [RGW:Active directory]:- "Access denied" error thrown for AD users after being idle for sometime | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | shylesh <shmohan> |
| Component: | RGW | Assignee: | Matt Benjamin (redhat) <mbenjamin> |
| Status: | CLOSED ERRATA | QA Contact: | shylesh <shmohan> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.0 | CC: | cbodley, ceph-eng-bugs, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil, vakulkar |
| Target Milestone: | rc | ||
| Target Release: | 2.0 | ||
| Hardware: | x86_64 | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | RHEL: ceph-10.2.2-33.el7cp Ubuntu: ceph_10.2.2-25redhat1xenial | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-08-23 19:45:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Verified on 10.2.2-25redhat1xenial and ceph-radosgw-10.2.2-33.el7cp.x86_64. Marking it as verified. (In reply to shylesh from comment #8) > Verified on 10.2.2-25redhat1xenial and ceph-radosgw-10.2.2-33.el7cp.x86_64. > Marking it as verified. Even with idle time of 45 minutes authentication is working fine. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1755.html |
Description of problem: If AD users doesn't do authentication for sometime (30 minutes or so) then trying to authenticate again throws "Access denied". However restarting rgw process fixes the problem. Not sure this is cache invalidation or something . Version-Release number of selected component (if applicable): 10.2.2-24redhat1xenial How reproducible: Always Steps to Reproduce: 1. Configure rgw with AD authentication 2. Do some I/O with s3 apis using the AD user 3. keep idle for 30 minutes or so 4. try to do I/O again with s3 apis Actual results: ubuntu@magna118:~$ python AD-s3.py Traceback (most recent call last): File "AD-s3.py", line 19, in <module> bucket = conn.create_bucket('ad-fresh-bucket') File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 621, in create_bucket response.status, response.reason, body) boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx000000000000000000045-00579c4b57-4d6848-default</RequestId><HostId>4d6848-default-default</HostId></Error> Expected results: Connection should be reestablished and user should not see "access denied" Additional info: here is the log snippet from rgw when we see "access denied" 2016-07-30 06:38:15.634904 7f1e4a7fc700 20 RGWEnv::set(): HTTP_HOST: magna118:8080 2016-07-30 06:38:15.634922 7f1e4a7fc700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity 2016-07-30 06:38:15.634926 7f1e4a7fc700 20 RGWEnv::set(): HTTP_DATE: Sat, 30 Jul 2016 06:38:15 GMT 2016-07-30 06:38:15.634928 7f1e4a7fc700 20 RGWEnv::set(): CONTENT_LENGTH: 0 2016-07-30 06:38:15.634931 7f1e4a7fc700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc 2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg==:sXd KPvugn9QTgwPpOVdCU3HPw4s= 2016-07-30 06:38:15.634935 7f1e4a7fc700 20 RGWEnv::set(): HTTP_USER_AGENT: Boto/2.38.0 Python/2.7.12 Linux/4.4.0-31-generic 2016-07-30 06:38:15.634937 7f1e4a7fc700 20 RGWEnv::set(): REQUEST_METHOD: PUT 2016-07-30 06:38:15.634938 7f1e4a7fc700 20 RGWEnv::set(): REQUEST_URI: /ad-fresh-bucket/ 2016-07-30 06:38:15.634939 7f1e4a7fc700 20 RGWEnv::set(): QUERY_STRING: 2016-07-30 06:38:15.634941 7f1e4a7fc700 20 RGWEnv::set(): REMOTE_USER: 2016-07-30 06:38:15.634942 7f1e4a7fc700 20 RGWEnv::set(): SCRIPT_URI: /ad-fresh-bucket/ 2016-07-30 06:38:15.634944 7f1e4a7fc700 20 RGWEnv::set(): SERVER_PORT: 8080 2016-07-30 06:38:15.634945 7f1e4a7fc700 20 CONTENT_LENGTH=0 2016-07-30 06:38:15.634946 7f1e4a7fc700 20 HTTP_ACCEPT_ENCODING=identity 2016-07-30 06:38:15.634947 7f1e4a7fc700 20 HTTP_AUTHORIZATION=AWS ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg==:sXdKPvugn9QTgwPpOVdCU3HPw4s= 2016-07-30 06:38:15.634949 7f1e4a7fc700 20 HTTP_DATE=Sat, 30 Jul 2016 06:38:15 GMT 2016-07-30 06:38:15.634950 7f1e4a7fc700 20 HTTP_HOST=magna118:8080 2016-07-30 06:38:15.634950 7f1e4a7fc700 20 HTTP_USER_AGENT=Boto/2.38.0 Python/2.7.12 Linux/4.4.0-31-generic 2016-07-30 06:38:15.634951 7f1e4a7fc700 20 QUERY_STRING= 2016-07-30 06:38:15.634951 7f1e4a7fc700 20 REMOTE_USER= 2016-07-30 06:38:15.634952 7f1e4a7fc700 20 REQUEST_METHOD=PUT 2016-07-30 06:38:15.634953 7f1e4a7fc700 20 REQUEST_URI=/ad-fresh-bucket/ 2016-07-30 06:38:15.634953 7f1e4a7fc700 20 SCRIPT_URI=/ad-fresh-bucket/ 2016-07-30 06:38:15.634954 7f1e4a7fc700 20 SERVER_PORT=8080 2016-07-30 06:38:15.634956 7f1e4a7fc700 1 ====== starting new request req=0x7f1e4a7f67e0 ===== 2016-07-30 06:38:15.634976 7f1e4a7fc700 2 req 69:0.000020::PUT /ad-fresh-bucket/::initializing for trans_id = tx000000000000000000045-00579c4b57-4d6848-default 2016-07-30 06:38:15.634981 7f1e4a7fc700 10 host=magna118 2016-07-30 06:38:15.634986 7f1e4a7fc700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2016-07-30 06:38:15.635018 7f1e4a7fc700 20 get_handler handler=25RGWHandler_REST_Bucket_S3 2016-07-30 06:38:15.635023 7f1e4a7fc700 10 handler=25RGWHandler_REST_Bucket_S3 2016-07-30 06:38:15.635025 7f1e4a7fc700 2 req 69:0.000070:s3:PUT /ad-fresh-bucket/::getting op 1 2016-07-30 06:38:15.635032 7f1e4a7fc700 10 op=27RGWCreateBucket_ObjStore_S3 2016-07-30 06:38:15.635033 7f1e4a7fc700 2 req 69:0.000077:s3:PUT /ad-fresh-bucket/:create_bucket:authorizing 2016-07-30 06:38:15.635039 7f1e4a7fc700 15 authorize_v2 LDAP auth uri=ldaps://msad-frog.eng.arb.redhat.com:636 2016-07-30 06:38:15.635145 7f1e4a7fc700 10 authorize_v2 try LDAP auth uri=ldaps://msad-frog.eng.arb.redhat.com:636 token.id=ceph 2016-07-30 06:38:15.635192 7f1e4a7fc700 20 get_system_obj_state: rctx=0x7f1e4a7f47b0 obj=default.rgw.users.keys:ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== state=0x7f1f0c00a168 s->prefetch_data=0 2016-07-30 06:38:15.635206 7f1e4a7fc700 10 cache get: name=default.rgw.users.keys+ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== : type miss (requested=6, cached=0) 2016-07-30 06:38:15.635856 7f1e4a7fc700 10 cache put: name=default.rgw.users.keys+ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== info.flags=0 2016-07-30 06:38:15.635864 7f1e4a7fc700 10 moving default.rgw.users.keys+ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== to cache LRU end 2016-07-30 06:38:15.635876 7f1e4a7fc700 5 error reading user info, uid=ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogIjVoeTEzNWhAMTIzIgogICAgfQp9Cg== can't authenticate 2016-07-30 06:38:15.635878 7f1e4a7fc700 10 failed to authorize request 2016-07-30 06:38:15.635879 7f1e4a7fc700 20 handler->ERRORHANDLER: err_no=-13 new_err_no=-13 2016-07-30 06:38:15.635961 7f1e4a7fc700 2 req 69:0.001005:s3:PUT /ad-fresh-bucket/:create_bucket:op status=0 2016-07-30 06:38:15.635967 7f1e4a7fc700 2 req 69:0.001011:s3:PUT /ad-fresh-bucket/:create_bucket:http status=403 2016-07-30 06:38:15.635972 7f1e4a7fc700 1 ====== req done req=0x7f1e4a7f67e0 op status=0 http_status=403 ====== 2016-07-30 06:38:15.635978 7f1e4a7fc700 20 process_request() returned -13 2016-07-30 06:38:15.636011 7f1e4a7fc700 1 civetweb: 0x7f1f0c004160: 10.8.128.118 - - [30/Jul/2016:06:38:15 +0000] "PUT /ad-fresh-bucket/ HTTP/1.1" 403 0 - Boto/2.38.0 Python/2.7.12 Linux/4.4.0-31-generic However please note that restarting rgw solves this problem and authentication works as usual.