Bug 1699206
| Summary: | ceph version 12.2.8-52.el7cp - NoSuchBucket for public containers when accessing from Horizon | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Michele Valsecchi <mvalsecc> |
| Component: | RGW | Assignee: | shilpa <smanjara> |
| Status: | CLOSED NOTABUG | QA Contact: | ceph-qe-bugs <ceph-qe-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.2 | CC: | cbodley, ceph-eng-bugs, kbader, mbenjamin, mwatts, sweil, tchandra |
| Target Milestone: | rc | ||
| Target Release: | 4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-06-11 05:56:57 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Stack trace of accessing Horizon `http://localhost/dashboard/project/containers/` In case of swift backend, we can notice that: the first request (http://10.0.0.110:8080/info) returns 200 and the following requests have the X-Auth-Token Header While in the Ceph backend we can notice that: the first request (http://10.0.0.110:8080/info) returns 404 but the following requests *have* the X-Auth-Token Header ---------------------------------------------------------------------------- SWIFT BACKEND ---------------------------------------------------------------------------- 2019-04-16 02:14:01,320 87 DEBUG swiftclient REQ: curl -i http://10.0.0.110:8080/info -X GET -H "Accept-Encoding: gzip" 2019-04-16 02:14:01,320 87 DEBUG swiftclient RESP STATUS: 200 OK <================== 2019-04-16 02:14:01,320 87 DEBUG swiftclient RESP HEADERS: {u'Date': u'Tue, 16 Apr 2019 02:14:01 GMT', u'Content-Length': u'1434', u'Content-Type': u'application/json; charset=UTF-8', u'X-Openstack-Request-Id': u'txf15006331c454c0e9ec37-005cb53a69', u'X-Trans-Id': u'txf15006331c454c0e9ec37-005cb53a69'} 2019-04-16 02:14:01,321 87 DEBUG swiftclient RESP BODY: {"formpost": {}, "bulk_delete": {"max_failed_deletes": 1000, "max_deletes_per_request": 10000}, "versioned_writes": {"allowed_flags": ["x-versions-location", "x-history-location"]}, "container_quotas": {}, "tempurl": {"outgoing_remove_headers": ["x-object-meta-*"], "methods": ["GET", "HEAD", "PUT", "POST", "DELETE"], "incoming_remove_headers": ["x-timestamp"], "allowed_digests": ["sha1", "sha256", "sha512"], "outgoing_allow_headers": ["x-object-meta-public-*"], "incoming_allow_headers": []}, "bulk_upload": {"max_failed_extractions": 1000, "max_containers_per_extraction": 10000}, "ratelimit": {"account_ratelimit": 0.0, "max_sleep_time_seconds": 60.0, "container_ratelimits": [], "container_listing_ratelimits": []}, "slo": {"max_manifest_segments": 1000, "yield_frequency": 10, "min_segment_size": 1, "max_manifest_size": 2097152}, "account_quotas": {}, "staticweb": {}, "swift": {"max_file_size": 5368709122, "account_listing_limit": 10000, "account_autocreate": true, "max_meta_count": 90, "max_meta_value_length": 256, "container_listing_limit": 10000, "extra_header_count": 0, "max_meta_overall_size": 4096, "version": "2.17.1.dev19", "max_meta_name_length": 128, "max_header_size": 8192, "policies": [{"default": true, "name": "Policy-0", "aliases": "Policy-0"}], "max_object_name_length": 1024, "max_account_name_length": 256, "strict_cors_mode": true, "allow_account_management": true, "max_container_name_length": 256}} 2019-04-16 02:14:01,322 85 DEBUG swiftclient REQ: curl -i http://10.0.0.110:8080/v1/AUTH_2b3073e9a97c426c801d581f9f94b227?format=json&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABctTVugQH8..." 2019-04-16 02:14:01,323 85 DEBUG swiftclient RESP STATUS: 200 OK 2019-04-16 02:14:01,323 85 DEBUG swiftclient RESP HEADERS: {u'Content-Length': u'98', u'X-Account-Object-Count': u'0', u'x-account-project-domain-id': u'default', u'X-Openstack-Request-Id': u'tx5d195dcc040448cb94814-005cb53a69', u'X-Account-Storage-Policy-Policy-0-Bytes-Used': u'0', u'X-Account-Storage-Policy-Policy-0-Container-Count': u'1', u'X-Timestamp': u'1554269586.84814', u'X-Account-Storage-Policy-Policy-0-Object-Count': u'0', u'X-Trans-Id': u'tx5d195dcc040448cb94814-005cb53a69', u'Date': u'Tue, 16 Apr 2019 02:14:01 GMT', u'X-Account-Bytes-Used': u'0', u'X-Account-Container-Count': u'1', u'Content-Type': u'application/json; charset=utf-8', u'Accept-Ranges': u'bytes'} 2019-04-16 02:14:01,323 85 DEBUG swiftclient RESP BODY: [{"count": 0, "last_modified": "2019-04-16T02:11:55.249080", "bytes": 0, "name": "case-02xxxx11"}] ---------------------------------------------------------------------------- CEPH BACKEND ---------------------------------------------------------------------------- 2019-04-16 03:25:59,375 85 INFO swiftclient REQ: curl -i http://10.0.0.6:8080/info -X GET -H "Accept-Encoding: gzip" 2019-04-16 03:25:59,377 85 INFO swiftclient RESP STATUS: 404 Not Found <================== 2019-04-16 03:25:59,377 85 INFO swiftclient RESP HEADERS: {u'Date': u'Tue, 16 Apr 2019 03:25:59 GMT', u'Content-Length': u'216', u'x-amz-request-id': u'tx0000000000000000db94c-005cb54b47-75d6b-default', u'Content-Type': u'application/xml', u'Accept-Ranges': u'bytes'} 2019-04-16 03:25:59,377 85 INFO swiftclient RESP BODY: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><BucketName>info</BucketName><RequestId>tx0000000000000000db94c-005cb54b47-75d6b-default</RequestId><HostId>75d6b-default-default</HostId></Error> 2019-04-16 03:25:59,378 83 DEBUG swiftclient REQ: curl -i http://10.0.0.6:8080/swift/v1?format=json&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABctUr58wN5..." <==== 2019-04-16 03:25:59,378 83 DEBUG swiftclient RESP STATUS: 200 OK 2019-04-16 03:25:59,378 83 DEBUG swiftclient RESP HEADERS: {u'Content-Length': u'50', u'X-Account-Storage-Policy-Default-Placement-Bytes-Used': u'25458', u'X-Account-Object-Count': u'2', u'X-Account-Storage-Policy-Default-Placement-Object-Count': u'2', u'X-Account-Storage-Policy-Default-Placement-Bytes-Used-Actual': u'28672', u'X-Timestamp': u'1555385159.37031', u'X-Account-Bytes-Used-Actual': u'28672', u'X-Trans-Id': u'tx0000000000000000976be-005cb54b47-8505f-default', u'Date': u'Tue, 16 Apr 2019 03:25:59 GMT', u'X-Account-Storage-Policy-Default-Placement-Container-Count': u'1', u'X-Account-Bytes-Used': u'25458', u'X-Account-Container-Count': u'1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx0000000000000000976be-005cb54b47-8505f-default'} 2019-04-16 03:25:59,378 83 DEBUG swiftclient RESP BODY: [{"name":"case-02xxxx11","count":2,"bytes":25458}] 2019-04-16 03:25:59,426 83 DEBUG swiftclient REQ: curl -i http://10.0.0.6:8080/swift/v1?format=json&marker=case-02xxxx11&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABctUr58wN5..." 2019-04-16 03:25:59,426 83 DEBUG swiftclient RESP STATUS: 200 OK <==== 2019-04-16 03:25:59,427 83 DEBUG swiftclient RESP HEADERS: {u'Content-Length': u'2', u'X-Account-Storage-Policy-Default-Placement-Bytes-Used': u'0', u'X-Account-Object-Count': u'0', u'X-Account-Storage-Policy-Default-Placement-Object-Count': u'0', u'X-Account-Storage-Policy-Default-Placement-Bytes-Used-Actual': u'0', u'X-Timestamp': u'1555385159.38575', u'X-Account-Bytes-Used-Actual': u'0', u'X-Trans-Id': u'tx0000000000000000976bf-005cb54b47-8505f-default', u'Date': u'Tue, 16 Apr 2019 03:25:59 GMT', u'X-Account-Storage-Policy-Default-Placement-Container-Count': u'0', u'X-Account-Bytes-Used': u'0', u'X-Account-Container-Count': u'0', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx0000000000000000976bf-005cb54b47-8505f-default'} 2019-04-16 03:25:59,427 83 DEBUG swiftclient RESP BODY: [] 2019-04-16 03:25:59,552 85 DEBUG swiftclient REQ: curl -i http://10.0.0.6:8080/swift/v1?format=json&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABctUr58wN5..." <==== 2019-04-16 03:25:59,552 85 DEBUG swiftclient RESP STATUS: 200 OK <==== 2019-04-16 03:25:59,553 85 DEBUG swiftclient RESP HEADERS: {u'Content-Length': u'50', u'X-Account-Storage-Policy-Default-Placement-Bytes-Used': u'25458', u'X-Account-Object-Count': u'2', u'X-Account-Storage-Policy-Default-Placement-Object-Count': u'2', u'X-Account-Storage-Policy-Default-Placement-Bytes-Used-Actual': u'28672', u'X-Timestamp': u'1555385159.54806', u'X-Account-Bytes-Used-Actual': u'28672', u'X-Trans-Id': u'tx0000000000000000db83e-005cb54b47-7b124-default', u'Date': u'Tue, 16 Apr 2019 03:25:59 GMT', u'X-Account-Storage-Policy-Default-Placement-Container-Count': u'1', u'X-Account-Bytes-Used': u'25458', u'X-Account-Container-Count': u'1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx0000000000000000db83e-005cb54b47-7b124-default'} 2019-04-16 03:25:59,553 85 DEBUG swiftclient RESP BODY: [{"name":"case-02xxxx11","count":2,"bytes":25458}] Executing the three steps below fixed the issue: now Horizon can access the container public links without the "X-Auth-Token: $TOKEN" header. 1) Enabling `rgw_swift_account_in_url` [client.rgw.rgwnode1] host = rgwnode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.rgwnode1/keyring rgw data = /var/lib/ceph/radosgw/ceph-rgw.rgwnode1 rgw socket path = /tmp/radosgw-rgwnode1.sock log file = /var/log/ceph/ceph-rgw-rgwnode1.log rgw frontends = civetweb port=192.168.128.34:80 num_threads=100 rgw_keystone_accepted_roles = admin, _member_, SwiftOperator rgw_keystone_admin_domain = default rgw_keystone_admin_password = <admin password> rgw_keystone_admin_project = admin rgw_keystone_admin_user = admin rgw_keystone_api_version = 3 rgw_keystone_implicit_tenants = True rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw keystone url = http://192.168.130.128:5000 rgw_swift_account_in_url = true <============= 2) Changing the swift endpoint to include info about the tenant ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep swift | 26526bdf95994c0a87702a85952b4ea0 | regionOne | swift | object-store | True | admin | http://172.17.3.21:8080/swift/v1 | | 51645c7cd1254e94b8820557cd372013 | regionOne | swift | object-store | True | public | http://10.0.0.109:8080/swift/v1 | | 81fa56836ee448d58f3ee3777593395a | regionOne | swift | object-store | True | internal | http://172.17.3.21:8080/swift/v1 | (overcloud) [stack@undercloud-0 ~]$openstack endpoint set --url 'http://172.17.3.21:8080/swift/v1/AUTH_%(tenant_id)s' 26526bdf95994c0a87702a85952b4ea0 (overcloud) [stack@undercloud-0 ~]$openstack endpoint set --url 'http://10.0.0.109:8080/swift/v1/AUTH_%(tenant_id)s' 51645c7cd1254e94b8820557cd372013 (overcloud) [stack@undercloud-0 ~]$openstack endpoint set --url 'http://172.17.3.21:8080/swift/v1/AUTH_%(tenant_id)s' 81fa56836ee448d58f3ee3777593395a (overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep swift | 26526bdf95994c0a87702a85952b4ea0 | regionOne | swift | object-store | True | admin | http://172.17.3.21:8080/swift/v1/AUTH_%(tenant_id)s | | 51645c7cd1254e94b8820557cd372013 | regionOne | swift | object-store | True | public | http://10.0.0.109:8080/swift/v1/AUTH_%(tenant_id)s | | 81fa56836ee448d58f3ee3777593395a | regionOne | swift | object-store | True | internal | http://172.17.3.21:8080/swift/v1/AUTH_%(tenant_id)s | ~~~ 3) Restarting the container(s) fixed the problem. ~~~ [root@controller-0 ~]# CONTAINERS=$(docker ps | grep -e ceph -e horizon | awk '{print $1}') [root@controller-0 ~]# for C in $CONTAINERS; do docker restart $C; done ~~~ (repeat per each controller) Now that we know how to fix this problem, let me ask a slightly different question. Using Swift, this behavior (i.e. access public link without "X-Auth-Token" header) works out of the box. However, with Ceph, this is not the case: clicking the check box "Public Access" from Horizon will create a public link... but the link is not accessible. If this is the spec, then let's close this ticket as NOTABUG, otherwise let's push this forward, so that when deploying Ceph with RHOSP: - `rgw_swift_account_in_url` is true by default. - the endpoint are configured so that they allow the above described behaviour Hi Matt, Right, not a bug. With implicit tenants being set to 'true', we also need 'rgw_swift_account_in_url' to be set along with proper endpoints for swift, like in #comment7. The relevant downstream doc seems to lack all this info. Will open a BZ and work with the doc team. https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/using_keystone_to_authenticate_ceph_object_gateway_users/configuring_the_ceph_object_gateway Closing as this is not a bug. RFE ticket[1] was opened to have the workflow described in #7 done automatically on deploying. For the time being, a doc BZ[2] was opened by shilpa to update the docs with the current workaround [1] https://bugzilla.redhat.com/show_bug.cgi?id=1717320 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1719104 |
Description of problem: This issue seems to be somehow similar to [1]. - When accessing a public container from Horizon the token is not passed and `No Suck Bucket` error is returned instead. - In this case I've veried that the browser is not having "X-Auth-Token" header set. - I've tried manually adding "X-Auth-Token: $OS_TOKEN" in the browser and the issue was gone - This works from CLI ("X-Auth-Token" is set automatically) - This does not happen with the exactly same setup when using swift as a backend. I suppose this is cause by Ceph <> Horizon integration not being setup correctly (i.e. Ceph returns the direct URL of the container instead of the URL used to authenticate the request, or Horizon does not know that has to pass the token in the header). [1] https://bugzilla.redhat.com/show_bug.cgi?id=1588681 Version-Release number of selected component (if applicable): - RHOSP13 - ceph version 12.2.8-52.el7cp (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable) How reproducible: 100% of the time Steps to Reproduce: 1. Create a public container (using cli or Horizon to create the container, won't affect the outcome) 2. Access the container link through Horizon Actual results: CLI works out of the box $ curl -vv -s -H "X-Auth-Token: $OS_TOKEN" http://localhost:8080/swift/v1/case_0xx35xxx_container/test.txt * About to connect() to localhost port 8080 (#0) * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > GET /swift/v1/case_0xx35xxx_container/test.txt HTTP/1.1 > User-Agent: curl/7.29.0 > Host: localhost:8080 > Accept: */* > X-Auth-Token: gAAAAABcrVne6MVZ9Bf-mP5o_e9KH6hVRsfdQAuiXza45KqFqpcUJqsST2Kt2khuhasr2D0Eqha-Oh6F9Gx6E_uVqsmv5JXa_A0fk5MyL6IfbLhDd88pje2vuIGAkcFDQvYY7_iAyZ7mnxHN8DDHNOHseykhHwCLVZS3AQWRjEC8Wd0b4CzAp2w > < HTTP/1.1 200 OK < Content-Length: 13 < Accept-Ranges: bytes < Last-Modified: Wed, 10 Apr 2019 00:51:16 GMT < X-Timestamp: 1554857476.51452 < etag: 59ca0efa9f5633cb0371bbc0355478d8 < X-Object-Meta-Orig-Filename: test.txt < X-Trans-Id: tx00000000000000001eb62-005cad5a4c-75d6b-default < X-Openstack-Request-Id: tx00000000000000001eb62-005cad5a4c-75d6b-default < Content-Type: text/plain < Date: Wed, 10 Apr 2019 02:51:56 GMT < Hello world! However, accessing Horizon from `http://localhost:8080/swift/v1/case_0xx35xxx_container/test.txt` shows `NoSuchBucket` Expected results: - Accessing the same link from Horizon should show Additional info: - I am able to replicate the issue in my environment. I'll attache the stacktrace from both swift and from ceph environments asap, so that we can compare the differences and pin down why Horizon does not pass the token in the header when used in combination with Ceph.