Bug 1430650
| Summary: | Stale short zone id's are left behind after deleting a zone in multisite env | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | shilpa <smanjara> |
| Component: | RGW | Assignee: | Casey Bodley <cbodley> |
| Status: | CLOSED ERRATA | QA Contact: | Tejas <tchandra> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.2 | CC: | cbodley, ceph-eng-bugs, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil |
| Target Milestone: | rc | ||
| Target Release: | 2.3 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | RHEL: ceph-10.2.7-2.el7cp Ubuntu: ceph_10.2.7-3redhat1xenial | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-06-19 13:30:20 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(In reply to shilpa from comment #0) > Description of problem: > Configure multisite with master and a non-master zone. Remove the non-master > zone from zonegroup and delete it. Short zone id entry in period is still > left behind > > Version-Release number of selected component (if applicable): > ceph-radosgw-10.2.5-37.el7cp.x86_64 > > How reproducible: > Always > > Steps to Reproduce: > 1. Remove zone from zonegroup > 2. Do period update --commit > 3. Delete the zone > 4. Check period conf file > > > Actual results: > Sorry the output is: "master_zone": "b8710807-0108-4845-b7f5-d3eb8c58e4de", "zones": [ { "id": "b8710807-0108-4845-b7f5-d3eb8c58e4de", "name": "us-east", "endpoints": [ "http:\/\/rgw2:8080" ], "log_meta": "true", "log_data": "false", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1407b5ff-602f-44cb-be73-0752f862f01f" } ], "short_zone_ids": [ { "key": "b532ed0a-8d79-452b-bb8e-d78476a78431", "val": 2927679014 }, { "key": "b8710807-0108-4845-b7f5-d3eb8c58e4de", "val": 2693931929 } ] }, Hi Shilpa, I raised this issue upstream a while ago in http://tracker.ceph.com/issues/15618, and the consensus at the time was to leave them in the period. These short zone ids are stored with objects that are written to that zone, and are used in sync logic to break ties when objects of the same name are uploaded to different zones - when this happens, all zones will end up with the object with the highest short zone id. When zones are created, we generate their short id by calculating the MD5 hash of its uuid. Since this isn't a cryptographic hash, we need to be able to prevent collisions where different zone uuids hash to the same short zone id. So we decided to keep old zone short ids in the period, so that we can use them to reject other zones that would collide. However, I do worry that large deployments, or those that use scripts to add/remove zones regularly, could see this short_zone_ids list grow quite large over time. So I would like to see some further investigation into whether a new zone that reuses a removed zone's short id would actually cause us to sync the wrong objects. This was discussed in today's bug scrub. We managed to convince ourselves that reusing old zone short ids is okay, and that we don't need to store them in the period indefinitely. I've reopened the upstream tracker issue and plan to fix for 2.3. This fix was merged upstream in https://github.com/ceph/ceph/pull/13949. Resolved in v10.2.7 upstream. Steps followed: 1. ina 3 way multisite setup, created a zone and did a period commit. 2. Removed the zone from the zonegroup, and did a period commit. 3. There was no stale short_zone_id left behind. Verified on build: ceph version 10.2.7-13.el7cp (4955aa6a90abc27bc043729db19df24e1c840eac) Thanks, Tejas Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1497 |
Description of problem: Configure multisite with master and a non-master zone. Remove the non-master zone from zonegroup and delete it. Short zone id entry in period is still left behind Version-Release number of selected component (if applicable): ceph-radosgw-10.2.5-37.el7cp.x86_64 How reproducible: Always Steps to Reproduce: 1. Remove zone from zonegroup 2. Do period update --commit 3. Delete the zone 4. Check period conf file Actual results: Zone id that is deleted was : "b532ed0a-8d79-452b-bb8e-d78476a78431" "master_zone": "b8710807-0108-4845-b7f5-d3eb8c58e4de", "zones": [ { "id": "b532ed0a-8d79-452b-bb8e-d78476a78431", "name": "us-west", "endpoints": [ "http:\/\/rgw1:8080" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false" }, { "id": "b8710807-0108-4845-b7f5-d3eb8c58e4de", "name": "us-east", "endpoints": [ "http:\/\/rgw2:8080" ], "log_meta": "true", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false" } ], "placement_targets": [ { "name": "default-placement", "tags": [] } ], "default_placement": "default-placement", "realm_id": "1407b5ff-602f-44cb-be73-0752f862f01f" } ], "short_zone_ids": [ { "key": "b532ed0a-8d79-452b-bb8e-d78476a78431", "val": 2927679014 }, { "key": "b8710807-0108-4845-b7f5-d3eb8c58e4de", "val": 2693931929 } ] Expected results: The zone id should be removed