Bug 1430650 - Stale short zone id's are left behind after deleting a zone in multisite env
Summary: Stale short zone id's are left behind after deleting a zone in multisite env
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 2.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 2.3
Assignee: Casey Bodley
QA Contact: Tejas
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-09 08:41 UTC by shilpa
Modified: 2017-07-30 15:41 UTC (History)
8 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2017-06-19 13:30:20 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1497 normal SHIPPED_LIVE Red Hat Ceph Storage 2.3 bug fix and enhancement update 2017-06-19 17:24:11 UTC
Ceph Project Bug Tracker 15618 None None None 2017-03-09 16:17 UTC

Description shilpa 2017-03-09 08:41:20 UTC
Description of problem:
Configure multisite with master and a non-master zone. Remove the non-master zone from zonegroup and delete it. Short zone id entry in period is still left behind

Version-Release number of selected component (if applicable):
ceph-radosgw-10.2.5-37.el7cp.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Remove zone from zonegroup
2. Do period update --commit
3. Delete the zone
4. Check period conf file


Actual results:

Zone id that is deleted was : "b532ed0a-8d79-452b-bb8e-d78476a78431"

            "master_zone": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "zones": [
                    {
                        "id": "b532ed0a-8d79-452b-bb8e-d78476a78431",
                        "name": "us-west",
                        "endpoints": [
                            "http:\/\/rgw1:8080"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    },
                    {
                        "id": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                        "name": "us-east",
                        "endpoints": [
                            "http:\/\/rgw2:8080"
                        ],
                        "log_meta": "true",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "1407b5ff-602f-44cb-be73-0752f862f01f"
            }
        ],
        "short_zone_ids": [
            {
                "key": "b532ed0a-8d79-452b-bb8e-d78476a78431",
                "val": 2927679014
            },
            {
                "key": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "val": 2693931929
            }
        ]


Expected results:
The zone id should be removed

Comment 2 shilpa 2017-03-09 08:48:16 UTC
(In reply to shilpa from comment #0)
> Description of problem:
> Configure multisite with master and a non-master zone. Remove the non-master
> zone from zonegroup and delete it. Short zone id entry in period is still
> left behind
> 
> Version-Release number of selected component (if applicable):
> ceph-radosgw-10.2.5-37.el7cp.x86_64
> 
> How reproducible:
> Always
> 
> Steps to Reproduce:
> 1. Remove zone from zonegroup
> 2. Do period update --commit
> 3. Delete the zone
> 4. Check period conf file
> 
> 
> Actual results:
> 

Sorry the output is:

              "master_zone": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "zones": [
                    {
                        "id": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                        "name": "us-east",
                        "endpoints": [
                            "http:\/\/rgw2:8080"
                        ],
                        "log_meta": "true",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "1407b5ff-602f-44cb-be73-0752f862f01f"
            }
        ],
        "short_zone_ids": [
            {               
                "key": "b532ed0a-8d79-452b-bb8e-d78476a78431",
                "val": 2927679014
            },
            {
                "key": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "val": 2693931929
            }
        ]
    },

Comment 3 Casey Bodley 2017-03-09 16:17:43 UTC
Hi Shilpa,

I raised this issue upstream a while ago in http://tracker.ceph.com/issues/15618, and the consensus at the time was to leave them in the period.

These short zone ids are stored with objects that are written to that zone, and are used in sync logic to break ties when objects of the same name are uploaded to different zones - when this happens, all zones will end up with the object with the highest short zone id.

When zones are created, we generate their short id by calculating the MD5 hash of its uuid. Since this isn't a cryptographic hash, we need to be able to prevent collisions where different zone uuids hash to the same short zone id. So we decided to keep old zone short ids in the period, so that we can use them to reject other zones that would collide.

However, I do worry that large deployments, or those that use scripts to add/remove zones regularly, could see this short_zone_ids list grow quite large over time. So I would like to see some further investigation into whether a new zone that reuses a removed zone's short id would actually cause us to sync the wrong objects.

Comment 4 Casey Bodley 2017-03-09 19:07:41 UTC
This was discussed in today's bug scrub. We managed to convince ourselves that reusing old zone short ids is okay, and that we don't need to store them in the period indefinitely. I've reopened the upstream tracker issue and plan to fix for 2.3.

Comment 5 Casey Bodley 2017-03-14 18:31:36 UTC
This fix was merged upstream in https://github.com/ceph/ceph/pull/13949.

Comment 7 Ken Dreyer (Red Hat) 2017-04-06 23:09:08 UTC
Resolved in v10.2.7 upstream.

Comment 11 Tejas 2017-05-02 09:17:28 UTC
Steps followed:
1. ina 3 way multisite setup, created a zone and did a period commit.
2. Removed the zone from the zonegroup, and did a period commit.
3. There was no stale short_zone_id  left behind.

Verified on build:
ceph version 10.2.7-13.el7cp (4955aa6a90abc27bc043729db19df24e1c840eac)

Thanks,
Tejas

Comment 13 errata-xmlrpc 2017-06-19 13:30:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497


Note You need to log in before you can comment on or make changes to this bug.