Bug 1430650

Summary: Stale short zone id's are left behind after deleting a zone in multisite env
Product: Red Hat Ceph Storage Reporter: shilpa <smanjara>
Component: RGWAssignee: Casey Bodley <cbodley>
Status: CLOSED ERRATA QA Contact: Tejas <tchandra>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.2CC: cbodley, ceph-eng-bugs, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil
Target Milestone: rc   
Target Release: 2.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.7-2.el7cp Ubuntu: ceph_10.2.7-3redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-19 13:30:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description shilpa 2017-03-09 08:41:20 UTC
Description of problem:
Configure multisite with master and a non-master zone. Remove the non-master zone from zonegroup and delete it. Short zone id entry in period is still left behind

Version-Release number of selected component (if applicable):
ceph-radosgw-10.2.5-37.el7cp.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Remove zone from zonegroup
2. Do period update --commit
3. Delete the zone
4. Check period conf file


Actual results:

Zone id that is deleted was : "b532ed0a-8d79-452b-bb8e-d78476a78431"

            "master_zone": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "zones": [
                    {
                        "id": "b532ed0a-8d79-452b-bb8e-d78476a78431",
                        "name": "us-west",
                        "endpoints": [
                            "http:\/\/rgw1:8080"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    },
                    {
                        "id": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                        "name": "us-east",
                        "endpoints": [
                            "http:\/\/rgw2:8080"
                        ],
                        "log_meta": "true",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "1407b5ff-602f-44cb-be73-0752f862f01f"
            }
        ],
        "short_zone_ids": [
            {
                "key": "b532ed0a-8d79-452b-bb8e-d78476a78431",
                "val": 2927679014
            },
            {
                "key": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "val": 2693931929
            }
        ]


Expected results:
The zone id should be removed

Comment 2 shilpa 2017-03-09 08:48:16 UTC
(In reply to shilpa from comment #0)
> Description of problem:
> Configure multisite with master and a non-master zone. Remove the non-master
> zone from zonegroup and delete it. Short zone id entry in period is still
> left behind
> 
> Version-Release number of selected component (if applicable):
> ceph-radosgw-10.2.5-37.el7cp.x86_64
> 
> How reproducible:
> Always
> 
> Steps to Reproduce:
> 1. Remove zone from zonegroup
> 2. Do period update --commit
> 3. Delete the zone
> 4. Check period conf file
> 
> 
> Actual results:
> 

Sorry the output is:

              "master_zone": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "zones": [
                    {
                        "id": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                        "name": "us-east",
                        "endpoints": [
                            "http:\/\/rgw2:8080"
                        ],
                        "log_meta": "true",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "1407b5ff-602f-44cb-be73-0752f862f01f"
            }
        ],
        "short_zone_ids": [
            {               
                "key": "b532ed0a-8d79-452b-bb8e-d78476a78431",
                "val": 2927679014
            },
            {
                "key": "b8710807-0108-4845-b7f5-d3eb8c58e4de",
                "val": 2693931929
            }
        ]
    },

Comment 3 Casey Bodley 2017-03-09 16:17:43 UTC
Hi Shilpa,

I raised this issue upstream a while ago in http://tracker.ceph.com/issues/15618, and the consensus at the time was to leave them in the period.

These short zone ids are stored with objects that are written to that zone, and are used in sync logic to break ties when objects of the same name are uploaded to different zones - when this happens, all zones will end up with the object with the highest short zone id.

When zones are created, we generate their short id by calculating the MD5 hash of its uuid. Since this isn't a cryptographic hash, we need to be able to prevent collisions where different zone uuids hash to the same short zone id. So we decided to keep old zone short ids in the period, so that we can use them to reject other zones that would collide.

However, I do worry that large deployments, or those that use scripts to add/remove zones regularly, could see this short_zone_ids list grow quite large over time. So I would like to see some further investigation into whether a new zone that reuses a removed zone's short id would actually cause us to sync the wrong objects.

Comment 4 Casey Bodley 2017-03-09 19:07:41 UTC
This was discussed in today's bug scrub. We managed to convince ourselves that reusing old zone short ids is okay, and that we don't need to store them in the period indefinitely. I've reopened the upstream tracker issue and plan to fix for 2.3.

Comment 5 Casey Bodley 2017-03-14 18:31:36 UTC
This fix was merged upstream in https://github.com/ceph/ceph/pull/13949.

Comment 7 Ken Dreyer (Red Hat) 2017-04-06 23:09:08 UTC
Resolved in v10.2.7 upstream.

Comment 11 Tejas 2017-05-02 09:17:28 UTC
Steps followed:
1. ina 3 way multisite setup, created a zone and did a period commit.
2. Removed the zone from the zonegroup, and did a period commit.
3. There was no stale short_zone_id  left behind.

Verified on build:
ceph version 10.2.7-13.el7cp (4955aa6a90abc27bc043729db19df24e1c840eac)

Thanks,
Tejas

Comment 13 errata-xmlrpc 2017-06-19 13:30:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497