Bug 1653314 - [RGW]: Manual resharding of a versioned bucket, causes a bucket stats discrepancy
Summary: [RGW]: Manual resharding of a versioned bucket, causes a bucket stats discrep...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 3.2
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: rc
: 3.2
Assignee: J. Eric Ivancich
QA Contact: Tejas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-26 14:21 UTC by Tejas
Modified: 2019-01-03 19:02 UTC (History)
8 users (show)

Fixed In Version: RHEL: ceph-12.2.8-47.el7cp Ubuntu: ceph_12.2.8-43redhat1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-03 19:02:25 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 37473 None None None 2018-11-30 12:52:35 UTC
Red Hat Product Errata RHBA-2019:0020 None None None 2019-01-03 19:02:34 UTC

Description Tejas 2018-11-26 14:21:56 UTC
Description of problem:
    On a 3.2 ceph build, if we manually reshard a versioned bucket, the bucket stats is incorrect after the reshard.

Version-Release number of selected component (if applicable):
ceph version 12.2.8-39.el7cp

How reproducible:
Always

Steps to Reproduce:
1. Create a versioned bucket and upload few objects.
    root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#2647",
    "master_ver": "0#0",
    "mtime": "2018-11-26 09:29:12.194358",
    "max_marker": "0#",
    "usage": {
        "rgw.main": {
            "size": 10584,
            "size_actual": 3612672,
            "size_utilized": 10584,
            "size_kb": 11,
            "size_kb_actual": 3528,
            "size_kb_utilized": 11,
            "num_objects": 882
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

2.  Reshard the bucket: root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=2 --yes-i-really-mean-it
tenant: 
bucket name: new2
old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.47876.3
new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1
total entries: 1000 2000 3000 3530

3.  Stats incorrect:
   root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#30,1#28",
    "master_ver": "0#0,1#0",
    "mtime": "2018-11-26 12:37:20.880362",
    "max_marker": "0#,1#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 882
        },
        "rgw.main": {
            "size": 21180,
            "size_actual": 7229440,
            "size_utilized": 21180,
            "size_kb": 21,
            "size_kb_actual": 7060,
            "size_kb_utilized": 21,
            "num_objects": 1765    <------- doubled
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
4. upload few more objects and reshard again:

      root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#768,1#787",
    "master_ver": "0#0,1#0",
    "mtime": "2018-11-26 12:37:20.880362",
    "max_marker": "0#,1#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 882
        },
        "rgw.main": {
            "size": 27168,
            "size_actual": 9273344,
            "size_utilized": 27168,
            "size_kb": 27,
            "size_kb_actual": 9056,
            "size_kb_utilized": 27,
            "num_objects": 2264     <---- 500 objects uploaded
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}
root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.51713.1",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#25,1#23,2#22,3#22",
    "master_ver": "0#0,1#0,2#0,3#0",
    "mtime": "2018-11-26 13:10:46.014517",
    "max_marker": "0#,1#,2#,3#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 1381
        },
        "rgw.main": {
            "size": 33168,
            "size_actual": 11321344,
            "size_utilized": 33168,
            "size_kb": 33,
            "size_kb_actual": 11056,
            "size_kb_utilized": 33,
            "num_objects": 2764    <--- 500 extra updated
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

 root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=4
tenant: 
bucket name: new2
old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1
new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51713.1
total entries: 1000 2000 3000 4000 5000 5526


As a result user stats is also incorrect:
root@magna090 ~]# radosgw-admin user stats --uid=local1
{
    "stats": {
        "total_entries": 6144,
        "total_bytes": 63144,
        "total_bytes_rounded": 21553152
    },
    "last_stats_sync": "2018-11-26 12:29:35.739406Z",
    "last_stats_update": "2018-11-26 13:10:47.065586Z"
}

Comment 5 J. Eric Ivancich 2018-11-30 04:32:42 UTC
Tentative upstream PR solution in https://github.com/ceph/ceph/pull/25333 .

Comment 19 errata-xmlrpc 2019-01-03 19:02:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0020


Note You need to log in before you can comment on or make changes to this bug.