Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1653314

Summary: [RGW]: Manual resharding of a versioned bucket, causes a bucket stats discrepancy
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tejas <tchandra>
Component: RGWAssignee: J. Eric Ivancich <ivancich>
Status: CLOSED ERRATA QA Contact: Tejas <tchandra>
Severity: high Docs Contact:
Priority: high    
Version: 3.2CC: cbodley, ceph-eng-bugs, hnallurv, kbader, mbenjamin, sweil, tchandra, tserlin
Target Milestone: rc   
Target Release: 3.2   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-12.2.8-47.el7cp Ubuntu: ceph_12.2.8-43redhat1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-03 19:02:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tejas 2018-11-26 14:21:56 UTC 
Description of problem:
    On a 3.2 ceph build, if we manually reshard a versioned bucket, the bucket stats is incorrect after the reshard.

Version-Release number of selected component (if applicable):
ceph version 12.2.8-39.el7cp

How reproducible:
Always

Steps to Reproduce:
1. Create a versioned bucket and upload few objects.
    root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#2647",
    "master_ver": "0#0",
    "mtime": "2018-11-26 09:29:12.194358",
    "max_marker": "0#",
    "usage": {
        "rgw.main": {
            "size": 10584,
            "size_actual": 3612672,
            "size_utilized": 10584,
            "size_kb": 11,
            "size_kb_actual": 3528,
            "size_kb_utilized": 11,
            "num_objects": 882
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

2.  Reshard the bucket: root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=2 --yes-i-really-mean-it
tenant: 
bucket name: new2
old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.47876.3
new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1
total entries: 1000 2000 3000 3530

3.  Stats incorrect:
   root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#30,1#28",
    "master_ver": "0#0,1#0",
    "mtime": "2018-11-26 12:37:20.880362",
    "max_marker": "0#,1#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 882
        },
        "rgw.main": {
            "size": 21180,
            "size_actual": 7229440,
            "size_utilized": 21180,
            "size_kb": 21,
            "size_kb_actual": 7060,
            "size_kb_utilized": 21,
            "num_objects": 1765    <------- doubled
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
4. upload few more objects and reshard again:

      root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#768,1#787",
    "master_ver": "0#0,1#0",
    "mtime": "2018-11-26 12:37:20.880362",
    "max_marker": "0#,1#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 882
        },
        "rgw.main": {
            "size": 27168,
            "size_actual": 9273344,
            "size_utilized": 27168,
            "size_kb": 27,
            "size_kb_actual": 9056,
            "size_kb_utilized": 27,
            "num_objects": 2264     <---- 500 objects uploaded
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}
root@magna090 ~]# radosgw-admin bucket stats --bucket new2
{
    "bucket": "new2",
    "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "bccd78d5-5017-4859-b321-b78a977a4057.51713.1",
    "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3",
    "index_type": "Normal",
    "owner": "local1",
    "ver": "0#25,1#23,2#22,3#22",
    "master_ver": "0#0,1#0,2#0,3#0",
    "mtime": "2018-11-26 13:10:46.014517",
    "max_marker": "0#,1#,2#,3#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 1381
        },
        "rgw.main": {
            "size": 33168,
            "size_actual": 11321344,
            "size_utilized": 33168,
            "size_kb": 33,
            "size_kb_actual": 11056,
            "size_kb_utilized": 33,
            "num_objects": 2764    <--- 500 extra updated
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

 root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=4
tenant: 
bucket name: new2
old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1
new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51713.1
total entries: 1000 2000 3000 4000 5000 5526


As a result user stats is also incorrect:
root@magna090 ~]# radosgw-admin user stats --uid=local1
{
    "stats": {
        "total_entries": 6144,
        "total_bytes": 63144,
        "total_bytes_rounded": 21553152
    },
    "last_stats_sync": "2018-11-26 12:29:35.739406Z",
    "last_stats_update": "2018-11-26 13:10:47.065586Z"
}
Comment 5 J. Eric Ivancich 2018-11-30 04:32:42 UTC 
Tentative upstream PR solution in https://github.com/ceph/ceph/pull/25333 .
Comment 19 errata-xmlrpc 2019-01-03 19:02:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0020