Bug 1655842

Summary: [RGW]: Reshard on versioned buckets causes inconsistency in user stats
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tejas <tchandra>
Component: RGWAssignee: J. Eric Ivancich <ivancich>
Status: CLOSED ERRATA QA Contact: Tejas <tchandra>
Severity: high Docs Contact:
Priority: high    
Version: 3.2CC: cbodley, ceph-eng-bugs, hnallurv, kbader, mbenjamin, sweil, tserlin
Target Milestone: rc   
Target Release: 3.2   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-12.2.8-50.el7cp Ubuntu: ceph_12.2.8-45redhat1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-03 19:02:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tejas 2018-12-04 04:23:28 UTC
Description of problem:
    This is in continuation to https://bugzilla.redhat.com/show_bug.cgi?id=1653314 .
Upon resharding of versioned buckets, we are observing that the user stats are inconsistent with bucket stats in the num_objects.

Version-Release number of selected component (if applicable):
ceph version 12.2.8-43redhat1xenial

~# radosgw-admin bucket stats --bucket=newb1
{
    "bucket": "newb1",
    "zonegroup": "4a07694a-5f4e-4f2b-a8d2-cccb20fbc7fd",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "f628bdd7-d5e7-45cc-a276-a3d3ef7f493f.234811.1",
    "marker": "f628bdd7-d5e7-45cc-a276-a3d3ef7f493f.231656.3",
    "index_type": "Normal",
    "owner": "new1",
    "ver": "0#44,1#41,2#38,3#39",
    "master_ver": "0#0,1#0,2#0,3#0",
    "mtime": "2018-12-03 08:41:36.423787",
    "max_marker": "0#,1#,2#,3#",
    "usage": {
        "rgw.none": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 2000
        },
        "rgw.main": {
            "size": 36000,
            "size_actual": 12288000,
            "size_utilized": 36000,
            "size_kb": 36,
            "size_kb_actual": 12000,
            "size_kb_utilized": 36,
            "num_objects": 3000    <---- correct number
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}
~# radosgw-admin user stats --uid new1
{
    "stats": {
        "total_entries": 4000,   <--- incorrect
        "total_bytes": 36000,
        "total_bytes_rounded": 12288000
    },
    "last_stats_sync": "0.000000",
    "last_stats_update": "2018-12-03 08:41:37.795730Z"
}

Created another non-versioned bucket with 2000objs.

~# radosgw-admin user stats --uid new1
{
    "stats": {
        "total_entries": 6000,
        "total_bytes": 60000,
        "total_bytes_rounded": 20480000
    },
    "last_stats_sync": "0.000000",
    "last_stats_update": "2018-12-04 03:52:46.475709Z"
}

Sync stats made it worse:

:~# radosgw-admin user stats --uid new1 --sync-stats
{
    "stats": {
        "total_entries": 7000,
        "total_bytes": 60000,
        "total_bytes_rounded": 20480000
    },
    "last_stats_sync": "2018-12-04 03:58:34.420115Z",
    "last_stats_update": "2018-12-04 03:58:34.409069Z"
}

Comment 8 errata-xmlrpc 2019-01-03 19:02:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0020