Bug 1653314
| Summary: | [RGW]: Manual resharding of a versioned bucket, causes a bucket stats discrepancy | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Tejas <tchandra> |
| Component: | RGW | Assignee: | J. Eric Ivancich <ivancich> |
| Status: | CLOSED ERRATA | QA Contact: | Tejas <tchandra> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.2 | CC: | cbodley, ceph-eng-bugs, hnallurv, kbader, mbenjamin, sweil, tchandra, tserlin |
| Target Milestone: | rc | ||
| Target Release: | 3.2 | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | RHEL: ceph-12.2.8-47.el7cp Ubuntu: ceph_12.2.8-43redhat1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-01-03 19:02:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Tentative upstream PR solution in https://github.com/ceph/ceph/pull/25333 . Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0020 |
Description of problem: On a 3.2 ceph build, if we manually reshard a versioned bucket, the bucket stats is incorrect after the reshard. Version-Release number of selected component (if applicable): ceph version 12.2.8-39.el7cp How reproducible: Always Steps to Reproduce: 1. Create a versioned bucket and upload few objects. root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#2647", "master_ver": "0#0", "mtime": "2018-11-26 09:29:12.194358", "max_marker": "0#", "usage": { "rgw.main": { "size": 10584, "size_actual": 3612672, "size_utilized": 10584, "size_kb": 11, "size_kb_actual": 3528, "size_kb_utilized": 11, "num_objects": 882 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } 2. Reshard the bucket: root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=2 --yes-i-really-mean-it tenant: bucket name: new2 old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.47876.3 new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1 total entries: 1000 2000 3000 3530 3. Stats incorrect: root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#30,1#28", "master_ver": "0#0,1#0", "mtime": "2018-11-26 12:37:20.880362", "max_marker": "0#,1#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 882 }, "rgw.main": { "size": 21180, "size_actual": 7229440, "size_utilized": 21180, "size_kb": 21, "size_kb_actual": 7060, "size_kb_utilized": 21, "num_objects": 1765 <------- doubled } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } 4. upload few more objects and reshard again: root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#768,1#787", "master_ver": "0#0,1#0", "mtime": "2018-11-26 12:37:20.880362", "max_marker": "0#,1#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 882 }, "rgw.main": { "size": 27168, "size_actual": 9273344, "size_utilized": 27168, "size_kb": 27, "size_kb_actual": 9056, "size_kb_utilized": 27, "num_objects": 2264 <---- 500 objects uploaded } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.51713.1", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#25,1#23,2#22,3#22", "master_ver": "0#0,1#0,2#0,3#0", "mtime": "2018-11-26 13:10:46.014517", "max_marker": "0#,1#,2#,3#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 1381 }, "rgw.main": { "size": 33168, "size_actual": 11321344, "size_utilized": 33168, "size_kb": 33, "size_kb_actual": 11056, "size_kb_utilized": 33, "num_objects": 2764 <--- 500 extra updated } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=4 tenant: bucket name: new2 old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1 new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51713.1 total entries: 1000 2000 3000 4000 5000 5526 As a result user stats is also incorrect: root@magna090 ~]# radosgw-admin user stats --uid=local1 { "stats": { "total_entries": 6144, "total_bytes": 63144, "total_bytes_rounded": 21553152 }, "last_stats_sync": "2018-11-26 12:29:35.739406Z", "last_stats_update": "2018-11-26 13:10:47.065586Z" }