Hide Forgot
Description of problem: On a 3.2 ceph build, if we manually reshard a versioned bucket, the bucket stats is incorrect after the reshard. Version-Release number of selected component (if applicable): ceph version 12.2.8-39.el7cp How reproducible: Always Steps to Reproduce: 1. Create a versioned bucket and upload few objects. root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#2647", "master_ver": "0#0", "mtime": "2018-11-26 09:29:12.194358", "max_marker": "0#", "usage": { "rgw.main": { "size": 10584, "size_actual": 3612672, "size_utilized": 10584, "size_kb": 11, "size_kb_actual": 3528, "size_kb_utilized": 11, "num_objects": 882 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } 2. Reshard the bucket: root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=2 --yes-i-really-mean-it tenant: bucket name: new2 old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.47876.3 new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1 total entries: 1000 2000 3000 3530 3. Stats incorrect: root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#30,1#28", "master_ver": "0#0,1#0", "mtime": "2018-11-26 12:37:20.880362", "max_marker": "0#,1#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 882 }, "rgw.main": { "size": 21180, "size_actual": 7229440, "size_utilized": 21180, "size_kb": 21, "size_kb_actual": 7060, "size_kb_utilized": 21, "num_objects": 1765 <------- doubled } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } 4. upload few more objects and reshard again: root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.51444.1", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#768,1#787", "master_ver": "0#0,1#0", "mtime": "2018-11-26 12:37:20.880362", "max_marker": "0#,1#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 882 }, "rgw.main": { "size": 27168, "size_actual": 9273344, "size_utilized": 27168, "size_kb": 27, "size_kb_actual": 9056, "size_kb_utilized": 27, "num_objects": 2264 <---- 500 objects uploaded } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } root@magna090 ~]# radosgw-admin bucket stats --bucket new2 { "bucket": "new2", "zonegroup": "d6822e5f-fd16-49fe-b579-37aebe61866b", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "bccd78d5-5017-4859-b321-b78a977a4057.51713.1", "marker": "bccd78d5-5017-4859-b321-b78a977a4057.47876.3", "index_type": "Normal", "owner": "local1", "ver": "0#25,1#23,2#22,3#22", "master_ver": "0#0,1#0,2#0,3#0", "mtime": "2018-11-26 13:10:46.014517", "max_marker": "0#,1#,2#,3#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 1381 }, "rgw.main": { "size": 33168, "size_actual": 11321344, "size_utilized": 33168, "size_kb": 33, "size_kb_actual": 11056, "size_kb_utilized": 33, "num_objects": 2764 <--- 500 extra updated } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } root@magna090 ~]# radosgw-admin bucket reshard --bucket new2 --num-shards=4 tenant: bucket name: new2 old bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51444.1 new bucket instance id: bccd78d5-5017-4859-b321-b78a977a4057.51713.1 total entries: 1000 2000 3000 4000 5000 5526 As a result user stats is also incorrect: root@magna090 ~]# radosgw-admin user stats --uid=local1 { "stats": { "total_entries": 6144, "total_bytes": 63144, "total_bytes_rounded": 21553152 }, "last_stats_sync": "2018-11-26 12:29:35.739406Z", "last_stats_update": "2018-11-26 13:10:47.065586Z" }
Tentative upstream PR solution in https://github.com/ceph/ceph/pull/25333 .
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0020