Bug 2239831

thanks for the logs Vidushi.

analyzed one of the bucket index shards '.dir.8dc62baa-afc5-496b-9ac1-692c719856ff.324610.4.1.5' for newcontainer-3 that has a large omap with > 200k entries.

# rados -p archive.rgw.buckets.index listomapkeys .dir.8dc62baa-afc5-496b-9ac1-692c719856ff.324610.4.1.5 ls | wc -l
229646

the following two reasons could be the cause:

1. I see that both primary and secondary zones have resharded the bucket 'newcontainer-3' once with num_shards~=127 respectively, which is correct. 
whereas on the archive zone, bucket has been dynamically resharded to only num_shards=29.

in line https://github.com/ceph/ceph/blob/main/src/rgw/rgw_quota.cc#L976, for multisite configurations we set the obj_multiplier to 8 instead of 2 to increase the shard count faster to reduce the number of reshard events required. But recently we disabled data logging in archive zone setting this bool to false in RGWRados::check_bucket_shards(), thus defaulting to the non-multisite reshard multiplier of 2, which is probably why the bucket is resharded to only 29 shards.


2. Of the listomapkey entries, you may also see thousands of non UTF-8 entries like this:

�1000_secobj-57783iG4G41Amnk8UmnZw3YuVRuqSJ4iAn9Ea
�1000_secobj-577847iRSEz-yDQ6qU0H66l5KnMpJOvuHJff6Z
�1000_secobj-577848inE1jic4gmSLkz2Jx6n.3.ohFdgapd4M

these "<0x80>1000_" are special bucket index entries (BI_BUCKET_OBJ_INSTANCE_INDEX) that are created for each version of the object uploaded. so if you rewrite an object n times, then we are going to have n number of these special entries other than the listable object versions.

(In reply to shilpa from comment #12)
> thanks for the logs Vidushi.
> 
> analyzed one of the bucket index shards
> '.dir.8dc62baa-afc5-496b-9ac1-692c719856ff.324610.4.1.5' for newcontainer-3
> that has a large omap with > 200k entries.
> 
> # rados -p archive.rgw.buckets.index listomapkeys
> .dir.8dc62baa-afc5-496b-9ac1-692c719856ff.324610.4.1.5 ls | wc -l
> 229646
> 
> the following two reasons could be the cause:
> 
> 1. I see that both primary and secondary zones have resharded the bucket
> 'newcontainer-3' once with num_shards~=127 respectively, which is correct. 
> whereas on the archive zone, bucket has been dynamically resharded to only
> num_shards=29.
> 
> in line https://github.com/ceph/ceph/blob/main/src/rgw/rgw_quota.cc#L976,
> for multisite configurations we set the obj_multiplier to 8 instead of 2 to
> increase the shard count faster to reduce the number of reshard events
> required. But recently we disabled data logging in archive zone setting this
> bool to false in RGWRados::check_bucket_shards(), thus defaulting to the
> non-multisite reshard multiplier of 2, which is probably why the bucket is
> resharded to only 29 shards.
> 
> 

Hi Shilpa, Since the dynamic resharding is not adequately resharding at the archive, should we try a manual reshard on the bucket to test if that helps resolve the large omaps?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7780

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days