Created attachment 1472783 [details]
Bucket List Pre/Post shard
Description of problem:
Our customer opened a case noting that there was an increase in IOPS when they enabled the dynamic sharding feature in their environment.
The issue reflected to be the same as luminous: Resharding hangs with versioning-enabled buckets and other in upstream trackers:
Version-Release number of selected component (if applicable):
We did have the customer capture the bucket list pre and post sharding and have evidence that the same buckets are being flagged over and over again.
Steps to Reproduce:
The Customer allows the sharding process to run
The bucket list is checked pre-run
The Bucket list is checked post-run
The buckets in the lists are identical in every respect
The buckets which were queued to be shared, should not been seen in the list again with identical info as when they were first flagged
The upstream PR is here, currently DNM, cleaning up:
Pushed to ceph-3.2-rhel-patches.
Here are some of the tests I used to verify the cases. All tests are based on inserting code at the top of the inner-most loop in RGWBucketReshard::do_reshard. In each case we're also testing resharding on a bucket with more than 30 objects and with the rgw_reshard_bucket_lock_duration set to 30.
Test 1: Insert sleep(1); -- this tests whether renewing the lock works when the resharding is taking somewhat longer than expected.
Test 2: Insert sleep(32); -- this tests proper recovery when we're unable to renew the lock before it expires.
Test 3: Insert static int i = 0; if (++i > 10) exit(1); -- this tests crashing of the radosgw code leaving things in a non-cleaned up state. I then restart everything and make sure I can read/write the bucket index (e.g., list the bucket, remove an object from the bucket). Furthermore I checked the radosgw log file to see if it includes "apparently successfully cleared resharding flags for bucket...".
*** Bug 1644212 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.