Hide Forgot
Created attachment 1472783 [details] Bucket List Pre/Post shard Description of problem: Our customer opened a case noting that there was an increase in IOPS when they enabled the dynamic sharding feature in their environment. The issue reflected to be the same as luminous: Resharding hangs with versioning-enabled buckets and other in upstream trackers: http://tracker.ceph.com/issues/24937 https://tracker.ceph.com/issues/24551 Version-Release number of selected component (if applicable): Luminous How reproducible: TBD We did have the customer capture the bucket list pre and post sharding and have evidence that the same buckets are being flagged over and over again. Steps to Reproduce: The Customer allows the sharding process to run The bucket list is checked pre-run The Bucket list is checked post-run Actual results: The buckets in the lists are identical in every respect Expected results: The buckets which were queued to be shared, should not been seen in the list again with identical info as when they were first flagged Additional info:
The upstream PR is here, currently DNM, cleaning up: https://github.com/ceph/ceph/pull/24406
Pushed to ceph-3.2-rhel-patches.
Here are some of the tests I used to verify the cases. All tests are based on inserting code at the top of the inner-most loop in RGWBucketReshard::do_reshard. In each case we're also testing resharding on a bucket with more than 30 objects and with the rgw_reshard_bucket_lock_duration set to 30. Test 1: Insert sleep(1); -- this tests whether renewing the lock works when the resharding is taking somewhat longer than expected. Test 2: Insert sleep(32); -- this tests proper recovery when we're unable to renew the lock before it expires. Test 3: Insert static int i = 0; if (++i > 10) exit(1); -- this tests crashing of the radosgw code leaving things in a non-cleaned up state. I then restart everything and make sure I can read/write the bucket index (e.g., list the bucket, remove an object from the bucket). Furthermore I checked the radosgw log file to see if it includes "apparently successfully cleared resharding flags for bucket...".
*** Bug 1644212 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0020