Description of problem:

1.  Unable to delete object/key from some buckets and later these buckets also as buckets still have objects.
2.  Bucket and Object version feature is  being used. 
3.  Bucket and object creation works fine on new buckets.
4.  Able to create objects, enable/suspend/re-enable versioning, then delete objects/keys, then delete the bucket.
5. But the 2 buckets creating issues and I can't delete, first, I can't delete the key/object inside the bucket, have tried enabling/disabling versioning but getting the same error message.

In boto getting this error: 

boto.exception.BotoServerError: BotoServerError: 500 Internal Server Error

and in RGW logs we have :

14:56:15.949977 7fc9091ed700  0 ERROR: rgw_bucket_sync_user_stats() for user=d57d0879288840e199bb1a49ae012c78, bucket=au-0001(@{i=.au-east.rgw.buckets.index,e=.au-east.rgw.buckets.extra}.au-east.rgw.buckets[au-east.541623.97]) returned -2 

 As soon as bucket version gets the version from [au-east.541623.97] to [au-east.541623.103] which is more than 100 and MAX_ECANCELED_RETRY is defined as 100.

14:59:15.951379 7fc9091ed700  0 ERROR: rgw_bucket_sync_user_stats() for user=d57d0879288840e199bb1a49ae012c78, bucket=au-0001(@{i=.au-east.rgw.buckets.index,e=.au-east.rgw.buckets.extra}.au-east.rgw.buckets[au-east.541623.103]) returned -2

Then we get "ERROR: exceeded max ECANCELED retries, aborting (EIO)".

243363 2016-01-01 14:58:41.638308 7fc8ce7b4700  1 ====== starting new request req=0x7fc914073ea0 =====
243364 2016-01-01 14:58:43.711243 7fc8ce7b4700  0 ERROR: exceeded max ECANCELED retries, aborting (EIO)
243365 2016-01-01 14:58:43.711264 7fc8ce7b4700  0 WARNING: set_req_state_err err_no=5 resorting to 500
243366 2016-01-01 14:58:43.711324 7fc8ce7b4700  1 ====== req done req=0x7fc914073ea0 http_status=500 ======

Later we tried deleting all the objects and buckets and these two buckets with the help of "radosgw-admin" command  but it is giving error.

$ radosgw-admin bucket rm --purge-objects --bucket=au-0001
$ radosgw-admin bucket rm --purge-objects --bucket=au-east-0001
$ radosgw-admin bucket rm --bucket=au-0001
$ radosgw-admin bucket rm --bucket=au-east-0001


Version-Release number of selected component (if applicable):
Upstream Hammer 
ceph-common-0.94.3-0.el7.x86_64
ceph-radosgw-0.94.3-0.el7.x86_64