Bug 1296130 - RGW: Unable to delete objects from some buckets
RGW: Unable to delete objects from some buckets
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
x86_64 Linux
high Severity high
: rc
: 1.3.3
Assigned To: Orit Wasserman
Depends On:
  Show dependency treegraph
Reported: 2016-01-06 07:10 EST by Vikhyat Umrao
Modified: 2017-07-30 11:56 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-03-14 02:40:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2110551 None None None 2016-03-14 02:40 EDT

  None (edit)
Description Vikhyat Umrao 2016-01-06 07:10:12 EST
Description of problem:

1.  Unable to delete object/key from some buckets and later these buckets also as buckets still have objects.
2.  Bucket and Object version feature is  being used. 
3.  Bucket and object creation works fine on new buckets.
4.  Able to create objects, enable/suspend/re-enable versioning, then delete objects/keys, then delete the bucket.
5. But the 2 buckets creating issues and I can't delete, first, I can't delete the key/object inside the bucket, have tried enabling/disabling versioning but getting the same error message.

In boto getting this error: 

boto.exception.BotoServerError: BotoServerError: 500 Internal Server Error

and in RGW logs we have :

14:56:15.949977 7fc9091ed700  0 ERROR: rgw_bucket_sync_user_stats() for user=d57d0879288840e199bb1a49ae012c78, bucket=au-0001(@{i=.au-east.rgw.buckets.index,e=.au-east.rgw.buckets.extra}.au-east.rgw.buckets[au-east.541623.97]) returned -2 

 As soon as bucket version gets the version from [au-east.541623.97] to [au-east.541623.103] which is more than 100 and MAX_ECANCELED_RETRY is defined as 100.

14:59:15.951379 7fc9091ed700  0 ERROR: rgw_bucket_sync_user_stats() for user=d57d0879288840e199bb1a49ae012c78, bucket=au-0001(@{i=.au-east.rgw.buckets.index,e=.au-east.rgw.buckets.extra}.au-east.rgw.buckets[au-east.541623.103]) returned -2

Then we get "ERROR: exceeded max ECANCELED retries, aborting (EIO)".

243363 2016-01-01 14:58:41.638308 7fc8ce7b4700  1 ====== starting new request req=0x7fc914073ea0 =====
243364 2016-01-01 14:58:43.711243 7fc8ce7b4700  0 ERROR: exceeded max ECANCELED retries, aborting (EIO)
243365 2016-01-01 14:58:43.711264 7fc8ce7b4700  0 WARNING: set_req_state_err err_no=5 resorting to 500
243366 2016-01-01 14:58:43.711324 7fc8ce7b4700  1 ====== req done req=0x7fc914073ea0 http_status=500 ======

Later we tried deleting all the objects and buckets and these two buckets with the help of "radosgw-admin" command  but it is giving error.

$ radosgw-admin bucket rm --purge-objects --bucket=au-0001
$ radosgw-admin bucket rm --purge-objects --bucket=au-east-0001
$ radosgw-admin bucket rm --bucket=au-0001
$ radosgw-admin bucket rm --bucket=au-east-0001

Version-Release number of selected component (if applicable):
Upstream Hammer 
Comment 21 yyang lee 2016-07-04 10:49:53 EDT
i have the same problem under ceph 0.94.7

Note You need to log in before you can comment on or make changes to this bug.