| Summary: | RGW: Unable to delete objects from some buckets | ||
|---|---|---|---|
| Product: | Red Hat Ceph Storage | Reporter: | Vikhyat Umrao <vumrao> |
| Component: | RGW | Assignee: | Orit Wasserman <owasserm> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | ceph-qe-bugs <ceph-qe-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 1.3.0 | CC: | cbodley, ceph-eng-bugs, kbader, mbenjamin, owasserm, sweil, vumrao, wzyuliyang911, yehuda |
| Target Milestone: | rc | ||
| Target Release: | 1.3.3 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-03-14 06:40:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
i have the same problem under ceph 0.94.7 |
Description of problem: 1. Unable to delete object/key from some buckets and later these buckets also as buckets still have objects. 2. Bucket and Object version feature is being used. 3. Bucket and object creation works fine on new buckets. 4. Able to create objects, enable/suspend/re-enable versioning, then delete objects/keys, then delete the bucket. 5. But the 2 buckets creating issues and I can't delete, first, I can't delete the key/object inside the bucket, have tried enabling/disabling versioning but getting the same error message. In boto getting this error: boto.exception.BotoServerError: BotoServerError: 500 Internal Server Error and in RGW logs we have : 14:56:15.949977 7fc9091ed700 0 ERROR: rgw_bucket_sync_user_stats() for user=d57d0879288840e199bb1a49ae012c78, bucket=au-0001(@{i=.au-east.rgw.buckets.index,e=.au-east.rgw.buckets.extra}.au-east.rgw.buckets[au-east.541623.97]) returned -2 As soon as bucket version gets the version from [au-east.541623.97] to [au-east.541623.103] which is more than 100 and MAX_ECANCELED_RETRY is defined as 100. 14:59:15.951379 7fc9091ed700 0 ERROR: rgw_bucket_sync_user_stats() for user=d57d0879288840e199bb1a49ae012c78, bucket=au-0001(@{i=.au-east.rgw.buckets.index,e=.au-east.rgw.buckets.extra}.au-east.rgw.buckets[au-east.541623.103]) returned -2 Then we get "ERROR: exceeded max ECANCELED retries, aborting (EIO)". 243363 2016-01-01 14:58:41.638308 7fc8ce7b4700 1 ====== starting new request req=0x7fc914073ea0 ===== 243364 2016-01-01 14:58:43.711243 7fc8ce7b4700 0 ERROR: exceeded max ECANCELED retries, aborting (EIO) 243365 2016-01-01 14:58:43.711264 7fc8ce7b4700 0 WARNING: set_req_state_err err_no=5 resorting to 500 243366 2016-01-01 14:58:43.711324 7fc8ce7b4700 1 ====== req done req=0x7fc914073ea0 http_status=500 ====== Later we tried deleting all the objects and buckets and these two buckets with the help of "radosgw-admin" command but it is giving error. $ radosgw-admin bucket rm --purge-objects --bucket=au-0001 $ radosgw-admin bucket rm --purge-objects --bucket=au-east-0001 $ radosgw-admin bucket rm --bucket=au-0001 $ radosgw-admin bucket rm --bucket=au-east-0001 Version-Release number of selected component (if applicable): Upstream Hammer ceph-common-0.94.3-0.el7.x86_64 ceph-radosgw-0.94.3-0.el7.x86_64