Bug 1318409 - RGW deletion is sequential and slow on large buckets of objects
Summary: RGW deletion is sequential and slow on large buckets of objects
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 1.3.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 2.1
Assignee: Yehuda Sadeh
QA Contact: Vasishta
Bara Ancincova
URL:
Whiteboard:
Depends On:
Blocks: 1383917
TreeView+ depends on / blocked
 
Reported: 2016-03-16 19:20 UTC by Benjamin Schmaus
Modified: 2021-08-30 11:54 UTC (History)
13 users (show)

Fixed In Version: RHEL: ceph-10.2.3-5.el7cp Ubuntu: ceph_10.2.3-6redhat1xenial
Doc Type: Bug Fix
Doc Text:
.Ceph Object Gateway now deletes large buckets in parallel Previously, the Ceph Object Gateway was unable to delete multiple large buckets at the same time. As a consequence, the process of deleting large buckets containing millions of objects was slow. A patch has been applied, and the Ceph Object Gateway now deletes large buckets in parallel, which makes the whole process significantly faster.
Clone Of:
Environment:
Last Closed: 2016-11-22 19:25:06 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 15557 0 None None None 2016-09-26 23:09:25 UTC
Red Hat Bugzilla 1418809 0 medium CLOSED Backport RHCS 2.2 : Have a flavor of bucket deletion in radosgw-admin to bypass garbage collection 2021-08-30 12:22:02 UTC
Red Hat Issue Tracker RHCEPH-1345 0 None None None 2021-08-30 11:54:36 UTC
Red Hat Knowledge Base (Solution) 2901171 0 None None None 2017-02-02 21:05:46 UTC
Red Hat Product Errata RHSA-2016:2815 0 normal SHIPPED_LIVE Moderate: Red Hat Ceph Storage security, bug fix, and enhancement update 2017-03-22 02:06:33 UTC

Internal Links: 1418809

Description Benjamin Schmaus 2016-03-16 19:20:37 UTC 
Description of problem:  When using RGW to delete large buckets with 50-100 or more million objects the process is slow and appears to be sequential in nature.


Version-Release number of selected component (if applicable):1.3.2


How reproducible:100%


Steps to Reproduce:
1.Create a Ceph cluster on 1.3.2
2.Load up a bucket with many millions of objects
3.Delete objects in the bucket

Actual results: Slow process takes time to delete.


Expected results: RGW should be able to delete multiple objects in parallel and speed up the process when doing large deletes


Additional info:
Comment 21 Ken Dreyer (Red Hat) 2016-09-26 23:09:26 UTC 
Jewel backport ongoing at https://github.com/ceph/ceph/pull/10661 for v10.2.4 (currently fails to build there, looks like the RGW team will need to resolve the failure)
Comment 22 Matt Benjamin (redhat) 2016-09-29 21:22:56 UTC 
(In reply to Ken Dreyer (Red Hat) from comment #21)
> Jewel backport ongoing at https://github.com/ceph/ceph/pull/10661 for
> v10.2.4 (currently fails to build there, looks like the RGW team will need
> to resolve the failure)

PR #10661 updated with a proposed compile fix and motivation.
Comment 24 Matt Benjamin (redhat) 2016-10-06 15:22:18 UTC 
To reproduce, create an rgw bucket and put any substantial number of objects, then measure the time required to sequentially delete each object and then the bucket.

After this change, that time can be compared with the result of

"time radosgw-admin bucket rm --bucket=<bucket name> --bypass-gc --purge-objects"

having first recreated the bucket and objects.
Comment 27 Vasishta 2016-11-02 12:01:41 UTC 
Hi,

I simultaneously tried purging a bucket and deleting all objects in another bucket . Each of those bucket had 5 million objects and buckets were of same size. When one bucket got purged, another bucket had more than 1.8 million objects.
Purging took 64 hours and deletion took 110+ hours.


As purging is faster than sequential deletion as per above observation I'm closing this bug.

Please let me know if there any concerns or issues.


Regards,
Vasishta
Comment 30 errata-xmlrpc 2016-11-22 19:25:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2815.html
Note You need to log in before you can comment on or make changes to this bug.