Bug 1674436 - GC erratic performance, very slow deletion performance
Summary: GC erratic performance, very slow deletion performance
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 3.1
Hardware: x86_64
OS: Linux
Target Milestone: z1
: 3.2
Assignee: Mark Kogan
QA Contact: Tejas
: 1672433 (view as bug list)
Depends On:
Blocks: 1629656 1680050
TreeView+ depends on / blocked
Reported: 2019-02-11 11:01 UTC by Ilan Green
Modified: 2019-05-30 19:22 UTC (History)
16 users (show)

Fixed In Version: RHEL: ceph-12.2.8-85.el7cp Ubuntu: ceph_12.2.8-71redhat1
Doc Type: Bug Fix
Doc Text:
.Garbage collection no longer consumes bandwidth without making forward progress Previously, some underlying bugs prevented garbage collection (GC) from making forward progress. Specifically, the marker was not always being advanced, GC was unable to process entries with zero-length chains, and the truncated flag was not always being set correctly. This caused GC to consume bandwidth without making any forward progress, thereby not freeing up disk space, slowing down other cluster work, and allowing OMAP entries related to GC to continue to increase. With this update, the underlying bugs have been fixed, and GC is able to make progress as expected freeing up disk space and OMAP entries.
Clone Of:
: 1680050 (view as bug list)
Last Closed: 2019-03-07 15:51:42 UTC

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0475 None None None 2019-03-07 15:51:50 UTC
Ceph Project Bug Tracker 38408 None None None 2019-02-21 17:17:39 UTC
Red Hat Knowledge Base (Solution) 3936141 None None None 2019-02-24 17:29:45 UTC

Description Ilan Green 2019-02-11 11:01:56 UTC
Description of problem:
GC draining is working very slow.
Having about 235 objects deleted in 24 hours in bursts (vs. a steady rate)

Version-Release number of selected component (if applicable):

How reproducible:
This currently happens only on the customer's system - gc backlog is increasing over 50 million by now

Steps to Reproduce:

Actual results:
GC backlog is increasing

Expected results:
GC to drain objects at a much faster rate

Additional info:
ceph.conf file to be provided soon
Output of radosgw-admin gc process --debug-rgw=20 --debug-ms=1 |& tee ./gc_proc.log to provided, too, soon

Comment 38 Vikhyat Umrao 2019-02-22 20:16:52 UTC
*** Bug 1672433 has been marked as a duplicate of this bug. ***

Comment 54 errata-xmlrpc 2019-03-07 15:51:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.