Bug 1674436

Summary: GC erratic performance, very slow deletion performance
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Ilan Green <igreen>
Component: RGWAssignee: Mark Kogan <mkogan>
Status: CLOSED ERRATA QA Contact: Tejas <tchandra>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.1CC: assingh, cbodley, ceph-eng-bugs, ceph-qe-bugs, edonnell, ivancich, kbader, kdreyer, mamccoma, mbenjamin, mkogan, sweil, tchandra, tpetr, tserlin, vumrao
Target Milestone: z1   
Target Release: 3.2   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-12.2.8-85.el7cp Ubuntu: ceph_12.2.8-71redhat1 Doc Type: Bug Fix
Doc Text:
.Garbage collection no longer consumes bandwidth without making forward progress Previously, some underlying bugs prevented garbage collection (GC) from making forward progress. Specifically, the marker was not always being advanced, GC was unable to process entries with zero-length chains, and the truncated flag was not always being set correctly. This caused GC to consume bandwidth without making any forward progress, thereby not freeing up disk space, slowing down other cluster work, and allowing OMAP entries related to GC to continue to increase. With this update, the underlying bugs have been fixed, and GC is able to make progress as expected freeing up disk space and OMAP entries.
 Story Points: ---
Clone Of:
: 1680050 (view as bug list) Environment:
Last Closed: 2019-03-07 15:51:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1629656, 1680050    

Description Ilan Green 2019-02-11 11:01:56 UTC 
Description of problem:
GC draining is working very slow.
Having about 235 objects deleted in 24 hours in bursts (vs. a steady rate)

Version-Release number of selected component (if applicable):
3.0z5

How reproducible:
This currently happens only on the customer's system - gc backlog is increasing over 50 million by now

Steps to Reproduce:
1.
2.
3.

Actual results:
GC backlog is increasing

Expected results:
GC to drain objects at a much faster rate


Additional info:
ceph.conf file to be provided soon
Output of radosgw-admin gc process --debug-rgw=20 --debug-ms=1 |& tee ./gc_proc.log to provided, too, soon
Comment 38 Vikhyat Umrao 2019-02-22 20:16:52 UTC 
*** Bug 1672433 has been marked as a duplicate of this bug. ***
Comment 54 errata-xmlrpc 2019-03-07 15:51:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0475