Description of problem: `radosgw-admin gc list --include-all` can take over an hour to return on RHCS 4.1 On RHCS 4.0 command returns in 90sec, with same workload executing. Version-Release number of selected component (if applicable): RHCS 4.1 How reproducible: yes Steps to Reproduce: 1. Two Identical Clusters: each 8x OSD/RGW nodes (192 OSDs) Site 1 = RHCS 4.0 Site 2 = RHCS 4.1 (14.2.8-91.el7cp) Clusters pre-filled to 25% RAW USED 62MB mean objsz: h(1|1|50,64|64|15,8192|8192|15,65536|65536|15,1048576|1048576|5)KB Manually set rgw_gc_obj_min_wait to 30min (2 hour lag by default) Workload delWrite (50% delete / 50% write), 48hour runtime 2. Same polling script on both clusters, executes every three minutes: `radosgw-admin gc list --include-all` 3. On RHCS 4.1 the command periodically requires 50min (or more) to complete. On RHCS 4.0 the command requires 90sec to return. Actual results: During the 48hr workload, Site1 (v4.0) saw 1526 samples compared to 109 in site2 (v4.1) Site2 (v4.1) sees occasional huge spikes of #pendingGCs and slow response One hour into workload, 17min delay Two hours in (timestamp 17:27:44), 50min delay Again at 18:27:41 - happens every hour, all on site2 (v4.1) v4.0 shows much more steady progression Expected results: Consistent time to complete cmd producing same number of samples in 48hr runtime Steady increase, and decrease in pending GCs, rather than huge spikes which coincide with very long command completion Additional info: Raw results gsheet https://docs.google.com/spreadsheets/d/1spUzXxiQu3RCioo7FM9vyrOt-g58kzJ9By6s26gys84/edit#gid=126852855 Pending GCs comparision (see attachment: w7pendingGCs.txt
Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
*** Bug 1898647 has been marked as a duplicate of this bug. ***
Created attachment 1739114 [details] text log of pending GC response times
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0081