Description of problem:
On a busy RGW gateway customer experiences issues where GC will pre-empt client RGW requests because the GC is busy doing cleanup.

Need to look at multiple options to fix:

1) Allow GC to be configurable on certain RGWs so that client requests can go to some RGWs and the GC runs on its own RGW host with no client requests thus mitigating the resource contention

2) Put some type of QOS policy in place for GC so that it never impacts client requests on a RGW

3) Provide ability to throttle GC requests

4) Some other approach not noted above


Version-Release number of selected component (if applicable):
2.4

How reproducible:
Sporadic but on a busy cluster usually regularly

Steps to Reproduce:
1.Create a busy cluster with lots of objects being written and deleted
2.GC will start to perform its tasks 
3.Clients will experience slow downs

Actual results:
Pre-empted client requests

Expected results:
Client requests should remain unimpaired

Additional info: