Description of problem: To avoid "thundering herds" of builds which could potentially impact cluster performance, a tier of machines dedicated to builds should be created. Using nodeSelectors for dedicated build systems in conjunction with Opaque Integer Resources or request limits, a queue/holding pen for builds can be created which limits the number of concurrent builds active in the cluster. It also confines builds to certain nodes, keeping them away from application pods.
This could help avoid problems such as: https://bugzilla.redhat.com/show_bug.cgi?id=1445564
The description describes a build tier. The more general need is to throttle builds in some manner. A build tier is one implementation - other throttling implementations should be considered and the best one selected.