Created attachment 1001134 [details]
dynflow_executor cpu usage from pidstat during 8, 6, 2 repo syncs

Description of problem:
Dynflow is a single process within Satellite 6/6.1 and can bottleneck other concurrent tasks.


Version-Release number of selected component (if applicable):
Satellite 6.1 Beta

How reproducible:
100%

Steps to Reproduce:
1. Sync 2 equal sized repos concurrently and time
2. Sync 4 equal sized repos concurrently and time
3. Observe that hardware is not a bottleneck and that dynflow is consuming more and more cpu time at the end of the concurrent tasks.
4. Observe that dynflow_executor is limited to 1 process and a single cpu resource during concurrent tasks
5. With no hardware bottlenecks, observe increase in sync timing

Actual results:
Dynflow can be a source of additional latency on concurrent tasks.

Expected results:
dynflow_executor to scale as more concurrent tasks are conducted such that it does not bottleneck any other tasks.  It is also important that dynflow is not allowed to scale to the point where it uses all the system resources as well. 

Additional info:
Attached graphs shows cpu usage of dynflow_executor during syncs.
top graph = dynflow_excutor cpu usage during 8 repo syncs
middle graph = dynflow_excutor cpu usage during 6 repo syncs
bottom graph = dynflow_excutor cpu usage during 2 repo syncs



The average timing values for the syncs are:
2 repos - 142.15
4 repos - 204.83
6 repos - 259.62
8 repos - 317.05

The growth in overall timing was measured from hammer cli command to sync each repository.  The above values come from RHEL 6.6