Description of problem:
Using ssh in a loop (serialized) is 3 times faster than same action with Remote Execution on clients equally distributed over 10 capsules (with dynflow database set to "in memory" on satellite and capsules).

Satellite is a VM with 20 cores and 47 GB of RAM. 10 capsules (again VMs) have 8 CPUs and 16 GB RAM. All (including hosts) on 10G network.


Version-Release number of selected component (if applicable):
satellite-6.3.0-21.0.beta.el7sat.noarch


How reproducible:
always


Steps to Reproduce:
1. Run ReX job on 30k hosts with command
   `systemctl stop rhsmcertd; systemctl disable rhsmcertd`
2. Try with a subset with simple loop and ssh


Actual results:
Job is now running for 23 hours, 24 minutes and reports 24800 hosts as done so far, i.e. more than 3 seconds per host.

I have tried to run simple loop with ssh (it is this complicated only because of IP ranges we are using:

# time \
    for ip1 in $( seq 0 30 ); do
        for ip2 in 0 1 2 3; do
            ip=$( expr $ip1 \* 8 + $ip2 )
            ssh -o "StrictHostKeyChecking no" \
                -i /root/id_rsa_perf
                root.$ip.100
                "systemctl stop rhsmcertd; systemctl disable rhsmcertd"
        done
    done

This ran the command on 124 hosts and finished 2m4.953s, i.e. slightly above 1 second per host.


Expected results:
I know satellite and capsules are doing much more than just sshing to the clients (e.g. storing results for later auditing), but as the load should be somehow distributed among 10 capsules and as we have already tuned database on satellite and capsules to be inmemory only, I would expect speed of this action to be close to what I'm able to achieve with ssh or faster.