Hide Forgot
Description of problem: This has been happening for a while but I keep forgetting to write it up. Workload is starting 250 pause pods, which boils down to 250 ose-pod containers and 250 gcr.io/google_containers/pause-amd64:3.0 containers. The workload is strictly starting containers. 40 pods are started at a time with a break between each batch. A pidstat profile of the run shows rhel-push-plugin as the #3 CPU user at ~5%. This seems high for a plugin which should be getting out of the way as quickly as it can, especially when no push is involved. See an example here: http://perf-infra.ec2.breakage.org/pbench/results/ip-172-31-12-67/pbench-user-benchmark_nodeVert314_2016-11-08_19:08:38/1/reference-result/tools-default/ip-172-31-48-59.us-west-2.compute.internal/pidstat/cpu_usage.html Click on each of the top 3 lines of the table to see the top users (openshift node, docker daemon and rhel-push-plugin). This pattern is consistent across runs Version-Release number of selected component (if applicable): 1.12.3-3 from Extras. How reproducible: Always
So is this just a matter of how much resources the plugin consumes or is it blocked at some point preventing pushes? (the title may refers to the former but the description the latter)
runcom, sorry for the title switch :-). This refers to the cpu resources consumed by the plugin while starting 250 OCP pods. There is no blockage and there are no pushes in this workload.
I'll try to reproduce this somehow, I suspect this is how Docker manages plugin s and has nothing to do with our plugin (since the plugin it's just an if not pushing then exists, it's really simple and basic).
Contact me offline if you want help reproducing. The cluster-loader script described at https://github.com/openshift/svt/tree/master/openshift_scalability can help when used with this input file: https://github.com/openshift/svt/blob/master/openshift_scalability/config/nodeVertical.yaml
The config file in comment 4 presumes 2 application nodes loaded to 250 pods each.
found nothing so far, started a conversation with upstream https://github.com/docker/docker/issues/28244
Antonio anything ever come of this?
We have no plans to ship another version of Docker at this time. RHEL7 is in final support stages where only security fixes will get released. Customers should move to use Podman which is available starting in RHEL 7.6.