Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: importers workload doesn't spread equally on all workers during migration during migration with 2 plans on single esx on VMware 6.7 with 39 VMs plan1 -19 VMs plan2 -20 VMs root@f02-h07-000-r640:~$ oc get pods -nopenshift-rhmtv |grep import |grep h17 |wc -l 2 root@f02-h07-000-r640:~$ oc get pods -nopenshift-rhmtv |grep import |grep h15 |wc -l 6 root@f02-h07-000-r640:~$ oc get pods -nopenshift-rhmtv |grep h18 |wc -l 35 root@f02-h07-000-r640:~$ oc get pods -nopenshift-rhmtv |grep h18 |grep forklift |wc -l 4 Version-Release number of selected component (if applicable): MTV : 2.0.0.20 CNV : 2.6.1
Please help us investigate why pods are not evenly spread across worker nodes. VMIO simply creates DataVolume CR, and importer pod are created by CDI. This is why we set the product to CNV.
What kind of storage is being used? CDI allows kubernetes to make scheduling decisions for the workloads. Why do you expect them to be perfectly balanced? Is there a problem with the import or is it finishing?
We are using Ceph (OCS block) and NFS storage, I believe that it was OCS in this case. We saw in some of the runs that the worker with Many pods had Out of Memory issue - Bug 1952121 - [ MTV ] - inventory container crash due to OOM - leading to controller pod restarts https://bugzilla.redhat.com/show_bug.cgi?id=1952121 Maybe it is related. Fabien suggested maybe the migration Pods can be set to request minimum memory for each one - and that will cause better balance, We do not expect perfect balance but something better than 90% work goes to one worker out of 3.
Cycle#1 - 20VMs : Tested again BZ1952450 on cloud38 : 1. 20 VMs 2. vmware 6.7 3. MTV: 2.0.0.12 4. CNV: 2.6.2 5. storage : ceph 6. Management Network - 10.1.36.0/22 19 importer pods running on worker > f02-h17-000-r640.rdu2.scalelab.redhat.com 1 importer pod running on worker > f02-h15-000-r640.rdu2.scalelab.redhat.com one worker isn't in use > f02-h18-000-r640.rdu2.scalelab.redhat.com Cycle#2 - 50VMs : Tested again BZ1952450 on cloud38 : 1. 50 VMs 2. vmware 6.7 3. MTV: 2.0.0.12 4. CNV: 2.6.2 5. storage : ceph 6. Management Network - 10.1.36.0/22 root@f02-h07-000-r640:~$ oc get pods -nopenshift-mtv -owide |grep importer |grep f02-h15-000-r640.rdu2.scalelab.redhat.com |wc -l 6 root@f02-h07-000-r640:~$ oc get pods -nopenshift-mtv -owide |grep importer |grep f02-h17-000-r640.rdu2.scalelab.redhat.com |wc -l 34 root@f02-h07-000-r640:~$ oc get pods -nopenshift-mtv -owide |grep importer |grep f02-h18-000-r640.rdu2.scalelab.redhat.com |wc -l 6 4 disk copy process completed 50-4=46
It seems that for some reason the scheduler has a preference for f02-h17-000-r640.rdu2.scalelab.redhat.com. When no migration is in progress, how many pods are running on each worker? You can use the following command: $ oc get node -l node-role.kubernetes.io/worker --no-headers -o name | sed s@^node/@@ | while read NODE ; do PODS=`oc get pod -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName | grep "$NODE" | wc -l` ; echo -e "$NODE\t$PODS" ; done Then, to know the requests and limits of the pods, can you run the following command: $ oc get node -l node-role.kubernetes.io/worker --no-headers -o name | sed s@^node/@@ | while read NODE ; do echo "======= $NODE =======" ; oc get pod -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,CPU_REQUEST:.spec.containers[].resources.requests.cpu,CPU_LIMIT:.spec.containers[].resources.limits.cpu,MEMORY_REQUEST:.spec.containers[].resources.requests.memory,MEMORY_LIMIT:.spec.containers[].resources.limits.memory | grep $NODE ; echo ; done This could help to get a hint at what can influence the scheduler and also see how the pods are spread at idle time.
You can specify resource request/limits on both CPU and memory in the CDI CR config stanza. For example: podResourceRequirements: limits: cpu: "4" memory: "1Gi" requests: cpu: "1" memory: "250Mi" That will instruct CDI to set requests and limits on the transfer pods.
pods were distributed equally between 2 workers while tested with bellow versions: (3rd worker was in maintenance during IP issue) CNV: 2.6.0 hco-v2.6.0-582 iib-46573 MTV: 2.0.0-beta.1-25 iib-47675
Thanks David. Fabien, what else is needed in this case?
Adam,Fabien As David mentioned that it was balancing good in 2.6.0 - what could have changed to cause it? mlehrer our perf team lead, suggested Maybe the migration pods should be assigned prefered with "Anti-affinity" rules to allow better balance which will eventually ensure more stable and performant migration? https://docs.openshift.com/container-platform/4.7/nodes/scheduling/nodes-scheduler-pod-affinity.html
Daniel, it's an interesting idea but is not always going to be the right answer for all environments and someone would probably complain that such a change causes undesirable effects for them.
(In reply to Fabien Dupont from comment #5) oot@f02-h07-000-r640:~$ oc get pods -owide -nopenshift-mtv NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES forklift-controller-564d55bb5-n5svg 2/2 Running 0 58m 10.129.2.10 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> forklift-operator-5f59c57fd8-fg77t 1/1 Running 0 58m 10.129.2.8 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> forklift-ui-85cdc8bbf6-z65fx 1/1 Running 0 58m 10.129.2.12 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> forklift-validation-59b7cd6654-zvcn4 1/1 Running 0 58m 10.129.2.11 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> root@f02-h07-000-r640:~$ root@f02-h07-000-r640:~$ oc get node -l node-role.kubernetes.io/worker --no-headers -o name | sed s@^node/@@ | while read NODE ; do PODS=`oc get pod -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName | grep "$NODE" | wc -l` ; echo -e "$NODE\t$PODS" ; done f02-h15-000-r640.rdu2.scalelab.redhat.com 43 f02-h17-000-r640.rdu2.scalelab.redhat.com 63 f02-h18-000-r640.rdu2.scalelab.redhat.com 64 root@f02-h07-000-r640:~$ root@f02-h07-000-r640:~$ oc get node -l node-role.kubernetes.io/worker --no-headers -o name | sed s@^node/@@ | while read NODE ; do echo "======= $NODE =======" ; oc get pod -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,CPU_REQUEST:.spec.containers[].resources.requests.cpu,CPU_LIMIT:.spec.containers[].resources.limits.cpu,MEMORY_REQUEST:.spec.containers[].resources.requests.memory,MEMORY_LIMIT:.spec.containers[].resources.limits.memory | grep $NODE ; echo ; done ======= f02-h15-000-r640.rdu2.scalelab.redhat.com ======= tuned-h4pqm f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> bridge-marker-7wzpt f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> kube-cni-linux-bridge-plugin-ssvs5 f02-h15-000-r640.rdu2.scalelab.redhat.com 60m <none> 30Mi <none> kubevirt-node-labeller-hmqmk f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> nmstate-handler-hlnfv f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-handler-qscxq f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> dns-default-h948g f02-h15-000-r640.rdu2.scalelab.redhat.com 50m <none> 70Mi <none> image-pruner-1621382400-k2rnr f02-h15-000-r640.rdu2.scalelab.redhat.com 100m <none> 256Mi <none> node-ca-cql7g f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 10Mi <none> ingress-canary-k6cwf f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 20Mi <none> coredns-f02-h15-000-r640.rdu2.scalelab.redhat.com f02-h15-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> keepalived-f02-h15-000-r640.rdu2.scalelab.redhat.com f02-h15-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> mdns-publisher-f02-h15-000-r640.rdu2.scalelab.redhat.com f02-h15-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> local-block-local-diskmaker-bvlq6 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> local-block-local-provisioner-mddcc f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> machine-config-daemon-sqwsg f02-h15-000-r640.rdu2.scalelab.redhat.com 20m <none> 50Mi <none> 521834147040c83ff28e758e82818ef73953172932f7400918cb0e6dcbhw8b8 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> c8c51e86914ab4fc3ed6ad87a3e8a5189caa9cd7ec74ab920fed7b92f1v8gzs f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> community-operators-t7s7f f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-operators-6k7dq f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-76027-jgl44 f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-76039-wnndd f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> node-exporter-5rp4b f02-h15-000-r640.rdu2.scalelab.redhat.com 8m <none> 180Mi <none> forklift-controller-564d55bb5-n5svg f02-h15-000-r640.rdu2.scalelab.redhat.com 100m 500m 350Mi 800Mi forklift-operator-5f59c57fd8-fg77t f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> forklift-ui-85cdc8bbf6-z65fx f02-h15-000-r640.rdu2.scalelab.redhat.com 100m 100m 150Mi 800Mi forklift-validation-59b7cd6654-zvcn4 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> multus-x9wbd f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 150Mi <none> network-metrics-daemon-5n45w f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 100Mi <none> network-check-target-x4fq9 f02-h15-000-r640.rdu2.scalelab.redhat.com 10m <none> 15Mi <none> nfd-worker-cz7d7 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> ovs-j9w2p f02-h15-000-r640.rdu2.scalelab.redhat.com 15m <none> 400Mi <none> sdn-qktvh f02-h15-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> sriov-cni-896wt f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> sriov-device-plugin-5bg4z f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> sriov-network-config-daemon-l56rk f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-cephfsplugin-j7dr2 f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-rbdplugin-trlcn f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> noobaa-core-0 f02-h15-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> noobaa-db-pg-0 f02-h15-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-crashcollector-a40c49f114069fdc507184043a233a14-tqlfs f02-h15-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> rook-ceph-mon-b-55f94ccd5f-bbh9v f02-h15-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-osd-1-85847fffc6-6hnhx f02-h15-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> ======= f02-h17-000-r640.rdu2.scalelab.redhat.com ======= tuned-d9dbh f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> bridge-marker-zgg8q f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> cdi-deployment-bb88f7896-78v5b f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> cdi-operator-6c98b44f9b-96vkq f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> cdi-uploadproxy-7899c485c8-znv79 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> cluster-network-addons-operator-96fd874f8-ccwlb f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> hco-operator-cd546d87-6hgr7 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> hco-webhook-c97895d68-zdb95 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> kube-cni-linux-bridge-plugin-2rnvf f02-h17-000-r640.rdu2.scalelab.redhat.com 60m <none> 30Mi <none> kubevirt-node-labeller-xgxfc f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> nmstate-handler-94pzf f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> ssp-operator-85b6784cf4-7x2jc f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-api-69b494df44-bfl4t f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-controller-8df4fd95d-t86bm f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-handler-zsbjq f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-operator-6895674749-wpzgx f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-template-validator-864cf6cdd4-jtsdn f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> vm-import-controller-8985949f6-lg74f f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> dns-default-s4qxc f02-h17-000-r640.rdu2.scalelab.redhat.com 50m <none> 70Mi <none> node-ca-khc4r f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 10Mi <none> ingress-canary-rzd94 f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 20Mi <none> router-default-6555564f68-8fzqb f02-h17-000-r640.rdu2.scalelab.redhat.com 100m <none> 256Mi <none> coredns-f02-h17-000-r640.rdu2.scalelab.redhat.com f02-h17-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> keepalived-f02-h17-000-r640.rdu2.scalelab.redhat.com f02-h17-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> mdns-publisher-f02-h17-000-r640.rdu2.scalelab.redhat.com f02-h17-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> local-block-local-diskmaker-9c8rg f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> local-block-local-provisioner-crkvp f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> machine-config-daemon-2k7kt f02-h17-000-r640.rdu2.scalelab.redhat.com 20m <none> 50Mi <none> hco-catalogsource-r4xh8 f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> iib-rc8c9 f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> ocs-catalogsource-rs8vz f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-marketplace-jlxf8 f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-64369-kkwzg f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-68063-nld4b f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-70486-v7k4g f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-72981-kxj2g f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-73572-6flnr f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> alertmanager-main-0 f02-h17-000-r640.rdu2.scalelab.redhat.com 4m <none> 200Mi <none> grafana-6f4d96d7fd-988jh f02-h17-000-r640.rdu2.scalelab.redhat.com 4m <none> 100Mi <none> node-exporter-v7sb9 f02-h17-000-r640.rdu2.scalelab.redhat.com 8m <none> 180Mi <none> openshift-state-metrics-587d97bb47-sxsfr f02-h17-000-r640.rdu2.scalelab.redhat.com 1m <none> 20Mi <none> prometheus-k8s-0 f02-h17-000-r640.rdu2.scalelab.redhat.com 70m <none> 1Gi <none> telemeter-client-6c4f7f9558-t4t5d f02-h17-000-r640.rdu2.scalelab.redhat.com 1m <none> 40Mi <none> thanos-querier-584cf5f85b-pljdd f02-h17-000-r640.rdu2.scalelab.redhat.com 5m <none> 12Mi <none> multus-p4whl f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 150Mi <none> network-metrics-daemon-92sdc f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 100Mi <none> network-check-target-wfq27 f02-h17-000-r640.rdu2.scalelab.redhat.com 10m <none> 15Mi <none> nfd-worker-5w8vx f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> ovs-6dpdq f02-h17-000-r640.rdu2.scalelab.redhat.com 15m <none> 400Mi <none> sdn-djmm9 f02-h17-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> sriov-cni-x7lb2 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> sriov-device-plugin-zzzz2 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> sriov-network-config-daemon-mdxml f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-cephfsplugin-m2h6v f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-cephfsplugin-provisioner-7d86f6dd86-rwh4q f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-rbdplugin-lm85j f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-rbdplugin-provisioner-7f95c9f9b7-sccjc f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> ocs-operator-764c47d7f5-rmht5 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> rook-ceph-crashcollector-64126ca74937b6fc76eae814e9644449-5g249 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-798db7f8gcrm5 f02-h17-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-mgr-a-54c85fd55d-knfhc f02-h17-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-mon-a-fccbbd884-pghtv f02-h17-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-osd-2-6c8cc7f94c-g9v6l f02-h17-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> ======= f02-h18-000-r640.rdu2.scalelab.redhat.com ======= v2v-vmware-7746f6966b-vrpnh f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> chaos-operator-ce-686cd94fc6-qg267 f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> tuned-ptbns f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> bridge-marker-nv6ws f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> cdi-apiserver-76f5c697c9-6wfgf f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> hostpath-provisioner-operator-bfd9677f-6xmpd f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> kube-cni-linux-bridge-plugin-ppdrj f02-h18-000-r640.rdu2.scalelab.redhat.com 60m <none> 30Mi <none> kubevirt-node-labeller-c88rh f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> nmstate-handler-c5qfz f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-api-69b494df44-nf44x f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-controller-8df4fd95d-xhxn5 f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-handler-jt2hz f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-operator-6895674749-j2xdp f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> virt-template-validator-864cf6cdd4-jvs7f f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> vm-import-operator-b5f699d5f-spp4v f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> dns-default-kn588 f02-h18-000-r640.rdu2.scalelab.redhat.com 50m <none> 70Mi <none> node-ca-fcx9l f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 10Mi <none> ingress-canary-mnjvb f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 20Mi <none> router-default-6555564f68-4phpf f02-h18-000-r640.rdu2.scalelab.redhat.com 100m <none> 256Mi <none> coredns-f02-h18-000-r640.rdu2.scalelab.redhat.com f02-h18-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> keepalived-f02-h18-000-r640.rdu2.scalelab.redhat.com f02-h18-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> mdns-publisher-f02-h18-000-r640.rdu2.scalelab.redhat.com f02-h18-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> migrator-9d6c8f546-5p99k f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 200Mi <none> local-block-local-diskmaker-74ltv f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> local-block-local-provisioner-2wvnm f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> local-storage-operator-59f7658d-nm6xp f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> machine-config-daemon-g7xl6 f02-h18-000-r640.rdu2.scalelab.redhat.com 20m <none> 50Mi <none> certified-operators-rksh2 f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> d1aee90db641d090bace05cd7656a85291df8b3a0080dc577c2b6ecfc8tr4v8 f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> redhat-osbs-62459-m68kg f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-69034-ktwz5 f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> redhat-osbs-70118-gz25q f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 50Mi <none> alertmanager-main-1 f02-h18-000-r640.rdu2.scalelab.redhat.com 4m <none> 200Mi <none> alertmanager-main-2 f02-h18-000-r640.rdu2.scalelab.redhat.com 4m <none> 200Mi <none> kube-state-metrics-749954d685-592kz f02-h18-000-r640.rdu2.scalelab.redhat.com 2m <none> 40Mi <none> node-exporter-pggcw f02-h18-000-r640.rdu2.scalelab.redhat.com 8m <none> 180Mi <none> prometheus-adapter-5949648776-bd9pq f02-h18-000-r640.rdu2.scalelab.redhat.com 1m <none> 25Mi <none> prometheus-adapter-5949648776-dbv8q f02-h18-000-r640.rdu2.scalelab.redhat.com 1m <none> 25Mi <none> prometheus-k8s-1 f02-h18-000-r640.rdu2.scalelab.redhat.com 70m <none> 1Gi <none> thanos-querier-584cf5f85b-2pxzb f02-h18-000-r640.rdu2.scalelab.redhat.com 5m <none> 12Mi <none> multus-cqpw2 f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 150Mi <none> network-metrics-daemon-4gft6 f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 100Mi <none> network-check-source-7b56ddbc7b-hfcw9 f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 40Mi <none> network-check-target-85ppx f02-h18-000-r640.rdu2.scalelab.redhat.com 10m <none> 15Mi <none> nfd-worker-jjbwr f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> ovs-5wpmg f02-h18-000-r640.rdu2.scalelab.redhat.com 15m <none> 400Mi <none> sdn-gwjkh f02-h18-000-r640.rdu2.scalelab.redhat.com 100m <none> 200Mi <none> sriov-cni-jhglz f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> sriov-device-plugin-z4nq7 f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> sriov-network-config-daemon-ndgtw f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-cephfsplugin-provisioner-7d86f6dd86-7l6bj f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-cephfsplugin-r5cbl f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-rbdplugin-dc6qt f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> csi-rbdplugin-provisioner-7f95c9f9b7-bp7xs f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> noobaa-endpoint-74fb466895-kp6l2 f02-h18-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> noobaa-operator-6569ffb554-6rq72 f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> ocs-metrics-exporter-67bbf44d5d-jxkj8 f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> rook-ceph-crashcollector-1b3cdc1f3b7aa08eaf1c6ad2b3d76266-94sjn f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-6cc8c6fd64w6m f02-h18-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-mon-c-549df7ff7f-pkzwz f02-h18-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-operator-554cd454b5-xpt4s f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> rook-ceph-osd-0-5d477dd89b-v9zhh f02-h18-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-6788f6djhfzd f02-h18-000-r640.rdu2.scalelab.redhat.com 0 <none> 0 <none> rook-ceph-tools-5c4f4f859b-mq5p6 f02-h18-000-r640.rdu2.scalelab.redhat.com <none> <none> <none> <none> root@f02-h07-000-r640:~$
small update... in the case of RHEV as a provider, the importers spreading seems to be worst since it's working only on a single worker out of three for example on migration plan with 30VMs with rhev as a provider all the importers pods running on single worker-host > f02-h17-000-r640.rdu2.scalelab.redhat.com the 2 others worker doesn't involve at all in the migration plan : MTV build 2.1.0-44 oc get pods -owide |grep importer f02-h07-000-r640.rdu2.scalelab.redhat.com: Tue Aug 10 10:15:50 2021 importer-rhv-red01-vm-migration-18-group0-16174989-a60a-4371-b31f-fe6a8e32bb10 1/1 Running 0 15m 10.131.0.182 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-19-group0-9cde9334-2ff8-41af-bf53-9364da21ff75 1/1 Running 0 15m 10.131.0.181 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-2-group0-f0487732-403d-4209-bd4f-dfd6dc4b9a90 1/1 Running 0 16m 10.131.0.180 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-20-group0-5c39d3e9-65db-4385-afec-3f8218e5e7ef 1/1 Running 0 16m 10.131.0.179 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-21-group0-ff52e69e-15a5-4da0-819e-66c0a40937bd 1/1 Running 0 17m 10.131.0.178 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-23-group0-2cd5f21c-9690-4503-8b86-8726a77d37ff 1/1 Running 0 14m 10.131.0.184 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-24-group0-b0ae3a15-0f87-4879-b9a6-c7246aebe16c 1/1 Running 0 13m 10.131.0.185 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-25-group0-67548994-350b-47e7-b16f-15b7245f9afe 1/1 Running 0 14m 10.131.0.183 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-42-group0-f9aa8121-30b8-49e7-ba32-a87484d6db68 1/1 Running 0 15s 10.131.0.201 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-44-group0-1125fa3f-19e3-44c1-9382-bca096bce321 1/1 Running 0 2m17s 10.131.0.198 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-45-group0-5c2da9e2-b3bd-4cb3-8221-b0d7eae4a3c2 1/1 Running 0 3m30s 10.131.0.197 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-47-group0-6bf7710c-95e9-4d7a-bdfb-c91692fb9063 1/1 Running 0 5m26s 10.131.0.196 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-48-group0-1edb3df5-44e4-4490-bf71-615347bb1992 1/1 Running 0 8m24s 10.131.0.194 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-49-group0-0278d70f-3773-4d3a-835e-925124337dc7 1/1 Running 0 9m22s 10.131.0.193 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-5-group0-b6e397f8-3de4-4c38-9275-040f0c949a88 1/1 Running 0 10m 10.131.0.191 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-50-group0-ca5296c8-e01a-4f82-85c8-3f66a42fa296 1/1 Running 0 11m 10.131.0.190 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-51-group0-d879b8ed-5bc9-40b8-9e77-f2f3e9b06006 1/1 Running 0 12m 10.131.0.189 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-52-group0-66f514f5-811b-4c24-9bc8-0272128aba59 1/1 Running 0 12m 10.131.0.187 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none>
update... in the case of RHEV as a provider, the importers spreading seems to be worst since it's working only on a single worker out of three for example on migration plan with 30VMs with rhev as a provider all the importers pods running on single worker-host > f02-h17-000-r640.rdu2.scalelab.redhat.com the 2 others worker doesn't involve at all in the migration plan : MTV build 2.1.0-44 oc get pods -owide |grep importer f02-h07-000-r640.rdu2.scalelab.redhat.com: Tue Aug 10 10:15:50 2021 importer-rhv-red01-vm-migration-18-group0-16174989-a60a-4371-b31f-fe6a8e32bb10 1/1 Running 0 15m 10.131.0.182 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-19-group0-9cde9334-2ff8-41af-bf53-9364da21ff75 1/1 Running 0 15m 10.131.0.181 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-2-group0-f0487732-403d-4209-bd4f-dfd6dc4b9a90 1/1 Running 0 16m 10.131.0.180 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-20-group0-5c39d3e9-65db-4385-afec-3f8218e5e7ef 1/1 Running 0 16m 10.131.0.179 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-21-group0-ff52e69e-15a5-4da0-819e-66c0a40937bd 1/1 Running 0 17m 10.131.0.178 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-23-group0-2cd5f21c-9690-4503-8b86-8726a77d37ff 1/1 Running 0 14m 10.131.0.184 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-24-group0-b0ae3a15-0f87-4879-b9a6-c7246aebe16c 1/1 Running 0 13m 10.131.0.185 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-25-group0-67548994-350b-47e7-b16f-15b7245f9afe 1/1 Running 0 14m 10.131.0.183 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-42-group0-f9aa8121-30b8-49e7-ba32-a87484d6db68 1/1 Running 0 15s 10.131.0.201 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-44-group0-1125fa3f-19e3-44c1-9382-bca096bce321 1/1 Running 0 2m17s 10.131.0.198 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-45-group0-5c2da9e2-b3bd-4cb3-8221-b0d7eae4a3c2 1/1 Running 0 3m30s 10.131.0.197 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-47-group0-6bf7710c-95e9-4d7a-bdfb-c91692fb9063 1/1 Running 0 5m26s 10.131.0.196 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-48-group0-1edb3df5-44e4-4490-bf71-615347bb1992 1/1 Running 0 8m24s 10.131.0.194 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-49-group0-0278d70f-3773-4d3a-835e-925124337dc7 1/1 Running 0 9m22s 10.131.0.193 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-5-group0-b6e397f8-3de4-4c38-9275-040f0c949a88 1/1 Running 0 10m 10.131.0.191 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-50-group0-ca5296c8-e01a-4f82-85c8-3f66a42fa296 1/1 Running 0 11m 10.131.0.190 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-51-group0-d879b8ed-5bc9-40b8-9e77-f2f3e9b06006 1/1 Running 0 12m 10.131.0.189 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none> importer-rhv-red01-vm-migration-52-group0-66f514f5-811b-4c24-9bc8-0272128aba59 1/1 Running 0 12m 10.131.0.187 f02-h17-000-r640.rdu2.scalelab.redhat.com <none> <none>
We prefer the Kubernetes to manage scheduling and not provide custom tuning for this specific case
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days