Bug 1986410
| Summary: | Pods created by latency tests should request available resources at the moment and/or allows the user to specify the amount | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Marius Cornea <mcornea> | |
| Component: | CNF Platform Validation | Assignee: | Talor Itzhak <titzhak> | |
| Status: | CLOSED WONTFIX | QA Contact: | Shereen Haj Makhoul <shajmakh> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 4.8 | CC: | aos-bugs, shajmakh, sscheink | |
| Target Milestone: | --- | |||
| Target Release: | 4.9.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause:
The latency test pod was configured to always ask for the total amount of isolated CPUs minus one.
Consequence:
This behavior is not suited for all users.
Fix:
Allow specifying the number of isolated CPUs via the LATENCY_TEST_CPUS environment variable.
Result:
The users will be able to control the number of isolated CPUs and thus have better control over the latency test pod behavior.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1989531 (view as bug list) | Environment: | ||
| Last Closed: | 2024-04-30 18:04:53 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1989531 | |||
It seems like there was some bug under the Bugzilla automation (https://github.com/openshift-kni/cnf-features-deploy/pull/644#issuecomment-896226129), so I'm moving the status manually back to modified. Verification:
versions:
cnf-tests:v4.9.0-28
ocp : 4.9.0-rc.0
performance-addon-operator.v4.9.0
podman run -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e IMAGE_REGISTRY=registry-proxy.engineering.redhat.com/rh-osbs -e CNF_TESTS_IMAGE=openshift4-cnf-tests:v4.9.0-28 -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=120 -e OSLAT_MAXIMUM_LATENCY=20 -e DISCOVERY_MODE=true registry-proxy.engineering.redhat.com/rh-osbs/openshift4-cnf-tests:v4.9.0-28 usr/bin/test-run.sh -ginkgo.focus="hwlatdetec" -ginkgo.v
check the cpus :
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 2
memory: 1Gi
Environment: <none>
Mounts:
As one can notice, the default cpu used is 2.
if we set cpu number as LATENCY_TEST_CPUS=3 that will be reflected as the pod's cpus:
Limits:
cpu: 3
memory: 1Gi
Requests:
cpu: 3
memory: 1Gi
Same for oslat (cpu number >=2) & cyclictest.
OCP is no longer using Bugzilla and this bug appears to have been left in an orphaned state. If the bug is still relevant, please open a new issue in the OCPBUGS Jira project: https://issues.redhat.com/projects/OCPBUGS/summary |
Description of problem: Pods created by latency tests should request available resources at the runtime and/or allows the user to specify the amount. Version-Release number of selected component (if applicable): 4.8 How reproducible: 100% Steps to Reproduce: 1. Run hwlatdetect tests on top of an SNO deployment with an existing performance profile podman run --name cnf-container-tests \ --net=host \ -v /home/kni/cnf_tests_dir:/kubeconfig:Z \ -v /home/kni/cnf_tests_dir/junit:/junit:Z \ -v /home/kni/cnf_tests_dir/report:/report:Z \ -e KUBECONFIG=/kubeconfig/kubeconfig \ -e IMAGE_REGISTRY=registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/ \ -e CNF_TESTS_IMAGE=cnf-tests \ -e LATENCY_TEST_RUN=true \ -e LATENCY_TEST_RUNTIME=600 \ -e OSLAT_MAXIMUM_LATENCY=20 \ -e CYCLICTEST_MAXIMUM_LATENCY=20 \ -e HWLATDETECT_MAXIMUM_LATENCY=20 \ -e ROLE_WORKER_CNF=master \ -e PERF_TEST_PROFILE=perfprofile-policy \ -e DISCOVERY_MODE=true \ quay.io/openshift-kni/cnf-tests \ /usr/bin/test-run.sh \ -ginkgo.focus="\[performance\]\[config\]|\[performance\]\ Latency\ Test\ with\ the\ hwlatdetect" \ --junit /junit -ginkgo.v Applied performance profile spec: spec: cpu: isolated: 2-23,26-47 reserved: 0-1,24-25 globallyDisableIrqLoadBalancing: true hugepages: defaultHugepagesSize: 1G pages: - count: 16 node: 0 size: 1G machineConfigPoolSelector: pools.operator.machineconfiguration.openshift.io/master: "" nodeSelector: node-role.kubernetes.io/master: "" numa: topologyPolicy: restricted realTimeKernel: enabled: true node resources: allocatable: cpu: "44" ephemeral-storage: "460405587616" hugepages-1Gi: 16Gi hugepages-2Mi: "0" memory: 80721928Ki pods: "250" capacity: cpu: "48" ephemeral-storage: 499572036Ki hugepages-1Gi: 16Gi hugepages-2Mi: "0" memory: 98625544Ki pods: "250" hwlatdetect pod: Name: hwlatdetect-76t58 Namespace: performance-addon-operators-testing Priority: 0 Node: <none> Labels: <none> Annotations: cpu-load-balancing.crio.io: true openshift.io/scc: node-exporter Status: Pending IP: IPs: <none> Containers: hwlatdetect-runner: Image: registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/cnf-tests Port: <none> Host Port: <none> Command: usr/bin/hwlatdetect-runner Args: -logtostderr=false -alsologtostderr=true -log_file=/host/hwlatdetect.log -hardlimit=20 -threshold=20 Limits: cpu: 43 memory: 1Gi Requests: cpu: 43 memory: 1Gi Environment: <none> Mounts: /host from logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rz6hm (ro) Conditions: Type Status PodScheduled False Volumes: logs: Type: HostPath (bare host directory volume) Path: /var/log HostPathType: Directory kube-api-access-rz6hm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Guaranteed Node-Selectors: kubernetes.io/hostname=sno.kni-qe-1.lab.eng.rdu2.redhat.com node-role.kubernetes.io/master= Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 54s (x2 over 55s) default-scheduler 0/1 nodes are available: 1 Insufficient cpu. node actual resource usage: oc adm top node W0727 09:29:16.492930 642972 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% sno.kni-qe-1.lab.eng.rdu2.redhat.com 3557m 8% 29183Mi 37% Actual results: hwlatdetect pod stays stuck in Pending because the requested resources are not available on the node Expected results: The requested resources by latency test pods should be set according to actual usage and/or allow user to override them. Additional info: