Bug 1989531

+++ This bug was initially created as a clone of Bug #1986410 +++

Description of problem:

Pods created by latency tests should request available resources at the runtime and/or allows the user to specify the amount.

Version-Release number of selected component (if applicable):
4.8

How reproducible:
100%

Steps to Reproduce:
1. Run hwlatdetect tests on top of an SNO deployment with an existing performance profile


podman run --name cnf-container-tests  \
  --net=host  \
  -v /home/kni/cnf_tests_dir:/kubeconfig:Z  \
  -v /home/kni/cnf_tests_dir/junit:/junit:Z \
  -v /home/kni/cnf_tests_dir/report:/report:Z \
  -e KUBECONFIG=/kubeconfig/kubeconfig \
  -e IMAGE_REGISTRY=registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/ \
  -e CNF_TESTS_IMAGE=cnf-tests \
  -e LATENCY_TEST_RUN=true \
  -e LATENCY_TEST_RUNTIME=600 \
  -e OSLAT_MAXIMUM_LATENCY=20 \
  -e CYCLICTEST_MAXIMUM_LATENCY=20 \
  -e HWLATDETECT_MAXIMUM_LATENCY=20 \
  -e ROLE_WORKER_CNF=master \
  -e PERF_TEST_PROFILE=perfprofile-policy \
  -e DISCOVERY_MODE=true \
  quay.io/openshift-kni/cnf-tests \
  /usr/bin/test-run.sh  \
  -ginkgo.focus="\[performance\]\[config\]|\[performance\]\ Latency\ Test\ with\ the\ hwlatdetect" \
  --junit /junit -ginkgo.v


Applied performance profile spec:

  spec:
    cpu:
      isolated: 2-23,26-47
      reserved: 0-1,24-25
    globallyDisableIrqLoadBalancing: true
    hugepages:
      defaultHugepagesSize: 1G
      pages:
      - count: 16
        node: 0
        size: 1G
    machineConfigPoolSelector:
      pools.operator.machineconfiguration.openshift.io/master: ""
    nodeSelector:
      node-role.kubernetes.io/master: ""
    numa:
      topologyPolicy: restricted
    realTimeKernel:
      enabled: true

node resources:

    allocatable:
      cpu: "44"
      ephemeral-storage: "460405587616"
      hugepages-1Gi: 16Gi
      hugepages-2Mi: "0"
      memory: 80721928Ki
      pods: "250"
    capacity:
      cpu: "48"
      ephemeral-storage: 499572036Ki
      hugepages-1Gi: 16Gi
      hugepages-2Mi: "0"
      memory: 98625544Ki
      pods: "250"

hwlatdetect pod:

Name:         hwlatdetect-76t58
Namespace:    performance-addon-operators-testing
Priority:     0
Node:         <none>
Labels:       <none>
Annotations:  cpu-load-balancing.crio.io: true
              openshift.io/scc: node-exporter
Status:       Pending
IP:           
IPs:          <none>
Containers:
  hwlatdetect-runner:
    Image:      registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/cnf-tests
    Port:       <none>
    Host Port:  <none>
    Command:
      usr/bin/hwlatdetect-runner
    Args:
      -logtostderr=false
      -alsologtostderr=true
      -log_file=/host/hwlatdetect.log
      -hardlimit=20
      -threshold=20
    Limits:
      cpu:     43
      memory:  1Gi
    Requests:
      cpu:        43
      memory:     1Gi
    Environment:  <none>
    Mounts:
      /host from logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rz6hm (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  logs:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:  Directory
  kube-api-access-rz6hm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Guaranteed
Node-Selectors:              kubernetes.io/hostname=sno.kni-qe-1.lab.eng.rdu2.redhat.com
                             node-role.kubernetes.io/master=
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  54s (x2 over 55s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.



node actual resource usage:

oc adm top node
W0727 09:29:16.492930  642972 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
NAME                                   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
sno.kni-qe-1.lab.eng.rdu2.redhat.com   3557m        8%     29183Mi         37%   

Actual results:

hwlatdetect pod stays stuck in Pending because the requested resources are not available on the node

Expected results:
The requested resources by latency test pods should be set according to actual usage and/or allow user to override them.

Additional info: