Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1986399

Summary: Latency testing pods should use all cpu balancing, irq balancing etc. annotations
Product: OpenShift Container Platform Reporter: Martin Sivák <msivak>
Component: CNF Platform ValidationAssignee: Talor Itzhak <titzhak>
Status: CLOSED ERRATA QA Contact: Niranjan Mallapadi Raghavender <mniranja>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.8CC: aos-bugs, dgonyier, shajmakh
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Latency testing pods were not configured with CPU balancing and IRQ balancing annotations. Consequence: Not disabling the load-balancing results with an additional workload on the pods` CPUs. Fix: Adding the missing annotations that will disable the load-balancing. Result: Disabling the load-balancing reduces the additional workload on the pods` CPUs.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-18 19:58:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1986587    

Description Martin Sivák 2021-07-27 12:53:13 UTC 
Description of problem:

oslat, cyclictest and hwlatdetect testing pods started using the procedure from https://docs.openshift.com/container-platform/4.7/scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.html#cnf-performing-end-to-end-tests-running-the-latency_tests need to disable cpu balancing and irq balancing on the used cpus.

The procedure is described in:

https://docs.openshift.com/container-platform/4.7/scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.html#performance-addon-operator-disabling-cpu-load-balancing-for-dpdk_cnf-master

and

https://docs.openshift.com/container-platform/4.7/scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.html#disabling_interrupt_processing_for_individual_pods_cnf-master


The annotations are defined in https://github.com/openshift-kni/performance-addon-operators/blob/master/functests/4_latency/latency.go#L272

Notice the annotation used the old value (true) instead of "disable" and there is no annotation for irq handling.


Version-Release number of selected component (if applicable):

master branch, all versions starting with 4.7
Comment 3 Shereen Haj Makhoul 2021-10-07 10:44:35 UTC 
Verification
-------------

Version:
Client Version: 4.9.0-0.nightly-2021-08-26-040328
Server Version: 4.9.0-0.nightly-2021-08-26-040328
Kubernetes Version: v1.22.0-rc.0+33d0ffa


Steps:
- podman run  -v $KUBECONFIG:/root/kubeconfig:Z -e KUBECONFIG=/root/kubeconfig -e IMAGE_REGISTRY=registry-proxy.engineering.redhat.com/rh-osbs/ -e CNF_TESTS_IMAGE=openshift4-cnf-tests:v4.9.0-30 -e PERF_TEST_PROFILE=performance -e ROLE_WORKER_CNF=worker-cnf -e LATENCY_TEST_RUN=true -e LATENCY_TEST_RUNTIME=20  -e DISCOVERY_MODE=true registry-proxy.engineering.redhat.com/rh-osbs/openshift4-cnf-tests:v4.9.0-30 usr/bin/test-run.sh -ginkgo.focus="\[performance\]\ Latency\ Test" -ginkgo.v

Results:

oc describe pods oslat-dnj56 -n performance-addon-operators-testing
Name:         oslat-dnj56
Namespace:    performance-addon-operators-testing
....
Annotations:  cpu-load-balancing.crio.io: disable
              irq-load-balancing.crio.io: disable
              k8s.ovn.org/pod-networks:
                {"default":{"ip_addresses":["10.135.0.81/23"],"mac_address":"0a:58:0a:87:00:51","gateway_ips":["10.135.0.1"],"ip_address":"10.135.0.81/23"...
              k8s.v1.cni.cncf.io/network-status:
                [{
=======================================


[root@cnfdt07-installer ~]# oc describe pods cyclictest-ktnb5 -n performance-addon-operators-testing
Name:         cyclictest-ktnb5
Namespace:    performance-addon-operators-testing
....
Annotations:  cpu-load-balancing.crio.io: disable
              irq-load-balancing.crio.io: disable
              k8s.ovn.org/pod-networks:
                {"default":{"ip_addresses":["10.135.0.82/23"],"mac_address":"0a:58:0a:87:00:52","gateway_ips":["10.135.0.1"],"ip_address":"10.135.0.82/23"...
              k8s.v1.cni.cncf.io/network-status:
                [{
========================================


[root@cnfdt07-installer ~]# oc describe pods hwlatdetect-ckvs2 -n performance-addon-operators-testing
Name:         hwlatdetect-ckvs2
Namespace:    performance-addon-operators-testing
....
Annotations:  cpu-load-balancing.crio.io: disable
              irq-load-balancing.crio.io: disable
              k8s.ovn.org/pod-networks:
                {"default":{"ip_addresses":["10.135.0.83/23"],"mac_address":"0a:58:0a:87:00:53","gateway_ips":["10.135.0.1"],"ip_address":"10.135.0.83/23"...
              k8s.v1.cni.cncf.io/network-status:
                [{

As observed in the above results, the latency pods now (oslat , cyclictest & hwlatdetect) indicate disabled load balancing.
Comment 4 Dwaine Gonyier 2021-10-07 13:26:58 UTC 
Just as FYI, here is the link to the commit diff for the fixed functest code.

https://github.com/openshift-kni/performance-addon-operators/commit/4e4a933be1adad51a30ce9000cb78ea20991b0e9#diff-a11540b25a190b62a4789801be3fa3b70b91847dc903680998aa8a5837550646
Comment 6 errata-xmlrpc 2021-10-18 19:58:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9 low-latency extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3846