Bug 2115805

Summary: Add E2E test case for Telco Friendly workload specific API
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: Performance Addon OperatorAssignee: Martin Sivák <msivak>
Status: CLOSED ERRATA QA Contact: Niranjan Mallapadi Raghavender <mniranja>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.12CC: grajaiya, mniranja, shajmakh
Target Milestone: ---   
Target Release: 4.11.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-29 06:46:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2105967    
Bug Blocks:    

Comment 4 Niranjan Mallapadi Raghavender 2022-08-24 16:32:34 UTC 
1. Setup ocp cluster 

[root@dell-r640-051 cluster-node-tuning-operator]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.1    True        False         4h38m   Cluster version is 4.11.1

2. Clone https://github.com/openshift/cluster-node-tuning-operator.git 

3. Switch to 4.11 branch
[root@dell-r640-051 cluster-node-tuning-operator]# git branch
* 4_11

4. Run ginkgo $NO_COLOR --v -r  -requireSuite test/e2e/performanceprofile/functests/2_performance_update/ 

Workload Hints Output. 

[rfe_id:28761][performance] Updating parameters in performance profile WorkloadHints when RealTime Workload with RealTime Kernel set to false
  [test_id:50991][crit:high][vendor:cnf-qe][level:acceptance]should update kernel arguments and tuned accordingly
  /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:570
Aug 24 10:53:25.362: [INFO]: MCP "worker" is targeting 3 node(s)
Aug 24 10:53:25.374: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Saving the old performance profile
STEP: Modifying profile
STEP: Updating the performance profile
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 10:53:25.401: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting for MCP being updated
Aug 24 10:53:55.420: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
Aug 24 10:59:25.499: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 10:59:25.589: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.hung_task_timeout_secs"
Aug 24 10:59:25.683: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.nmi_watchdog"
Aug 24 10:59:25.775: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.sched_rt_runtime_us"
Aug 24 10:59:25.862: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n vm.stat_interval"
Aug 24 10:59:25.944: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.hung_task_timeout_secs"
Aug 24 10:59:26.040: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.nmi_watchdog"
Aug 24 10:59:26.116: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.sched_rt_runtime_us"
Aug 24 10:59:26.195: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n vm.stat_interval"
Aug 24 10:59:26.272: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
Aug 24 10:59:26.367: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 10:59:26.446: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
Aug 24 10:59:26.531: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 10:59:26.625: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 10:59:26.741: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
Aug 24 10:59:26.818: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Restoring the old performance profile
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 10:59:26.960: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting when mcp finishes updates
Aug 24 10:59:56.980: [INFO]: MCP "worker-cnf" is targeting 2 node(s)

• [SLOW TEST:961.769 seconds]
[rfe_id:28761][performance] Updating parameters in performance profile
/root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:36
  WorkloadHints
  /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:564
    when RealTime Workload with RealTime Kernel set to false
    /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:569
      [test_id:50991][crit:high][vendor:cnf-qe][level:acceptance]should update kernel arguments and tuned accordingly
      /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:570
------------------------------
[rfe_id:28761][performance] Updating parameters in performance profile WorkloadHints when HighPower Consumption workload enabled
  [test_id:50992][crit:high][vendor:cnf-qe][level:acceptance]should update kernel arguments and tuned accordingly
  /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:603
Aug 24 11:09:27.131: [INFO]: MCP "worker" is targeting 3 node(s)
Aug 24 11:09:27.143: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Saving the old performance profile
STEP: Modifying profile
STEP: Updating the performance profile
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 11:09:27.169: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting for MCP being updated
Aug 24 11:09:57.188: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
Aug 24 11:19:27.327: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 11:19:27.424: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.sched_rt_runtime_us"
Aug 24 11:19:27.524: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n vm.stat_interval"
Aug 24 11:19:27.607: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.hung_task_timeout_secs"
Aug 24 11:19:27.690: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.nmi_watchdog"
Aug 24 11:19:27.769: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.sched_rt_runtime_us"
Aug 24 11:19:27.851: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n vm.stat_interval"
Aug 24 11:19:27.928: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.hung_task_timeout_secs"
Aug 24 11:19:28.012: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.nmi_watchdog"
Aug 24 11:19:28.096: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
Aug 24 11:19:28.173: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 11:19:28.251: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
Aug 24 11:19:28.328: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 11:19:28.422: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 11:19:28.545: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
Aug 24 11:19:28.622: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Restoring the old performance profile
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 11:19:28.747: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting when mcp finishes updates
Aug 24 11:19:58.767: [INFO]: MCP "worker-cnf" is targeting 2 node(s)

• [SLOW TEST:1141.820 seconds]
[rfe_id:28761][performance] Updating parameters in performance profile
/root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:36
  WorkloadHints
  /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:564
    when HighPower Consumption workload enabled
    /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:602
      [test_id:50992][crit:high][vendor:cnf-qe][level:acceptance]should update kernel arguments and tuned accordingly
      /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:603
------------------------------
[rfe_id:28761][performance] Updating parameters in performance profile WorkloadHints when realtime and high power consumption enabled
  [test_id:50993][crit:high][vendor:cnf-qe][level:acceptance]should update kernel arguments and tuned accordingly
  /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:636
Aug 24 11:28:28.952: [INFO]: MCP "worker" is targeting 3 node(s)
Aug 24 11:28:28.964: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Saving the old performance profile
STEP: Patching the performance profile with workload hints
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 11:28:28.995: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting when mcp finishes updates
Aug 24 11:28:59.015: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
Aug 24 11:38:29.161: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 11:38:29.262: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.hung_task_timeout_secs"
Aug 24 11:38:29.377: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.nmi_watchdog"
Aug 24 11:38:29.485: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.sched_rt_runtime_us"
Aug 24 11:38:29.568: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n vm.stat_interval"
Aug 24 11:38:29.649: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.hung_task_timeout_secs"
Aug 24 11:38:29.736: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.nmi_watchdog"
Aug 24 11:38:29.816: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n kernel.sched_rt_runtime_us"
Aug 24 11:38:29.903: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Executing he command "sysctl -n vm.stat_interval"
Aug 24 11:38:29.994: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
Aug 24 11:38:30.081: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 11:38:30.161: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Restoring the old performance profile
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 11:38:30.279: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting when mcp finishes updates
Aug 24 11:39:00.302: [INFO]: MCP "worker-cnf" is targeting 2 node(s)

• [SLOW TEST:1171.493 seconds]
[rfe_id:28761][performance] Updating parameters in performance profile
/root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:36
  WorkloadHints
  /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:564
    when realtime and high power consumption enabled
    /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:635
      [test_id:50993][crit:high][vendor:cnf-qe][level:acceptance]should update kernel arguments and tuned accordingly
      /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:636
------------------------------
[rfe_id:28761][performance] Updating parameters in performance profile Offlined CPU API
  [disruptive] should set offline cpus after deploy PAO
  /root/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/2_performance_update/updating_profile.go:708
Aug 24 11:48:00.446: [INFO]: MCP "worker" is targeting 3 node(s)
Aug 24 11:48:00.457: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
Aug 24 11:48:00.463: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 11:48:00.563: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Enable UserLevelNetworking and add Devices in Profile
STEP: Updating the performance profile
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 11:48:00.711: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting for MCP being updated
Aug 24 11:48:30.732: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
Aug 24 12:06:30.990: [INFO]: found mcd machine-config-daemon-ss97c for node ocp-worker-0.demo.lab.mniranja
Aug 24 12:06:31.119: [INFO]: found mcd machine-config-daemon-5cjb5 for node ocp-worker-1.demo.lab.mniranja
STEP: Reverting the Profile
STEP: Applying changes in performance profile and waiting until mcp will start updating
Aug 24 12:06:31.271: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
STEP: Waiting when mcp finishes updates
Aug 24 12:07:01.290: [INFO]: MCP "worker-cnf" is targeting 2 node(s)
Comment 7 errata-xmlrpc 2022-08-29 06:46:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.11.2 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6143