1973502 – Unable to change the net queue size to reserved cpu count on supported Network devices on Baremetal using PAO

Bug 1973502 - Unable to change the net queue size to reserved cpu count on supported Network devices on Baremetal using PAO

Summary: Unable to change the net queue size to reserved cpu count on supported Networ...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Performance Addon Operator
Sub Component:
Version:	4.8
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Yanir Quinn
QA Contact:	Niranjan Mallapadi Raghavender
Docs Contact:
URL:
Whiteboard:
Depends On:	1974718 1975803
Blocks:
TreeView+	depends on / blocked

Reported:	2021-06-18 02:57 UTC by Niranjan Mallapadi Raghavender
Modified:	2021-09-15 07:50 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-09-15 07:48:56 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift-kni performance-addon-operators pull 652	0	None	Merged	e2e test Automation for netqueues	2021-09-15 07:50:16 UTC

Description Niranjan Mallapadi Raghavender 2021-06-18 02:57:40 UTC

Description of problem:

Unable to change the net queue size to reserved cpu count on supported Network devices on Baremetal  using PAO 



Version-Release number of selected component (if applicable):
ocp-4.8.0-fc.9



How reproducible:


Steps to Reproduce:
1. Install ocp-4.8.0-fc9 on Baremetal with sriov network devices supporting net queues
2. Install Performance addon operator from brew with below profile:
Spec:
  Additional Kernel Args:
    nmi_watchdog=0
    audit=0
    mce=off
    processor.max_cstate=1
    idle=poll
    intel_idle.max_cstate=0
  Cpu:
    Isolated:  3-4
    Reserved:  0-2
  Hugepages:
    Default Hugepages Size:  1G
    Pages:
      Count:  1
      Node:   0
      Size:   1G
      Count:  2
      Size:   2M
  Net:
    User Level Networking:  true
  Node Selector:
    node-role.kubernetes.io/worker-cnf:
  Numa:
    Topology Policy:  single-numa-node
  Real Time Kernel:
    Enabled:  true

3. Apply the profile. 

Actual results:

 oc exec -ti  tuned-282x6 -- ethtool -l con1
Channel parameters for con1:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          1
Combined:       80
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          1
Combined:       80

Expected results:

 oc exec -ti  tuned-282x6 -- ethtool -l con1
Channel parameters for con1:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          1
Combined:       80
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          1
Combined:       1

Additional info:

Tuned Logs:

oc logs tuned-282x6
I0616 13:29:39.254496   32635 trace.go:205] Trace[1298498081]: "Reflector ListAndWatch" name:github.com/openshift/cluster-node-tuning-operator/pkg/generated/informers/externalversions/factory.go:101 (16-Jun-2021 13:29:23.857) (total time: 15396ms):
Trace[1298498081]: ---"Objects listed" 15395ms (13:29:00.253)
Trace[1298498081]: [15.396029051s] [15.396029051s] END
I0616 13:29:39.254035   32635 tuned.go:295] disabling system tuned...
I0616 13:29:39.254625   32635 trace.go:205] Trace[2019727887]: "Reflector ListAndWatch" name:github.com/openshift/cluster-node-tuning-operator/pkg/generated/informers/externalversions/factory.go:101 (16-Jun-2021 13:29:23.857) (total time: 15396ms):
Trace[2019727887]: ---"Objects listed" 15396ms (13:29:00.254)
Trace[2019727887]: [15.396201765s] [15.396201765s] END
I0616 13:29:39.255065   32635 tuned.go:948] started events processor
I0616 13:29:39.255105   32635 tuned.go:312] extracting Tuned profiles
I0616 13:29:39.256694   32635 tuned.go:991] started controller
I0616 13:29:39.704283   32635 tuned.go:390] written "/etc/tuned/recommend.d/50-openshift.conf" to set Tuned profile openshift-node-performance-performance
I0616 13:29:40.387745   32635 tuned.go:638] re-applying profile () as the previous application did not complete
I0616 13:29:40.387782   32635 tuned.go:403] starting tuned...
2021-06-16 13:29:40,524 INFO     tuned.daemon.application: dynamic tuning is globally disabled
2021-06-16 13:29:40,533 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
2021-06-16 13:29:40,534 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
2021-06-16 13:29:40,534 INFO     tuned.daemon.daemon: Using 'openshift-node-performance-performance' profile
2021-06-16 13:29:40,537 INFO     tuned.profiles.loader: loading profile: openshift-node-performance-performance
2021-06-16 13:29:40,628 INFO     tuned.daemon.controller: starting controller
2021-06-16 13:29:40,628 INFO     tuned.daemon.daemon: starting tuning
2021-06-16 13:29:42,061 INFO     tuned.plugins.base: instance cpu: assigning devices cpu54, cpu25, cpu78, cpu58, cpu23, cpu46, cpu22, cpu17, cpu10, cpu70, cpu35, cpu21, cpu42, cpu13, cpu6, cpu7, cpu47, cpu64, cpu3, cpu66, cpu37, cpu53, cpu61, cpu68, cpu45, cpu48, cpu20, cpu55, cpu57, cpu26, cpu60, cpu44, cpu79, cpu72, cpu36, cpu31, cpu32, cpu39, cpu49, cpu63, cpu67, cpu16, cpu51, cpu28, cpu52, cpu12, cpu0, cpu24, cpu1, cpu65, cpu18, cpu77, cpu38, cpu62, cpu11, cpu75, cpu29, cpu76, cpu43, cpu69, cpu74, cpu2, cpu73, cpu40, cpu41, cpu27, cpu34, cpu30, cpu56, cpu59, cpu14, cpu4, cpu8, cpu19, cpu5, cpu9, cpu71, cpu50, cpu33, cpu15
2021-06-16 13:29:42,075 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform
2021-06-16 13:29:42,245 INFO     tuned.plugins.base: instance disk: assigning devices sda, sdb
2021-06-16 13:29:42,366 INFO     tuned.plugins.base: instance net: assigning devices ens8f0, eno3, con1, ens1f0, eno1, eno4, ens1f1, eno2
2021-06-16 13:30:13,701 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu54'
2021-06-16 13:30:13,842 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu22'
2021-06-16 13:30:13,987 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu17'
2021-06-16 13:30:14,132 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu10'
2021-06-16 13:30:14,259 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu70'
2021-06-16 13:30:14,410 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu13'
2021-06-16 13:30:14,528 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu47'
2021-06-16 13:30:14,629 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu64'
2021-06-16 13:30:14,750 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu3'
2021-06-16 13:30:14,854 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu48'
2021-06-16 13:30:14,974 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu20'
2021-06-16 13:30:15,120 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu57'
2021-06-16 13:30:15,234 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu26'
2021-06-16 13:30:15,357 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu44'
2021-06-16 13:30:15,472 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu79'
2021-06-16 13:30:15,600 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu72'
2021-06-16 13:30:15,732 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu31'
2021-06-16 13:30:15,833 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu39'
2021-06-16 13:30:15,925 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu49'
2021-06-16 13:30:16,024 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu67'
2021-06-16 13:30:16,101 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu51'
2021-06-16 13:30:16,203 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu28'
2021-06-16 13:30:16,319 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu52'
2021-06-16 13:30:16,401 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu12'
2021-06-16 13:30:16,482 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu24'
2021-06-16 13:30:16,559 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu77'
2021-06-16 13:30:16,632 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu65'
2021-06-16 13:30:16,710 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu18'
2021-06-16 13:30:16,824 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu38'
2021-06-16 13:30:16,916 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu75'
2021-06-16 13:30:17,022 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu29'
2021-06-16 13:30:17,124 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu76'
2021-06-16 13:30:17,189 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu69'
2021-06-16 13:30:17,269 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu74'
2021-06-16 13:30:17,339 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu73'
2021-06-16 13:30:17,421 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu41'
2021-06-16 13:30:17,492 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu27'
2021-06-16 13:30:17,567 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu59'
2021-06-16 13:30:17,663 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu56'
2021-06-16 13:30:17,738 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu14'
2021-06-16 13:30:17,837 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu71'
2021-06-16 13:30:17,936 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu50'
2021-06-16 13:30:18,020 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu15'
2021-06-16 13:30:18,092 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu25'
2021-06-16 13:30:18,170 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu78'
2021-06-16 13:30:18,274 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu58'
2021-06-16 13:30:18,376 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu23'
2021-06-16 13:30:18,477 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu46'
2021-06-16 13:30:18,562 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu35'
2021-06-16 13:30:18,656 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu21'
2021-06-16 13:30:18,752 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu42'
2021-06-16 13:30:18,839 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu6'
2021-06-16 13:30:18,924 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu7'
2021-06-16 13:30:19,015 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu66'
2021-06-16 13:30:19,110 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu37'
2021-06-16 13:30:19,200 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu53'
2021-06-16 13:30:19,296 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu61'
2021-06-16 13:30:19,407 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu68'
2021-06-16 13:30:19,521 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu45'
2021-06-16 13:30:19,638 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu55'
2021-06-16 13:30:19,735 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu60'
2021-06-16 13:30:19,835 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu36'
2021-06-16 13:30:19,920 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu32'
2021-06-16 13:30:20,014 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu63'
2021-06-16 13:30:20,111 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu16'
2021-06-16 13:30:20,196 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu0'
2021-06-16 13:30:20,294 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu1'
2021-06-16 13:30:20,393 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu62'
2021-06-16 13:30:20,491 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu11'
2021-06-16 13:30:20,576 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu43'
2021-06-16 13:30:20,651 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu40'
2021-06-16 13:30:20,722 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu34'
2021-06-16 13:30:20,808 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu30'
2021-06-16 13:30:20,889 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu4'
2021-06-16 13:30:20,967 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu8'
2021-06-16 13:30:21,038 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu19'
2021-06-16 13:30:21,095 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu5'
2021-06-16 13:30:21,182 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu9'
2021-06-16 13:30:21,264 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu33'
2021-06-16 13:30:21,332 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu2'
2021-06-16 13:30:21,332 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 3
2021-06-16 13:30:21,338 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist
2021-06-16 13:30:21,338 ERROR    tuned.plugins.plugin_sysctl: sysctl option net.core.busy_read will not be set, failed to read the original value.
2021-06-16 13:30:21,339 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist
2021-06-16 13:30:21,339 ERROR    tuned.plugins.plugin_sysctl: sysctl option net.core.busy_poll will not be set, failed to read the original value.
2021-06-16 13:30:21,339 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist
2021-06-16 13:30:21,339 ERROR    tuned.plugins.plugin_sysctl: sysctl option kernel.numa_balancing will not be set, failed to read the original value.
2021-06-16 13:30:21,339 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2021-06-16 13:30:21,578 ERROR    tuned.units.manager: BUG: Unhandled exception in start_tuning: invalid literal for int() with base 10: 'n/a'
2021-06-16 13:30:21,578 ERROR    tuned.units.manager: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tuned/units/manager.py", line 119, in _try_call
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/instance/instance.py", line 78, in apply_tuning
    self._plugin.instance_apply_tuning(self)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/base.py", line 261, in instance_apply_tuning
    self._instance_apply_static(instance)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/base.py", line 318, in _instance_apply_static
    self._execute_all_device_commands(instance, instance.assigned_devices)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/base.py", line 441, in _execute_all_device_commands
    self._execute_device_command(instance, command, device, new_value)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/base.py", line 502, in _execute_device_command
    command["custom"](True, new_value, device, False, False)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_net.py", line 468, in _channels
    return self._custom_parameters("channels", start, value, device, verify)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_net.py", line 423, in _custom_parameters
    dev_params = params_current)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_net.py", line 400, in _set_device_parameters
    if context == "channels" and (int(dev_params[next(iter(d))]) == 0 or str(dev_params[next(iter(d))]) == 'n/a'):
ValueError: invalid literal for int() with base 10: 'n/a'

2021-06-16 13:30:21,603 WARNING  tuned.plugins.plugin_vm: Option 'transparent_hugepages' is not supported on current hardware.
2021-06-16 13:30:21,606 INFO     tuned.plugins.plugin_systemd: setting 'CPUAffinity' to '0 1 2 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79' in the '/etc/systemd/system.conf'
2021-06-16 13:30:25,559 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/cpu-partitioning/script.sh' with arguments '['start']'
2021-06-16 13:30:26,309 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2021-06-16 13:30:26,312 INFO     tuned.daemon.daemon: static tuning from profile 'openshift-node-performance-performance' applied

Comment 4 Martin Sivák 2021-06-24 09:10:36 UTC

The NTO change is merged, a nightly should get the fix soon.

Comment 5 Niranjan Mallapadi Raghavender 2021-06-28 08:50:07 UTC

version:
[root@bkr-hv03 pao]# oc version
Client Version: 4.8.0-rc.1
Server Version: 4.8.0-rc.1
Kubernetes Version: v1.21.0-rc.0+766a5fe
pao version: v4.8.0-41



Performance Profile:

spec:
  cpu:
    isolated: 2-39,42-79
    reserved: 0-1,40-41
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 1
      size: 1G
  net:
    userLevelNetworking: true
  nodeSelector:
    node-role.kubernetes.io/worker-cnf: ""
  numa:
    topologyPolicy: restricted
  realTimeKernel:
    enabled: true


Nodes:
NAME                                      STATUS   ROLES               AGE   VERSION
helix06.lab.eng.tlv2.redhat.com           Ready    worker,worker-cnf   24h   v1.21.0-rc.0+766a5fe
helix07.lab.eng.tlv2.redhat.com           Ready    worker,worker-cnf   24h   v1.21.0-rc.0+766a5fe
master-0.hlxcl1.lab.eng.tlv2.redhat.com   Ready    master              25h   v1.21.0-rc.0+766a5fe
master-1.hlxcl1.lab.eng.tlv2.redhat.com   Ready    master              25h   v1.21.0-rc.0+766a5fe
master-2.hlxcl1.lab.eng.tlv2.redhat.com   Ready    master              25h   v1.21.0-rc.0+766a5fe

Tuned Pods:
NAME                                            READY   STATUS    RESTARTS   AGE   IP            NODE                                      NOMINATED NODE   READINESS GATES
cluster-node-tuning-operator-665f8c95b4-hxqj6   1/1     Running   0          23h   10.128.0.58   master-1.hlxcl1.lab.eng.tlv2.redhat.com   <none>           <none>
tuned-9x7mf                                     1/1     Running   1          25h   10.46.55.13   master-1.hlxcl1.lab.eng.tlv2.redhat.com   <none>           <none>
tuned-gmng7                                     1/1     Running   2          24h   10.46.55.7    helix07.lab.eng.tlv2.redhat.com           <none>           <none>
tuned-gtpdh                                     1/1     Running   3          24h   10.46.55.6    helix06.lab.eng.tlv2.redhat.com           <none>           <none>
tuned-m8xn7                                     1/1     Running   1          25h   10.46.55.14   master-2.hlxcl1.lab.eng.tlv2.redhat.com   <none>           <none>
tuned-wsdtx                                     1/1     Running   1          25h   10.46.55.12   master-0.hlxcl1.lab.eng.tlv2.redhat.com   <none>           <none>


helix06.lab.eng.tlv2.redhat.com
===============================

List of Network devices: eno1np0 eno2np1 ens7f3 ens4f1 ens7f1 ens7f2 ens4f0 ens7f0 

Jun 28 04:36:55.073: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l eno1np0]' (err=<nil>):
  stdout=Channel parameters for eno1np0:
Pre-set maximums:
RX:		37
TX:		37
Other:		n/a
Combined:	74
Current hardware settings:
RX:		0
TX:		0
Other:		n/a
Combined:	4

Jun 28 04:36:57.207: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l eno1np0 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=4


channelCurrentCombined =  4
d = eno2np1
Jun 28 04:36:59.197: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l eno2np1]' (err=<nil>):
  stdout=Channel parameters for eno2np1:
Pre-set maximums:
RX:		37
TX:		37
Other:		n/a
Combined:	74
Current hardware settings:
RX:		0
TX:		0
Other:		n/a
Combined:	4


Jun 28 04:37:01.342: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l eno2np1 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=4


channelCurrentCombined =  4
d = ens7f3
Jun 28 04:37:03.617: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l ens7f3]' (err=<nil>):
  stdout=Channel parameters for ens7f3:
Pre-set maximums:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a
Current hardware settings:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a


Jun 28 04:37:05.866: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l ens7f3 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=n/a


unable to retrieve current multi-purpose channels hardware settings
d = ens4f1
Jun 28 04:37:08.008: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l ens4f1]' (err=<nil>):
  stdout=Channel parameters for ens4f1:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	63
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	4


Jun 28 04:37:10.168: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l ens4f1 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=4


channelCurrentCombined =  4
d = ens7f1
Jun 28 04:37:12.430: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l ens7f1]' (err=<nil>):
  stdout=Channel parameters for ens7f1:
Pre-set maximums:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a
Current hardware settings:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a


Jun 28 04:37:14.592: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l ens7f1 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=n/a


unable to retrieve current multi-purpose channels hardware settings
d = ens7f2
Jun 28 04:37:16.841: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l ens7f2]' (err=<nil>):
  stdout=Channel parameters for ens7f2:
Pre-set maximums:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a
Current hardware settings:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a


Jun 28 04:37:18.899: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l ens7f2 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=n/a


unable to retrieve current multi-purpose channels hardware settings
d = ens4f0
Jun 28 04:37:21.195: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l ens4f0]' (err=<nil>):
  stdout=Channel parameters for ens4f0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	63
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	4


Jun 28 04:37:23.448: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l ens4f0 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=4


channelCurrentCombined =  4
d = ens7f0
Jun 28 04:37:25.737: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- ethtool -l ens7f0]' (err=<nil>):
  stdout=Channel parameters for ens7f0:
Pre-set maximums:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a
Current hardware settings:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a


Jun 28 04:37:27.797: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gtpdh -- bash -c ethtool -l ens7f0 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=n/a


unable to retrieve current multi-purpose channels hardware settings

Jun 28 04:37:39.260: [INFO]: run command 'oc [exec -i -n openshift-cluster-node-tuning-operator tuned-gmng7 -- bash -c ethtool -l ens1f1 | sed -n '/Current hardware settings:/,/Combined/{s/^Combined:\s*//p}']' (err=<nil>):
  stdout=n/a

unable to retrieve current multi-purpose channels hardware settings


As we can see from the above output the devices eno1np0, eno2np1, ens4f1, ens4f0  have their netqueues count equal to reserved cpu count which is 4. 


The oc logs on the tuned pod running on worker node doesn't show any traceback as seen in #Description

2021-06-28 08:01:27,268 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu46'
2021-06-28 08:01:27,270 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 2
2021-06-28 08:01:27,285 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'net.core.busy_read', the parameter does not exist
2021-06-28 08:01:27,286 ERROR    tuned.plugins.plugin_sysctl: sysctl option net.core.busy_read will not be set, failed to read the original value.
2021-06-28 08:01:27,286 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'net.core.busy_poll', the parameter does not exist
2021-06-28 08:01:27,286 ERROR    tuned.plugins.plugin_sysctl: sysctl option net.core.busy_poll will not be set, failed to read the original value.
2021-06-28 08:01:27,286 ERROR    tuned.plugins.plugin_sysctl: Failed to read sysctl parameter 'kernel.numa_balancing', the parameter does not exist
2021-06-28 08:01:27,286 ERROR    tuned.plugins.plugin_sysctl: sysctl option kernel.numa_balancing will not be set, failed to read the original value.
2021-06-28 08:01:27,292 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2021-06-28 08:01:38,186 WARNING  tuned.plugins.plugin_vm: Option 'transparent_hugepages' is not supported on current hardware.
2021-06-28 08:01:38,187 INFO     tuned.plugins.plugin_systemd: setting 'CPUAffinity' to '0 1 40 41' in the '/etc/systemd/system.conf'
2021-06-28 08:01:54,338 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/cpu-partitioning/script.sh' with arguments '['start']'
2021-06-28 08:01:56,369 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2

Note You need to log in before you can comment on or make changes to this bug.