Description of problem: During the Performance Scale test, we noticed a notable decrease in performance when conducting the regular SRIOV test. The issue persists when conducting a baseline performance test (P-P) using a 100G Mellanox Card. The results are very poor when compare to the previous baseline test results with RHEL9.0. Version-Release number of selected component (if applicable): OpenStack Puddle Version: RHOS-17.1-RHEL-9-20230511.n.1 Red Hat OpenStack Platform release 17.1.0 Beta (Wallaby) Red Hat Enterprise Linux release 9.2 Beta (Plow) dpdk-22.11-3.el9_2.x86_64 dpdk-tools-22.11-3.el9_2.x86_64 Trex v2.87 How reproducible: 100% reproduced in NFV Perf lab Steps to Reproduce: Topology -------- Traffic direction -----> Trex Port 1----- Switch Port 1 ---- | Switch Port2 -----DUT Port1 | dpdk-testpmd Port1 (io mode forwarding) Trex Port 2----- Switch Port 2 ---- | Switch Port3 -----DUT Port2 | dpdk-testpmd Port2 (io mode forwarding) Traffic direction -----> Scenarios: P-P Trafficgen(Trex) Profile: ------------------------ Frame size: 64 Number flows: 1024 Flow modification: src-ip Direction: Bi-directional Loss measurement: 0.002 Final validation time: 120sec Protocol: UDP traffic rate: 100% Testmode: binary-search Binary-Search trial summary --------------------------- # | Start Time Stop Time | Duration | Rate | Type | Result 1 | 2023-05-26 01:14:17.904171 - 2023-05-26 01:14:50.000901 | 32.10 seconds | 100.000000% | search | retry-to-fail 2 | 2023-05-26 01:14:51.924724 - 2023-05-26 01:15:24.023045 | 32.10 seconds | 100.000000% | search | retry-to-fail 3 | 2023-05-26 01:15:25.924985 - 2023-05-26 01:15:55.971689 | 30.05 seconds | 50.000000% | search | fail 4 | 2023-05-26 01:15:57.881566 - 2023-05-26 01:16:27.929149 | 30.05 seconds | 25.000000% | search | fail 5 | 2023-05-26 01:16:30.210199 - 2023-05-26 01:17:00.258472 | 30.05 seconds | 12.500000% | search | fail 6 | 2023-05-26 01:17:02.163033 - 2023-05-26 01:17:32.203322 | 30.04 seconds | 6.250000% | search | fail 7 | 2023-05-26 01:17:34.113451 - 2023-05-26 01:18:04.158768 | 30.05 seconds | 3.125000% | search | fail 8 | 2023-05-26 01:18:07.218696 - 2023-05-26 01:18:37.262415 | 30.04 seconds | 1.562500% | search | pass 9 | 2023-05-26 01:18:39.184309 - 2023-05-26 01:19:09.229085 | 30.04 seconds | 2.343750% | search | pass 10 | 2023-05-26 01:19:11.140802 - 2023-05-26 01:19:41.188491 | 30.05 seconds | 2.734375% | search | pass 11 | 2023-05-26 01:19:43.099415 - 2023-05-26 01:20:13.144775 | 30.05 seconds | 2.929688% | search | pass 12 | 2023-05-26 01:20:15.063560 - 2023-05-26 01:20:45.102557 | 30.04 seconds | 3.027344% | search | pass 13 | 2023-05-26 01:20:46.999714 - 2023-05-26 01:21:17.047702 | 30.05 seconds | 3.076172% | search | pass 14 | 2023-05-26 01:21:18.976979 - 2023-05-26 01:21:49.025012 | 30.05 seconds | 3.100586% | search | fail 15 | 2023-05-26 01:21:50.928224 - 2023-05-26 01:22:20.975406 | 30.05 seconds | 3.088379% | search | fail 16 | 2023-05-26 01:22:22.900850 - 2023-05-26 01:22:52.945056 | 30.04 seconds | 3.082275% | search | pass 17 | 2023-05-26 01:22:54.863569 - 2023-05-26 01:24:54.903265 | 120.04 seconds | 3.085327% | validation | fail 18 | 2023-05-26 01:24:56.830066 - 2023-05-26 01:25:26.875271 | 30.05 seconds | 3.082242% | search | fail 19 | 2023-05-26 01:25:28.784322 - 2023-05-26 01:25:58.827230 | 30.04 seconds | 3.079160% | search | fail 20 | 2023-05-26 01:26:00.729920 - 2023-05-26 01:26:30.770783 | 30.04 seconds | 3.076080% | search | pass 21 | 2023-05-26 01:26:32.694113 - 2023-05-26 01:28:32.744747 | 120.05 seconds | 3.077620% | validation | fail 22 | 2023-05-26 01:28:34.658382 - 2023-05-26 01:29:04.702197 | 30.04 seconds | 3.074542% | search | fail 23 | 2023-05-26 01:29:06.620420 - 2023-05-26 01:29:36.664711 | 30.04 seconds | 3.071468% | search | pass 24 | 2023-05-26 01:29:38.941375 - 2023-05-26 01:31:38.985711 | 120.04 seconds | 3.073005% | validation | fail 25 | 2023-05-26 01:31:40.956366 - 2023-05-26 01:32:10.999301 | 30.04 seconds | 3.069932% | search | pass 26 | 2023-05-26 01:32:12.933452 - 2023-05-26 01:34:12.979944 | 120.05 seconds | 3.071469% | validation | fail 27 | 2023-05-26 01:34:14.900116 - 2023-05-26 01:34:44.942819 | 30.04 seconds | 3.068397% | search | fail 28 | 2023-05-26 01:34:47.144820 - 2023-05-26 01:35:17.194986 | 30.05 seconds | 3.065329% | search | pass 29 | 2023-05-26 01:35:19.097244 - 2023-05-26 01:37:19.137741 | 120.04 seconds | 3.066863% | validation | fail 30 | 2023-05-26 01:37:21.035989 - 2023-05-26 01:37:51.080092 | 30.04 seconds | 3.063796% | search | fail 31 | 2023-05-26 01:37:52.972417 - 2023-05-26 01:38:23.014293 | 30.04 seconds | 3.067632% | search | fail 32 | 2023-05-26 01:38:25.188356 - 2023-05-26 01:38:55.236596 | 30.05 seconds | 3.071902% | search | pass 33 | 2023-05-26 01:38:57.154540 - 2023-05-26 01:40:57.207749 | 120.05 seconds | 3.069767% | validation | fail 34 | 2023-05-26 01:40:59.127289 - 2023-05-26 01:41:29.172890 | 30.05 seconds | 3.066697% | search | fail 35 | 2023-05-26 01:41:31.092206 - 2023-05-26 01:42:01.143754 | 30.05 seconds | 3.071435% | search | fail 36 | 2023-05-26 01:42:03.917144 - 2023-05-26 01:42:33.961346 | 30.04 seconds | 3.049389% | search | pass 37 | 2023-05-26 01:42:35.880161 - 2023-05-26 01:43:05.932150 | 30.05 seconds | 3.060412% | search | fail 38 | 2023-05-26 01:43:07.835728 - 2023-05-26 01:43:37.878255 | 30.04 seconds | 3.054900% | search | pass 39 | 2023-05-26 01:43:39.784717 - 2023-05-26 01:45:39.822919 | 120.04 seconds | 3.057656% | validation | fail 40 | 2023-05-26 01:45:42.112958 - 2023-05-26 01:46:12.157166 | 30.04 seconds | 3.054598% | search | pass 41 | 2023-05-26 01:46:14.091563 - 2023-05-26 01:48:14.135894 | 120.04 seconds | 3.056127% | validation | fail 42 | 2023-05-26 01:48:16.056925 - 2023-05-26 01:48:46.101957 | 30.05 seconds | 3.053071% | search | pass 43 | 2023-05-26 01:48:48.002060 - 2023-05-26 01:50:48.053257 | 120.05 seconds | 3.054599% | validation | fail 44 | 2023-05-26 01:50:50.251535 - 2023-05-26 01:51:20.306318 | 30.05 seconds | 3.051545% | search | pass 45 | 2023-05-26 01:51:22.214993 - 2023-05-26 01:53:22.258151 | 120.04 seconds | 3.053072% | validation | fail 46 | 2023-05-26 01:53:24.157144 - 2023-05-26 01:53:54.203469 | 30.05 seconds | 3.050019% | search | pass 47 | 2023-05-26 01:53:56.116424 - 2023-05-26 01:55:56.158133 | 120.04 seconds | 3.051545% | validation | fail 48 | 2023-05-26 01:55:58.062538 - 2023-05-26 01:56:28.105136 | 30.04 seconds | 3.048494% | search | fail 49 | 2023-05-26 01:56:30.007373 - 2023-05-26 01:57:00.050981 | 30.04 seconds | 3.045445% | search | pass 50 | 2023-05-26 01:57:02.221598 - 2023-05-26 01:59:02.269238 | 120.05 seconds | 3.046970% | validation | fail 51 | 2023-05-26 01:59:04.444058 - 2023-05-26 01:59:34.483551 | 30.04 seconds | 3.043923% | search | fail 52 | 2023-05-26 01:59:36.389301 - 2023-05-26 02:00:06.425981 | 30.04 seconds | 3.047734% | search | fail 53 | 2023-05-26 02:00:08.337400 - 2023-05-26 02:00:38.387417 | 30.05 seconds | 3.044686% | search | fail 54 | 2023-05-26 02:00:40.314492 - 2023-05-26 02:01:10.360236 | 30.05 seconds | 3.049642% | search | pass 55 | 2023-05-26 02:01:12.277590 - 2023-05-26 02:03:12.329750 | 120.05 seconds | 3.047164% | validation | fail 56 | 2023-05-26 02:03:14.507297 - 2023-05-26 02:03:44.556057 | 30.05 seconds | 3.044117% | search | fail 57 | 2023-05-26 02:03:46.470154 - 2023-05-26 02:04:16.518852 | 30.05 seconds | 3.049358% | search | fail 58 | 2023-05-26 02:04:18.428681 - 2023-05-26 02:04:48.475391 | 30.05 seconds | 3.046308% | search | fail 59 | 2023-05-26 02:04:50.999645 - 2023-05-26 02:05:21.045928 | 30.05 seconds | 3.043262% | search | fail 60 | 2023-05-26 02:05:22.953340 - 2023-05-26 02:05:52.996749 | 30.04 seconds | 3.035303% | search | fail 61 | 2023-05-26 02:05:55.159319 - 2023-05-26 02:06:25.208293 | 30.05 seconds | 3.031323% | search | fail 62 | 2023-05-26 02:06:27.107001 - 2023-05-26 02:06:57.156487 | 30.05 seconds | 3.028292% | search | fail 63 | 2023-05-26 02:06:59.330160 - 2023-05-26 02:07:29.375876 | 30.05 seconds | 3.025264% | search | fail 64 | 2023-05-26 02:07:31.286133 - 2023-05-26 02:08:01.323442 | 30.04 seconds | 2.977476% | search | pass 65 | 2023-05-26 02:08:03.232015 - 2023-05-26 02:08:33.280790 | 30.05 seconds | 3.001370% | search | fail 66 | 2023-05-26 02:08:35.499324 - 2023-05-26 02:09:05.545386 | 30.05 seconds | 2.989423% | search | pass 67 | 2023-05-26 02:09:07.478621 - 2023-05-26 02:09:37.525786 | 30.05 seconds | 2.995396% | search | fail 68 | 2023-05-26 02:09:39.503905 - 2023-05-26 02:10:09.552798 | 30.05 seconds | 2.992401% | search | fail 69 | 2023-05-26 02:10:11.466830 - 2023-05-26 02:10:41.513264 | 30.05 seconds | 2.989408% | search | pass 70 | 2023-05-26 02:10:43.421530 - 2023-05-26 02:12:43.475024 | 120.05 seconds | 2.990905% | validation | fail 71 | 2023-05-26 02:12:45.647682 - 2023-05-26 02:13:15.693035 | 30.05 seconds | 2.987914% | search | fail 72 | 2023-05-26 02:13:17.977631 - 2023-05-26 02:13:48.025667 | 30.05 seconds | 2.984926% | search | pass 73 | 2023-05-26 02:13:49.949616 - 2023-05-26 02:15:49.989471 | 120.04 seconds | 2.986420% | validation | fail 74 | 2023-05-26 02:15:51.994109 - 2023-05-26 02:16:22.046566 | 30.05 seconds | 2.983433% | search | pass 75 | 2023-05-26 02:16:24.283949 - 2023-05-26 02:18:24.328587 | 120.04 seconds | 2.984926% | validation | fail 76 | 2023-05-26 02:18:26.314167 - 2023-05-26 02:18:56.362824 | 30.05 seconds | 2.981942% | search | pass 77 | 2023-05-26 02:18:58.267946 - 2023-05-26 02:20:58.314502 | 120.05 seconds | 2.983434% | validation | fail 78 | 2023-05-26 02:21:00.423368 - 2023-05-26 02:21:30.462844 | 30.04 seconds | 2.980451% | search | pass 79 | 2023-05-26 02:21:32.699746 - 2023-05-26 02:23:32.750796 | 120.05 seconds | 2.981942% | validation | fail 80 | 2023-05-26 02:23:34.975904 - 2023-05-26 02:24:05.025177 | 30.05 seconds | 2.978960% | search | pass 81 | 2023-05-26 02:24:07.135843 - 2023-05-26 02:26:07.187064 | 120.05 seconds | 2.980451% | validation | fail 82 | 2023-05-26 02:26:09.099845 - 2023-05-26 02:26:39.148634 | 30.05 seconds | 2.977471% | search | pass 83 | 2023-05-26 02:26:41.086096 - 2023-05-26 02:28:41.130129 | 120.04 seconds | 2.978961% | validation | pass Binary-Search result: -------------------- run-id: 0e10319d-54d7-45bb-ba6e-2998bc8a6487 tags: DUT=perf202.perf.lab.eng.bos.redhat.com OSP=RHOS-17.1-RHEL-9-20230511.n.1 Tgen=perf122.perf.lab.eng.bos.redhat.com Trex=v2.87 dpdk=22.11.1 kernel=5.14.0-283.el9.x86_64 topo=P-P-Mlxc6-100G tuned=cpu-partitioning benchmark: trafficgen <..trim..> iteration-id: 4A2B1E46-FB6D-11ED-8B08-2C78D998FD6A unique params: primary-period name: measurement samples: sample-id: 4A844E58-FB6D-11ED-8AAD-2C78D998FD6A primary period-id: 54EFF89C-FB6D-11ED-9911-2C78D998FD6A period range: begin: 1685068001086 end: 1685068121130 result: (trafficgen::rx-pps) samples: 8862619.927140 mean: 8862619.927140 min: 8862619.927140 max: 8862619.927140 stddev: NaN stddevpct: NaN Compute Node diagnosis details (DUT) during the test ---------------------------------------------------- DMIDECODE BIOS: Vend: Dell Inc. Vers: 1.6.5 Date: 04/15/2022 BIOS Rev: 1.6 FW Rev: System: Mfr: Dell Inc. Prod: PowerEdge R650 Vers: Not Specified CPU: 2 of 2 CPU sockets populated, 36 cores/36 threads per CPU 72 total cores, 72 total threads Mfr: Intel Fam: Xeon Freq: 2100 MHz Vers: Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz Memory: Total: 262144 MiB (256 GiB) DIMMs: 16 of 96 populated MaxCapacity: 12582912 MiB (12288 GiB / 12.00 TiB) MEMORY Stats graphed as percent of MemTotal: MemUsed _________________________________________......... 82.9% Buffers .................................................. 0.0% Cached .................................................. 0.5% HugePages ________________________________________.......... 79.7% Dirty .................................................. 0.0% RAM: 250.9 GiB total ram 208.1 GiB (83%) used 206.8 GiB (82%) used excluding Buffers/Cached 0 GiB (0%) dirty HugePages: 200 GiB pre-allocated to HugePages (80% of total ram) 8 GiB of HugePages (4%) in-use by applications LowMem/Slab/PageTables/Shmem: 0.78 GiB (0%) of total ram used for Slab 0.01 GiB (0%) of total ram used for PageTables 0.01 GiB (0%) of total ram used for Shmem Swap: No system swap space configured OS Hostname: perf202.xxxxxx.redhat.com Distro: [redhat-release] Red Hat Enterprise Linux release 9.2 Beta (Plow) [rhosp-release] Red Hat OpenStack Platform release 17.1.0 Beta (Wallaby) [os-release] Red Hat Enterprise Linux 9.2 Beta (Plow) 9.2 (Plow) RHN: (missing) RHSM: hostname = subscription.rhsm.redhat.com proxy_hostname = YUM: 3 enabled plugins: debuginfo-install, product-id, subscription-manager Runlevel: N 3 (default multi-user) SELinux: enforcing (default enforcing) Arch: mach=x86_64 cpu=x86_64 platform=x86_64 Kernel: Booted kernel: 5.14.0-283.el9.x86_64 GRUB default: Build version: Linux version 5.14.0-283.el9.x86_64 (mockbuild.eng.bos.redhat.com) (gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4), GNU ld version 2.35.2-37.el9) #1 SMP PREEMPT_DYNAMIC Thu Feb 23 19:04:04 EST 2023 Booted kernel cmdline: BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-283.el9.x86_64 root=LABEL=img-rootfs ro console=ttyS0 console=ttyS0,115200n81 no_timer_check crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M default_hugepagesz=1GB hugepagesz=1G hugepages=200 iommu=pt intel_iommu=on isolcpus=10-71 console=tty0 console=ttyS0,115200 no_timer_check nofb vga=normal console=tty0 console=ttyS0,115200 audit=1 nousb boot=LABEL=mkfs_boot skew_tick=1 nohz=on nohz_full=10-71 rcu_nocbs=10-71 tuned.non_isolcpus=000003ff intel_pstate=disable nosoftlockup GRUB default kernel cmdline: # tuned-adm active Current active profile: cpu-partitioning # grep Cpus_allowed /proc/self/status Cpus_allowed: 00,00000000,000003ff Cpus_allowed_list: 0-9 # for i in $(pgrep -f nova-compute); do grep -i cpus_allowed_list /proc/$i/task/*/status ; done /proc/4264/task/4264/status:Cpus_allowed_list: 0-9 /proc/4264/task/4433/status:Cpus_allowed_list: 0-9 /proc/4264/task/4434/status:Cpus_allowed_list: 0-9 /proc/4264/task/4435/status:Cpus_allowed_list: 0-9 /proc/4264/task/4436/status:Cpus_allowed_list: 0-9 /proc/4264/task/4437/status:Cpus_allowed_list: 0-9 /proc/4264/task/4438/status:Cpus_allowed_list: 0-9 /proc/4264/task/4439/status:Cpus_allowed_list: 0-9 /proc/4264/task/4440/status:Cpus_allowed_list: 0-9 /proc/4264/task/4441/status:Cpus_allowed_list: 0-9 /proc/4264/task/4442/status:Cpus_allowed_list: 0-9 /proc/4264/task/4443/status:Cpus_allowed_list: 0-9 /proc/4264/task/4444/status:Cpus_allowed_list: 0-9 /proc/4264/task/4445/status:Cpus_allowed_list: 0-9 /proc/4264/task/4446/status:Cpus_allowed_list: 0-9 /proc/4264/task/4447/status:Cpus_allowed_list: 0-9 /proc/4264/task/4448/status:Cpus_allowed_list: 0-9 /proc/4264/task/4449/status:Cpus_allowed_list: 0-9 /proc/4264/task/4450/status:Cpus_allowed_list: 0-9 /proc/4264/task/4451/status:Cpus_allowed_list: 0-9 /proc/4264/task/4452/status:Cpus_allowed_list: 0-9 /proc/4264/task/4453/status:Cpus_allowed_list: 0-9 # lspci -v -s 0000:98:00.0 98:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] Subsystem: Mellanox Technologies ConnectX-5 Ex EN network interface card, 100GbE dual-port QSFP28, PCIe4.0 x16, tall bracket; MCX516A-CDAT Flags: bus master, fast devsel, latency 0, IRQ 18, NUMA node 1, IOMMU group 168 Memory at d4000000 (64-bit, prefetchable) [size=32M] Expansion ROM at d1000000 [disabled] [size=1M] Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Count=64 Masked- Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [40] Power Management version 3 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [180] Single Root I/O Virtualization (SR-IOV) Capabilities: [1c0] Secondary PCI Express Capabilities: [230] Access Control Services Capabilities: [320] Lane Margining at the Receiver <?> Capabilities: [370] Physical Layer 16.0 GT/s <?> Capabilities: [420] Data Link Feature <?> Kernel driver in use: mlx5_core Kernel modules: mlx5_core # ethtool -i ens2f0np0 driver: mlx5_core version: 5.14.0-283.el9.x86_64 firmware-version: 16.31.1014 (MT_0000000013) expansion-rom-version: bus-info: 0000:98:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes ETHTOOL Interface Status: ens2f0np0 0000:98:00.0 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f0v0 0000:98:00.2 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f0v1 0000:98:00.3 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f0v2 0000:98:00.4 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f0v3 0000:98:00.5 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f1np1 0000:98:00.1 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f1v0 0000:98:01.2 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f1v1 0000:98:01.3 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f1v2 0000:98:01.4 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) ens2f1v3 0000:98:01.5 link=up 100000Mb/s full (autoneg=Y) rx ring 1024/8192 drv mlx5_core v5.14.0-283.el9.x86_64 / fw 16.31.1014 (MT_0000000013) Interface Errors: ens2f0np0 rx_steer_missed_packets: 2 tx_pause_ctrl_phy: 3344819 rx_discards_phy: 15143854953 rx_prio0_discards: 15143854953 tx_global_pause: 3344819 tx_global_pause_duration: 440127287 - - - - - - - - - - - - - - - - - - - ens2f1np1 rx_steer_missed_packets: 1 tx_pause_ctrl_phy: 3341836 rx_discards_phy: 15143860795 rx_prio0_discards: 15143860795 tx_global_pause: 3341836 tx_global_pause_duration: 439885716 NETDEV Interface RxMiBytes RxPackets RxErrs RxDrop RxFifo RxComp RxFrame RxMultCast ========= ========= ========= ====== ====== ====== ====== ======= ========== ens2f0np0 1 3 k 0 2325 (29%) 0 0 0 2902 (36%) ens2f0v0 3 12 k 0 0 0 0 0 12353 (50%) ens2f0v1 3 12 k 0 0 0 0 0 12352 (50%) ens2f0v2 3 12 k 0 0 0 0 0 12352 (50%) ens2f0v3 3 12 k 0 0 0 0 0 12352 (50%) ens2f1np1 6 36 k 0 2324 (3%) 0 0 0 35580 (48%) ens2f1v0 3 11 k 0 0 0 0 0 11053 (50%) ens2f1v1 3 11 k 0 0 0 0 0 11053 (50%) ens2f1v2 3 11 k 0 0 0 0 0 11053 (50%) ens2f1v3 3 11 k 0 0 0 0 0 11053 (50%) - - - - - - - - - - - - - - - - - Interface TxMiBytes TxPackets TxErrs TxDrop TxFifo TxComp TxColls TxCarrier ========= ========= ========= ====== ====== ====== ====== ======= ========== ens2f0np0 0 0 k 0 0 0 0 0 0 ens2f0v0 1 9 k 0 0 0 0 0 0 ens2f0v1 1 9 k 0 0 0 0 0 0 ens2f0v2 1 9 k 0 0 0 0 0 0 ens2f0v3 1 9 k 0 0 0 0 0 0 ens2f1np1 0 0 k 0 0 0 0 0 0 ens2f1v0 1 9 k 0 0 0 0 0 0 ens2f1v1 1 9 k 0 0 0 0 0 0 ens2f1v2 1 9 k 0 0 0 0 0 0 ens2f1v3 1 9 k 0 0 0 0 0 0 PCI INTERRUPTS 298: .▊.▊.▊.▊.▊.............................................................. IR-PCI-MSI 79691776-edge mlx5_comp0@pci:0000:98:00.0 299: ...▊.................................................................... IR-PCI-MSI 79691777-edge mlx5_comp1@pci:0000:98:00.0 300: .....▊.................................................................. IR-PCI-MSI 79691778-edge mlx5_comp2@pci:0000:98:00.0 301: .....▊.▊................................................................ IR-PCI-MSI 79691779-edge mlx5_comp3@pci:0000:98:00.0 302: .▊.......▊.............................................................. IR-PCI-MSI 79691780-edge mlx5_comp4@pci:0000:98:00.0 303: .......▊...▊............................................................ IR-PCI-MSI 79691781-edge mlx5_comp5@pci:0000:98:00.0 304: .......▊.....▊.......................................................... IR-PCI-MSI 79691782-edge mlx5_comp6@pci:0000:98:00.0 305: ...▊...........▊........................................................ IR-PCI-MSI 79691783-edge mlx5_comp7@pci:0000:98:00.0 306: .....▊...........▊...................................................... IR-PCI-MSI 79691784-edge mlx5_comp8@pci:0000:98:00.0 307: .........▊.........▊.................................................... IR-PCI-MSI 79691785-edge mlx5_comp9@pci:0000:98:00.0 308: .▊...................▊.................................................. IR-PCI-MSI 79691786-edge mlx5_comp10@pci:0000:98:00.0 309: .....▊.................▊................................................ IR-PCI-MSI 79691787-edge mlx5_comp11@pci:0000:98:00.0 310: .▊.......................▊.............................................. IR-PCI-MSI 79691788-edge mlx5_comp12@pci:0000:98:00.0 311: ...▊.......................▊............................................ IR-PCI-MSI 79691789-edge mlx5_comp13@pci:0000:98:00.0 312: .......▊.....................▊.......................................... IR-PCI-MSI 79691790-edge mlx5_comp14@pci:0000:98:00.0 313: ...▊...........................▊........................................ IR-PCI-MSI 79691791-edge mlx5_comp15@pci:0000:98:00.0 314: ...▊.............................▊...................................... IR-PCI-MSI 79691792-edge mlx5_comp16@pci:0000:98:00.0 315: .▊.▊.....▊.........................▊.................................... IR-PCI-MSI 79691793-edge mlx5_comp17@pci:0000:98:00.0 316: .▊...................................▊.................................. IR-PCI-MSI 79691794-edge mlx5_comp18@pci:0000:98:00.0 317: .....▊.................................▊................................ IR-PCI-MSI 79691795-edge mlx5_comp19@pci:0000:98:00.0 318: ...▊...▊.▊...............................▊.............................. IR-PCI-MSI 79691796-edge mlx5_comp20@pci:0000:98:00.0 319: .▊.........................................▊............................ IR-PCI-MSI 79691797-edge mlx5_comp21@pci:0000:98:00.0 320: .......▊.....................................▊.......................... IR-PCI-MSI 79691798-edge mlx5_comp22@pci:0000:98:00.0 321: .▊.............................................▊........................ IR-PCI-MSI 79691799-edge mlx5_comp23@pci:0000:98:00.0 322: ...▊.▊.▊.........................................▊...................... IR-PCI-MSI 79691800-edge mlx5_comp24@pci:0000:98:00.0 323: .........▊.........................................▊.................... IR-PCI-MSI 79691801-edge mlx5_comp25@pci:0000:98:00.0 324: .▊.▊.▊...▊...........................................▊.................. IR-PCI-MSI 79691802-edge mlx5_comp26@pci:0000:98:00.0 325: ...▊...................................................▊................ IR-PCI-MSI 79691803-edge mlx5_comp27@pci:0000:98:00.0 326: .....▊...................................................▊.............. IR-PCI-MSI 79691804-edge mlx5_comp28@pci:0000:98:00.0 327: .........▊.................................................▊............ IR-PCI-MSI 79691805-edge mlx5_comp29@pci:0000:98:00.0 328: .▊.▊.▊.......................................................▊.......... IR-PCI-MSI 79691806-edge mlx5_comp30@pci:0000:98:00.0 329: .▊.▊...▊.▊.....................................................▊........ IR-PCI-MSI 79691807-edge mlx5_comp31@pci:0000:98:00.0 330: ...▊.............................................................▊...... IR-PCI-MSI 79691808-edge mlx5_comp32@pci:0000:98:00.0 331: .........▊.........................................................▊.... IR-PCI-MSI 79691809-edge mlx5_comp33@pci:0000:98:00.0 332: .........▊...........................................................▊.. IR-PCI-MSI 79691810-edge mlx5_comp34@pci:0000:98:00.0 333: .....▊.................................................................▊ IR-PCI-MSI 79691811-edge mlx5_comp35@pci:0000:98:00.0 444: .▊.▊.▊.▊.▊.............................................................. IR-PCI-MSI 79693824-edge mlx5_comp0@pci:0000:98:00.1 445: ...▊.▊.................................................................. IR-PCI-MSI 79693825-edge mlx5_comp1@pci:0000:98:00.1 446: ...▊.▊.▊................................................................ IR-PCI-MSI 79693826-edge mlx5_comp2@pci:0000:98:00.1 447: .▊.....▊................................................................ IR-PCI-MSI 79693827-edge mlx5_comp3@pci:0000:98:00.1 448: .▊.......▊.............................................................. IR-PCI-MSI 79693828-edge mlx5_comp4@pci:0000:98:00.1 449: .......▊...▊............................................................ IR-PCI-MSI 79693829-edge mlx5_comp5@pci:0000:98:00.1 450: ...▊.....▊...▊.......................................................... IR-PCI-MSI 79693830-edge mlx5_comp6@pci:0000:98:00.1 451: .▊.............▊........................................................ IR-PCI-MSI 79693831-edge mlx5_comp7@pci:0000:98:00.1 452: .........▊.......▊...................................................... IR-PCI-MSI 79693832-edge mlx5_comp8@pci:0000:98:00.1 453: ...▊...............▊.................................................... IR-PCI-MSI 79693833-edge mlx5_comp9@pci:0000:98:00.1 454: .....▊...............▊.................................................. IR-PCI-MSI 79693834-edge mlx5_comp10@pci:0000:98:00.1 455: .........▊.............▊................................................ IR-PCI-MSI 79693835-edge mlx5_comp11@pci:0000:98:00.1 456: .▊.▊...▊.................▊.............................................. IR-PCI-MSI 79693836-edge mlx5_comp12@pci:0000:98:00.1 457: .....▊.....................▊............................................ IR-PCI-MSI 79693837-edge mlx5_comp13@pci:0000:98:00.1 458: .▊.▊.▊...▊...................▊.......................................... IR-PCI-MSI 79693838-edge mlx5_comp14@pci:0000:98:00.1 459: .▊.............................▊........................................ IR-PCI-MSI 79693839-edge mlx5_comp15@pci:0000:98:00.1 460: .......▊.........................▊...................................... IR-PCI-MSI 79693840-edge mlx5_comp16@pci:0000:98:00.1 461: .▊.▊.▊.▊.▊.........................▊.................................... IR-PCI-MSI 79693841-edge mlx5_comp17@pci:0000:98:00.1 462: .....▊...............................▊.................................. IR-PCI-MSI 79693842-edge mlx5_comp18@pci:0000:98:00.1 463: .........▊.............................▊................................ IR-PCI-MSI 79693843-edge mlx5_comp19@pci:0000:98:00.1 464: .▊.▊.▊.▊.▊...............................▊.............................. IR-PCI-MSI 79693844-edge mlx5_comp20@pci:0000:98:00.1 465: .....▊.....................................▊............................ IR-PCI-MSI 79693845-edge mlx5_comp21@pci:0000:98:00.1 466: .......▊.....................................▊.......................... IR-PCI-MSI 79693846-edge mlx5_comp22@pci:0000:98:00.1 467: .▊.............................................▊........................ IR-PCI-MSI 79693847-edge mlx5_comp23@pci:0000:98:00.1 468: .▊.▊.▊.▊.▊.......................................▊...................... IR-PCI-MSI 79693848-edge mlx5_comp24@pci:0000:98:00.1 469: ...▊...▊...........................................▊.................... IR-PCI-MSI 79693849-edge mlx5_comp25@pci:0000:98:00.1 470: .▊.▊.▊.▊.▊...........................................▊.................. IR-PCI-MSI 79693850-edge mlx5_comp26@pci:0000:98:00.1 471: .......▊...............................................▊................ IR-PCI-MSI 79693851-edge mlx5_comp27@pci:0000:98:00.1 472: ...▊.....................................................▊.............. IR-PCI-MSI 79693852-edge mlx5_comp28@pci:0000:98:00.1 473: .......▊...................................................▊............ IR-PCI-MSI 79693853-edge mlx5_comp29@pci:0000:98:00.1 474: .▊.▊.▊.▊.▊...................................................▊.......... IR-PCI-MSI 79693854-edge mlx5_comp30@pci:0000:98:00.1 475: .▊.▊.▊.▊.▊.....................................................▊........ IR-PCI-MSI 79693855-edge mlx5_comp31@pci:0000:98:00.1 476: .▊.▊.▊.▊.▊.......................................................▊...... IR-PCI-MSI 79693856-edge mlx5_comp32@pci:0000:98:00.1 477: .▊.▊.▊.▊.▊.........................................................▊.... IR-PCI-MSI 79693857-edge mlx5_comp33@pci:0000:98:00.1 478: .........▊...........................................................▊.. IR-PCI-MSI 79693858-edge mlx5_comp34@pci:0000:98:00.1 479: .▊.▊.▊.▊.▊.............................................................▊ IR-PCI-MSI 79693859-edge mlx5_comp35@pci:0000:98:00.1 PS CHECK Total number of threads/processes: 2145 / 988 Top users of CPU & MEM: USER %CPU %MEM RSS root 203.8% 0.1% 1.16 GiB 42436 0.6% 0.0% 0.17 GiB openvsw+ 0.1% 0.0% 0.19 GiB Uninteruptible sleep threads/processes (0/0): [None] Defunct zombie threads/processes (0/0): [None] Top CPU-using processes: USER PID %CPU %MEM VSZ-MiB RSS-MiB TTY STAT START TIME COMMAND root 1010087 199 0.1 131545 353 ? SLl 01:13 50:18 /usr/bin/dpdk-testpmd --lcores 0@0,1@11,2@13 root 3504 1.2 0.0 1474 27 ? Sl May25 14:56 /usr/sbin/virtnodedevd --config /etc/libvirt/virtnodedevd.conf root 1010367 1.1 0.0 186 28 ? Sl 01:14 0:17 /usr/bin/python3 /usr/local/bin/roadblocker.py --role=follower root 3527 1.0 0.0 1333 27 ? Sl May25 12:07 /usr/sbin/virtqemud --config /etc/libvirt/virtqemud.conf root 1009472 0.9 0.0 14 12 ? S 01:12 0:15 pidstat -p ALL 42436 4264 0.6 0.0 1847 169 ? Sl May25 7:58 /usr/bin/python3 /usr/bin/nova-compute root 3022 0.2 0.0 366 40 ? Ssl May25 3:13 /usr/bin/python3 -Es /usr/sbin/tuned root 1009490 0.2 0.0 147 1 ? Sl 01:12 0:04 /usr/bin/turbostat -d -d openvsw+ 2096 0.1 0.0 1214 191 ? S<Lsl May25 2:09 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer rpc 1619 0.0 0.0 13 6 ? Ss May25 0:00 /usr/bin/rpcbind -w -f Top MEM-using processes: USER PID %CPU %MEM VSZ-MiB RSS-MiB TTY STAT START TIME COMMAND root 1010087 199 0.1 131545 353 ? SLl 01:13 50:18 /usr/bin/dpdk-testpmd --lcores 0@0,1@11,2@13 openvsw+ 2096 0.1 0.0 1214 191 ? S<Lsl May25 2:09 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer 42436 4264 0.6 0.0 1847 169 ? Sl May25 7:58 /usr/bin/python3 /usr/bin/nova-compute 42435 3625 0.0 0.0 128 114 ? S May25 0:14 neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/networking-ovn-metadata-agent 42435 3623 0.0 0.0 123 114 ? S May25 0:41 neutron-sriov-nic-agent (/usr/bin/python3 /usr/bin/neutron-sriov-nic-agent root 1282 0.0 0.0 154 103 ? Ss May25 0:19 /usr/lib/systemd/systemd-journald 42435 4141 0.0 0.0 126 101 ? S May25 0:00 neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/networking-ovn-metadata-agent 42435 4140 0.0 0.0 126 101 ? S May25 0:00 neutron-ovn-metadata-agent (/usr/bin/python3 /usr/bin/networking-ovn-metadata-agent root 4177 0.0 0.0 149 70 ? Sl May25 0:00 /usr/bin/python3 /bin/privsep-helper --config-file root 1735 0.0 0.0 623 62 ? Ssl May25 0:15 /usr/sbin/rsyslogd -n Top thread-spawning processes: # USER PID %CPU %MEM VSZ-MiB RSS-MiB TTY STAT START TIME COMMAND 22 42436 4264 0.6 0.0 1847 169 ? - May25 7:58 /usr/bin/python3 /usr/bin/nova-compute 19 root 3504 1.2 0.0 1474 27 ? - May25 14:56 /usr/sbin/virtnodedevd --config /etc/libvirt/virtnodedevd.conf 18 root 4315 0.0 0.0 1329 17 ? - May25 0:00 /usr/sbin/virtproxyd --config /etc/libvirt/virtproxyd.conf 18 root 3552 0.0 0.0 1330 17 ? - May25 0:00 /usr/sbin/virtstoraged --config /etc/libvirt/virtstoraged.conf 18 root 3527 1.0 0.0 1333 27 ? - May25 12:07 /usr/sbin/virtqemud --config /etc/libvirt/virtqemud.conf 18 root 3461 0.0 0.0 1329 17 ? - May25 0:00 /usr/sbin/virtsecretd --config /etc/libvirt/virtsecretd.conf 17 openvsw+ 2096 0.1 0.0 1214 191 ? - May25 2:09 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer 12 polkitd 3178 0.0 0.0 2913 23 ? - May25 0:00 /usr/lib/polkit-1/polkitd --no-debug 6 root 2986 0.0 0.0 54 6 ? - May25 0:00 /usr/sbin/gssproxy -D 6 root 1010087 199 0.1 131545 353 ? - 01:13 50:16 /usr/bin/dpdk-testpmd --lcores 0@0,1@11,2@13 Note: In the same host, the intel XXV710 card binary-search results reached to ~38Mpps rate of packets. Expected results: We are expecting similar performance results while comparing it with RHEL9.0. The results documents will be shared in the internal comments Additional info: Bench-trafficgen binary search log reports will be shared in the internal comments.
The issue has persisted after I referred resolution mentioned in the KCS: https://access.redhat.com/solutions/7007609 ~# grubby --update-kernel=ALL --remove-args=nomodeset ~# systemctl reboot # cat /proc/cmdline BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-283.el9.x86_64 root=LABEL=img-rootfs ro console=ttyS0 console=ttyS0,115200n81 no_timer_check crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M default_hugepagesz=1GB hugepagesz=1G hugepages=200 iommu=pt intel_iommu=on isolcpus=10-71 console=tty0 console=ttyS0,115200 no_timer_check nofb vga=normal console=tty0 console=ttyS0,115200 audit=1 nousb boot=LABEL=mkfs_boot skew_tick=1 nohz=on nohz_full=10-71 rcu_nocbs=10-71 tuned.non_isolcpus=000003ff intel_pstate=disable nosoftlockup # echo "kernel.printk = 0 4 0 0" > /etc/sysctl.d/95-printk.conf # sysctl --system | grep kernel.printk kernel.printk = 0 4 0 0