1. Please describe the problem:
We compared performance of ELN and RHEL9 candidate kernels and noticed significant drop in XDP drop [1] on mlx5 (25G).

On any rhel9 candidate kernel we are able to drop 19-20M pkts/sec but on an ELN kernels, we are reaching just 15M pkts/sec (CPU utillization remains the same - around 100%). 

We don't see such regression on ixgbe or i40e.


[1] https://github.com/xdp-project/xdp-tools/tree/master/xdp-bench#the-drop-command

2. What is the Version-Release number of the kernel:

kernel-6.4.0-0.rc6.20230616git40f71e7cd3c6.50.eln126
  https://koji.fedoraproject.org/koji/buildinfo?buildID=2217297

We tested just x86_64, not sure about other archs.

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

Yes, it (most likely) comes from some patch in mentioned kernel, the previous one (kernel-6.4.0-0.rc6.20230614gitb6dad5178cea.49.eln126 - https://koji.fedoraproject.org/koji/taskinfo?taskID=102148156) is ok.



4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Reproducible always.

Steps:
- install affected kernel, kernel-modules-extra and kernel-modules-internal packages (for pktgen)
Traffic generator machine (wsfd-advnetlab65.anl.eng.rdu2.dc.redhat.com):
git clone https://github.com/torvalds/linux.git
cd linux/samples/pktgen/
./pktgen_sample03_burst_single_flow.sh -m MAC -d IP -i INF

DUT machine (receiver) (wsfd-advnetlab66.anl.eng.rdu2.dc.redhat.com):
dnf install -y git nano cmake clang llvm elfutils-libelf-devel libpcap-devel perf bpftool m4
git clone https://github.com/xdp-project/xdp-tools.git
cd xdp-tools/
./configure
make
./xdp-bench drop INF


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Not sure, i've not checked.


6. Are you running any modules that not shipped with directly Fedora's kernel?:
No.


7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.
-

Reproducible: Always