Bug 2196224

Description of problem:
When running DPDK checkup, there are some nodes, that when the traffic generator andthe VM are scheduled on, the checkup ends with packet loss.


Version-Release number of selected component (if applicable):
CNV 4.13.0
container-native-virtualization-kubevirt-dpdk-checkup-rhel9:v4.13.0-37


How reproducible:
Most of the times (on specific nodes).


Steps to Reproduce:
1. Create namespace for the job, and change context to the new namespace.
$ oc create ns dpdk-checkup-ns
$ oc project dpdk-checkup-ns

2. Label the worker nodes with "worker-dpdk" label.

3. Apply the resources manifests in the attached file in their numeric order:
$ oc apply -f 1-dpdk-checkup-resources.yaml
$ oc apply -f 2-dpdk-checkup-scc.yaml
...
change the resources according to your cluster.

Please note:
Due to https://bugzilla.redhat.com/show_bug.cgi?id=2193235, you cannot set which nodes will be used for scheduling the VM and the traffic generator.
Therefore, you must W/A it by either uncordoning 2 workers and leaving only one as schedulable, or removing the "dpdk-workers" label from 2 nodes and keeping it on only one node.

4. After the job is completed - check the ConfigMap:
$ oc get cm dpdk-checkup-config -o yaml
...
  status.failureReason: 'not all generated packets had reached DPDK VM: Sent from
    traffic generator: 480000000; Received on DPDK VM: 110323573'
  status.result.DPDKRxPacketDrops: "0"
  status.result.DPDKRxTestPackets: "110323573"
  status.result.DPDKTxPacketDrops: "0"
  status.result.DPDKVMNode: cnv-qe-infra-06.cnvqe2.lab.eng.rdu2.redhat.com
  status.result.trafficGeneratorInErrorPackets: "0"
  status.result.trafficGeneratorNode: cnv-qe-infra-06.cnvqe2.lab.eng.rdu2.redhat.com
  status.result.trafficGeneratorOutputErrorPackets: "0"
  status.result.trafficGeneratorTxPackets: "480000000"
  status.startTimestamp: "2023-05-08T09:49:24Z"
  status.succeeded: "false"


Actual results:
<BUG> Note these fields:
  status.failureReason: 'not all generated packets had reached DPDK VM: Sent from
    traffic generator: 480000000; Received on DPDK VM: 110323573'
  status.succeeded: "false"


Expected results:
Successful job, no packet loss.


Additional info:
1. The diff between Tx bytes and Rx byte can be seen in the job log:
$ $ oc logs dpdk-checkup-8nhz9
...
2023/05/08 10:08:47 GetPortStats JSON: {
    "id": "a7mhi4qm",
    "jsonrpc": "2.0",
    "result": {
        "ibytes": 0,
        "ierrors": 0,
        "ipackets": 0,
        "m_cpu_util": 0.0,
        "m_total_rx_bps": 0.0,
        "m_total_rx_pps": 0.0,
        "m_total_tx_bps": 4063406080.0,
        "m_total_tx_pps": 7469495.5,
        "obytes": 32640000000,
        "oerrors": 0,
        "opackets": 480000000
    }
}
2023/05/08 10:08:48 GetPortStats JSON: {
    "id": "ntnu7u0h",
    "jsonrpc": "2.0",
    "result": {
        "ibytes": 30720000000,
        "ierrors": 844,
        "ipackets": 480000000,
        "m_cpu_util": 0.0,
        "m_total_rx_bps": 1902393984.0,
        "m_total_rx_pps": 3715611.0,
        "m_total_tx_bps": 0.0,
        "m_total_tx_pps": 0.0,
        "obytes": 0,
        "oerrors": 0,
        "opackets": 0

(compare the obytes in the first summary with the ibytes in the second summary).

2. The issue was found on 2 separate clusters bm01-cnvqe2-rdu2 and bm02-cnvqe2-rdu2.
On bm01-cnvqe2 the problematic node is cnv-qe-infra-06.cnvqe2.lab.eng.rdu2.redhat.com
On bm02-cnvqe2 the checkup cannot currently run, so I'm not sure which was the problematic node(s).