Description of problem:
1% packet loss on dpdk interface on skylakes cpu that is not consistently present across rebots:


First reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	171	10	0	0
End	20562886	662	0	2516
% drops		0.003	0.000	0.012
Total % drops	0.015			
				
Second reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	24303968	53799	0	14531
End	48123179	216201	0	24630
% drops		0.682	0.000	0.042
Total % drops	0.724			
				
Third reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	120	9	0	0
End	24262297	227377	0	12917
% drops		0.937	0.000	0.053
Total % drops	0.990			
				
Fourth reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	126	7	0	0
End	24186737	186242	57	37346
% drops		0.770	0.000	0.154
Total % drops	0.925			
				
Fifth reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	120	9	0	0
End	20937555	57	0	5443
% drops		0.000	0.000	0.026
Total % drops	0.026

We do not observe this issue on broadwell CPUs and updating to latest kernel/qemu/libvirt/etc worsen the situation

Version-Release number of selected component (if applicable):
openvswitch-2.9.0-56.el7fdp.1.x86_64
kernel-3.10.0-862.3.3.el7.x86_64                            Mon Jun 18 15:24:22 2018
dpdk-17.11-7.el7.x86_64                                     Mon Jun 18 16:01:19 2018
libvirt-3.9.0-14.el7_5.5.x86_64                             Mon Jun 18 15:56:33 2018
qemu-kvm-rhev-2.10.0-21.el7_5.3.x86_64                      Mon Jun 18 15:56:23 2018

How reproducible:
Randomly

Steps to Reproduce:
1. Hit the packet loss issue
2. Reboot the compute node multiple times
3.

Actual results:
Inconsistency in packet loss as we might not hit the issue before we reboot the compute node again.

Expected results:
Consistent results

Additional info:
cmdline:
BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.3.3.el7.x86_64 root=UUID=68da77cc-ba27-41f2-ba68-fb165a0d503f ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet isolcpus=1,3,5,7-43,45,47,49,51-87 default_hugepagesz=1GB hugepagesz=1G hugepages=480 iommu=pt intel_iommu=on processor.max_cstate=0 intel_idle.max_cstate=0 transparent_hugepage=never nmi_watchdog=0 audit=0 hpet=disable tsc=reliable skew_tick=1 rcu_nocbs=1,3,5,7-43,45,47,49,51-87 tuned.non_isolcpus=00055000,00000055 intel_pstate=disable nosoftlockup

systemd/system.conf:CPUAffinity=0 2 4 6 44 46 48 50

nova.conf:
vcpu_pin_set=1,3,5,7-43,45,47,49,51-87,^8,^10,^52,^54,^1,^3,^5,^7,^9,^11,^45,^47,^49,^51,^53,^55

tuned profile:
isolated_cores=1,3,5,7-43,45,47,49,51-87

ovs:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 1:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 3:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 5:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 7:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 8:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 9:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 10:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 11:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 45:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 47:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 49:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 51:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 52:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 53:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 54:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 55:


dpdk:
[dhill@supportshell openvswitch]$ cat driverctl_list-overrides
0000:5e:00.0 vfio-pci
0000:5e:00.1 vfio-pci
0000:87:00.0 vfio-pci
0000:87:00.1 vfio-pci
0000:af:00.0 vfio-pci
0000:af:00.1 vfio-pci


core/queue affinity:

pmd thread numa_id 1 core_id 1:
        port: vhu38c9ef12-48    queue-id:  7    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  3    pmd usage:  0 %
pmd thread numa_id 1 core_id 3:
        port: vhu38c9ef12-48    queue-id:  2    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  6    pmd usage:  0 %
pmd thread numa_id 1 core_id 5:
        port: vhu6841e460-0b    queue-id:  6    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  2    pmd usage:  0 %
pmd thread numa_id 1 core_id 7:
        port: vhu38c9ef12-48    queue-id:  3    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  7    pmd usage:  0 %
pmd thread numa_id 0 core_id 8:
pmd thread numa_id 1 core_id 9:
        port: vhu38c9ef12-48    queue-id:  5    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  1    pmd usage:  0 %
pmd thread numa_id 0 core_id 10:
pmd thread numa_id 1 core_id 11:
        port: vhu6841e460-0b    queue-id:  5    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  1    pmd usage:  0 %
pmd thread numa_id 1 core_id 45:
        port: vhu38c9ef12-48    queue-id:  4    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  0    pmd usage:  0 %
pmd thread numa_id 1 core_id 47:
        port: vhu6841e460-0b    queue-id:  7    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  3    pmd usage:  0 %
pmd thread numa_id 1 core_id 49:
        port: vhu38c9ef12-48    queue-id:  0    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  4    pmd usage:  0 %
pmd thread numa_id 1 core_id 51:
        port: vhu6841e460-0b    queue-id:  4    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  0    pmd usage:  0 %
pmd thread numa_id 0 core_id 52:
pmd thread numa_id 1 core_id 53:
        port: vhu38c9ef12-48    queue-id:  1    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  5    pmd usage:  0 %
pmd thread numa_id 0 core_id 54:
pmd thread numa_id 1 core_id 55:
        port: vhu38c9ef12-48    queue-id:  6    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  2    pmd usage:  0 %