1662597 – 1% packet loss on dpdk interface on skylakes cpu

Bug 1662597 - 1% packet loss on dpdk interface on skylakes cpu

Summary: 1% packet loss on dpdk interface on skylakes cpu

Keywords:
Status:	CLOSED INSUFFICIENT_DATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openvswitch
Sub Component:
Version:	10.0 (Newton)
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Andreas Karis
QA Contact:	Yariv
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-12-30 16:07 UTC by David Hill
Modified:	2019-07-01 15:26 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-07-01 15:26:24 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description David Hill 2018-12-30 16:07:17 UTC

Description of problem:
1% packet loss on dpdk interface on skylakes cpu that is not consistently present across rebots:


First reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	171	10	0	0
End	20562886	662	0	2516
% drops		0.003	0.000	0.012
Total % drops	0.015			
				
Second reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	24303968	53799	0	14531
End	48123179	216201	0	24630
% drops		0.682	0.000	0.042
Total % drops	0.724			
				
Third reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	120	9	0	0
End	24262297	227377	0	12917
% drops		0.937	0.000	0.053
Total % drops	0.990			
				
Fourth reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	126	7	0	0
End	24186737	186242	57	37346
% drops		0.770	0.000	0.154
Total % drops	0.925			
				
Fifth reboot				
	tx_packets	tx_dropped	ifconfig dropped	netstat buffer errors
Start	120	9	0	0
End	20937555	57	0	5443
% drops		0.000	0.000	0.026
Total % drops	0.026

We do not observe this issue on broadwell CPUs and updating to latest kernel/qemu/libvirt/etc worsen the situation

Version-Release number of selected component (if applicable):
openvswitch-2.9.0-56.el7fdp.1.x86_64
kernel-3.10.0-862.3.3.el7.x86_64                            Mon Jun 18 15:24:22 2018
dpdk-17.11-7.el7.x86_64                                     Mon Jun 18 16:01:19 2018
libvirt-3.9.0-14.el7_5.5.x86_64                             Mon Jun 18 15:56:33 2018
qemu-kvm-rhev-2.10.0-21.el7_5.3.x86_64                      Mon Jun 18 15:56:23 2018

How reproducible:
Randomly

Steps to Reproduce:
1. Hit the packet loss issue
2. Reboot the compute node multiple times
3.

Actual results:
Inconsistency in packet loss as we might not hit the issue before we reboot the compute node again.

Expected results:
Consistent results

Additional info:
cmdline:
BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.3.3.el7.x86_64 root=UUID=68da77cc-ba27-41f2-ba68-fb165a0d503f ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet isolcpus=1,3,5,7-43,45,47,49,51-87 default_hugepagesz=1GB hugepagesz=1G hugepages=480 iommu=pt intel_iommu=on processor.max_cstate=0 intel_idle.max_cstate=0 transparent_hugepage=never nmi_watchdog=0 audit=0 hpet=disable tsc=reliable skew_tick=1 rcu_nocbs=1,3,5,7-43,45,47,49,51-87 tuned.non_isolcpus=00055000,00000055 intel_pstate=disable nosoftlockup

systemd/system.conf:CPUAffinity=0 2 4 6 44 46 48 50

nova.conf:
vcpu_pin_set=1,3,5,7-43,45,47,49,51-87,^8,^10,^52,^54,^1,^3,^5,^7,^9,^11,^45,^47,^49,^51,^53,^55

tuned profile:
isolated_cores=1,3,5,7-43,45,47,49,51-87

ovs:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 1:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 3:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 5:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 7:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 8:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 9:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 10:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 11:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 45:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 47:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 49:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 51:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 52:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 53:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 0 core_id 54:
ovs-appctl_dpif-netdev.pmd-rxq-show:pmd thread numa_id 1 core_id 55:


dpdk:
[dhill@supportshell openvswitch]$ cat driverctl_list-overrides
0000:5e:00.0 vfio-pci
0000:5e:00.1 vfio-pci
0000:87:00.0 vfio-pci
0000:87:00.1 vfio-pci
0000:af:00.0 vfio-pci
0000:af:00.1 vfio-pci


core/queue affinity:

pmd thread numa_id 1 core_id 1:
        port: vhu38c9ef12-48    queue-id:  7    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  3    pmd usage:  0 %
pmd thread numa_id 1 core_id 3:
        port: vhu38c9ef12-48    queue-id:  2    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  6    pmd usage:  0 %
pmd thread numa_id 1 core_id 5:
        port: vhu6841e460-0b    queue-id:  6    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  2    pmd usage:  0 %
pmd thread numa_id 1 core_id 7:
        port: vhu38c9ef12-48    queue-id:  3    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  7    pmd usage:  0 %
pmd thread numa_id 0 core_id 8:
pmd thread numa_id 1 core_id 9:
        port: vhu38c9ef12-48    queue-id:  5    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  1    pmd usage:  0 %
pmd thread numa_id 0 core_id 10:
pmd thread numa_id 1 core_id 11:
        port: vhu6841e460-0b    queue-id:  5    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  1    pmd usage:  0 %
pmd thread numa_id 1 core_id 45:
        port: vhu38c9ef12-48    queue-id:  4    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  0    pmd usage:  0 %
pmd thread numa_id 1 core_id 47:
        port: vhu6841e460-0b    queue-id:  7    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  3    pmd usage:  0 %
pmd thread numa_id 1 core_id 49:
        port: vhu38c9ef12-48    queue-id:  0    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  4    pmd usage:  0 %
pmd thread numa_id 1 core_id 51:
        port: vhu6841e460-0b    queue-id:  4    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  0    pmd usage:  0 %
pmd thread numa_id 0 core_id 52:
pmd thread numa_id 1 core_id 53:
        port: vhu38c9ef12-48    queue-id:  1    pmd usage:  0 %
        port: vhuf597f3ff-28    queue-id:  5    pmd usage:  0 %
pmd thread numa_id 0 core_id 54:
pmd thread numa_id 1 core_id 55:
        port: vhu38c9ef12-48    queue-id:  6    pmd usage:  0 %
        port: vhu6841e460-0b    queue-id:  2    pmd usage:  0 %

Comment 2 David Hill 2018-12-30 16:21:48 UTC

Intel engineers suspect a CPU jitter issue and if we look at the Inst_jitter value across the reboots, the values are 3 times higher when we have 0% packet loss:

Linux Jitter testing program version 1.9
The pragram will execute a dummy function 80000 times
Display is updated every 20000 displayUpdate intervals
Thread affinity will be set to core_id:1
Timings are in CPU Core cycles
Inst_Min:    Minimum Excution time during the display update interval(default is ~1 second)
Inst_Max:    Maximum Excution time during the display update interval(default is ~1 second)
Inst_jitter: Jitter in the Excution time during rhe display update interval. This is the value of interest
last_Exec:   The Excution time of last iteration just before the display update
Abs_Min:     Absolute Minimum Excution time since the program started or statistics were reset
Abs_Max:     Absolute Maximum Excution time since the program started or statistics were reset
tmp:         Cumulative value calcualted by the dummy function
Interval:    Time interval between the display updates in Core Cycles
Sample No:   Sample number


rx_jitter:
   Inst_Min   Inst_Max   Inst_jitter last_Exec  Abs_min    Abs_max      tmp       Interval     Sample No
    235676    4448456    4212780     237986     187418    6539754    3427270656 5032732318        200

tx_jitter:
   Inst_Min   Inst_Max   Inst_jitter last_Exec  Abs_min    Abs_max      tmp       Interval     Sample No
    233496    4446326    4212830     236306     192510    4471034      74776576 5042368446        200



tx_jitter (no packet loss)
   Inst_Min   Inst_Max   Inst_jitter last_Exec  Abs_min    Abs_max      tmp       Interval     Sample No
    216046   12830160   12614114     236950     134728   14919230    4083351552 6302145314        200

rx_jitter (no packet loss)
   Inst_Min   Inst_Max   Inst_jitter last_Exec  Abs_min    Abs_max      tmp       Interval     Sample No
    235038   10732892   10497854     237876     190770   11054744     912457728 5783430602        200

Comment 3 David Hill 2018-12-30 16:28:11 UTC

The tool output referenced above is from here [1] and is referenced here [2].

[1] https://git.fd.io/pma_tools/tree/jitter
[2] https://docs.fd.io/csit/master/report/vpp_performance_tests/test_environment.html

Note You need to log in before you can comment on or make changes to this bug.