Description of problem:
TCP packet re-ordering noticed between instance on same compute using dpdk network .
A 1GB file is downloaded from instance over http to another instance using wget over vxlan tenant netdev/dpdk network which are running on same compute node.
tcp packet are seen out of order by ~ 0.1ms.
These are not drops, but actual segements arriving late.
Version-Release number of selected component (if applicable):
Only in customer environment
Steps to Reproduce:
i. Instance using single queue on same compute node.
ii. Confirmed that a single vcpu servicing eth2 interrupts in the guest.
iii. Non-isolated cpus used for guest. CPUAffinity is moved aware from these cores via systemd though.
iv. HT enabled.
v. Tested with 4 pmd cores . Tested with single pmd core.
The issue is still noticed.
vi. MTU of vhu interface is 1950, but tcpdump indicates 1514 segment size.
Despite fragmentation, we still don't expect packets to go out of order.
I think this 1514 size frames are seen because of the MTU on br-tun is 1500.
This bridge is automatically created by neutron.
I dont see any option to configure an MTU for br-tun. Any suggestions?
I have requested customer to test again with
sudo ip link set dev br-tun mtu 1950
It's possible this is related to https://mail.openvswitch.org/pipermail/ovs-dev/2018-June/348267.html
Would the customer be willing to try a build? I can patch this in and send it along just as a wild guess (seems to possibly match the symptoms?)
Created attachment 1452950 [details]
Created attachment 1454356 [details]
Patched RPM - not src version
According to our records, this should be resolved by openvswitch-2.9.0-56.el7fdp. This build is available now.
Created attachment 1486911 [details]
openvswitch package with the EMC reorder backport
This is an RPM with a 6 patch series backported, most notably this one:
From 9b4f08cdcaf253175edda088683bdd3db9e4c097 Mon Sep 17 00:00:00 2001
From: Vishal Deep Ajmera <email@example.com>
Date: Fri, 27 Jul 2018 23:56:37 +0530
Subject: [PATCH] dpif-netdev: Avoid reordering of packets in a batch with same megaflow
full commit message here:
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.
If this bug does not require doc text, please set the 'requires_doc_text' flag to -.
Till 10z8 we were using openvswitch-2.6.1-16.git20161206.el7ost.x86_64.rpm
10z9 deployments moved to openvswitch-2.9.0-56.el7fdp.x86_64.rpm
Can you advise which version and openvswitch to test?