Bug 1571884 - [RFE] Production grade packet capture in OVS with netdev (DPDK) datapath
Summary: [RFE] Production grade packet capture in OVS with netdev (DPDK) datapath
Keywords:
Status: CLOSED DUPLICATE of bug 1411311
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openvswitch
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Flavio Leitner
QA Contact: Ofer Blaut
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-25 15:13 UTC by Andreas Karis
Modified: 2018-10-09 03:45 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-07 16:04:12 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Andreas Karis 2018-04-25 15:13:20 UTC
Description of problem:
[RFE] Production grade packet capture in OVS with netdev (DPDK) datapath

OVS-TCPDUMP with DPDK can have a significant impact on OVS DPDK performance. While it is still a valid and recommended option for troubleshooting OVS DPDK issues and for getting packet captures, we need a tool that does perform better.

Additional info from an internal discussion:

To answer to the question "do we have a way to capture packets in production with OVS-DPDK" the answer is "no".

That said, we understand the ask, and either this can be done at the ToR switch level (RSPAN), either with another certified SDN/vSwitch (Contrail, VTS, Nuage, ...). (...) Knowing that there is no free lunch and copying packets cost extra cycles per-packet when done on the compute node. And as adding cores won't help with OVS-DPDK, enabling packet capture may simply divide the performance by 2 or 3 per core. Knowing that we recommend to use one core, we can expect/hope/speculate that we will be between 1 and 2 Mpps / core with packet copy.

Comment 2 Flavio Leitner 2018-05-08 18:47:35 UTC
ovs-tcpdump leverages OVS mirror to send to another port which is a kernel device. That's where regular tcpdump is attached. This method works regardless of the datapath is DPDK or KERNEL, which is great, but of course, the context switch imposes a performance penalty.

The other existing solution is dpdk-tcpdump but that requires the binaries to match in a level that becomes too fragile to run in any environment that matters.

I suspect you want a less intrusive traffic capture mechanism when using DPDK datapath.  One option would be use leverage OVS mirror again, but sending the copies to an OVS DPDKR port (rings), which is then pulled by an application at full core speed dumping to a buffer/file.

That would require to burn another CPU, at least, and memory to allocate the ring and yet there would be an impact when copying the packets. Maybe it is just better to capture in another point, for example, at the switch port?

fbl

Comment 3 Andreas Karis 2018-05-09 21:53:00 UTC
Hi,

It's of course always an option to dump at the switch port but that doesn't capture packets somewhere on the virtualized path, e.g. for troubleshooting within the hypervisor. 

~~~
I suspect you want a less intrusive traffic capture mechanism when using DPDK datapath.  One option would be use leverage OVS mirror again, but sending the copies to an OVS DPDKR port (rings), which is then pulled by an application at full core speed dumping to a buffer/file.

That would require to burn another CPU, at least, and memory to allocate the ring and yet there would be an impact when copying the packets.
~~~
Is that a feature/path that's worth investigation? Let's keep this Bugzilla / RFE open and see if we get further feedback from the field.

- Andreas


Note You need to log in before you can comment on or make changes to this bug.