Bug 1255343 - RFE: tcpdump should support saving to pcapng format.
RFE: tcpdump should support saving to pcapng format.
Status: CLOSED UPSTREAM
Product: Fedora
Classification: Fedora
Component: tcpdump (Show other bugs)
rawhide
All Linux
unspecified Severity low
: ---
: ---
Assigned To: Martin Sehnoutka
Fedora Extras Quality Assurance
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-20 06:41 EDT by Jeremy Harris
Modified: 2017-02-03 05:18 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-02-03 05:18:35 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jeremy Harris 2015-08-20 06:41:11 EDT
Description of problem:

 When dealing with captures taken from "any" interface, on systems having
 bonding and/or VLANs (and, I guess, bridges and VMs) having the
 "Linux cooked capture" analysis section for each frame show the interface
 would aid sorting out apparent duplicates.

 Presumably this would involve a extension to the capture file specification
 and changes in multiple components; not a little job.  Sorry if opening
 the bug against wireshark is not appropriate.

Version-Release number of selected component (if applicable):
 Wireshark 1.12.6
 tcpdump version 4.7.4
 libpcap version 1.7.3

How reproducible:
 100%
Comment 1 Peter Lemenkov 2015-08-20 07:24:35 EDT
Jeremy, that's an interesting question. Fortunately no further file specification expansion is required and I'd like to propose you something different.

Let me describe this issue in more details so other readers of this ticket will understand what you want to achieve.

Imagine a system where we have a lot of interfaces. I'm sitting in front of the system like this - it has few interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
5: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
6: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
11: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
12: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
15: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
38: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
43: vnet16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
44: vnet13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
45: vnet14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
50: vnet15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
53: vnet17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
54: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
86: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
87: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
88: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
91: vnet11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
92: vnet12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
93: vnet19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
94: vnet20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
100: vnet9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
101: vnet10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
102: vnet21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
103: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
104: vnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500

Let's say I'd like to capture traffic from br0, and virbr0. There are two ways to do so.

0. Dumb one (with tcpdump - I suspect Jeremy uses it): /ust/sbin/tcpdump -s 0 -i any -w ~/all_traffic_with_duplicates.log 
1. More complicate one (with dumpcap from Wireshark): 

/usr/sbin/dumpcap -t -n -b filesize:1024 -Z none -w /var/tmp/tcpdump/tcpdump -g -i br0 -f "port 80 or port 5060 or port 1812 or port 1813 or port 22222" -i virbr0 -f "port 5060 or port 1812 or port 1813 or port 22222"

See I've posted a slightly more complicated string. I just copied an actual command line and don't want to bother with shortening it right now.

The former (dumb way) captures the entire traffic both from br0 and virbr0 and from physical interfaces attached to it thus doubling packet's number. It's due to limitations of tcpdump itself.

So one has a huge chunks of data (50% loss btw) but easily manageable - merging, selecting, extracting with everything tool available.

The latter way is more clever, space-efficient, and it even saves an interface names which greatly helps debugging network issues later. Jeremy hope you're still reading this boring text! If you're indeed reading then I advise you to use this dumpcap trick rather than stick with good-old "tcpdump -i any". Just make yourself comfortable with uids/guids requirements of dumpcap (similar to tcpdump but slightly different).

One issue with the latter approach still remains though! Although dumpcap makes a perfect pcapng-packets with all the required information available, other tools lacks behind!

Imagine a person debugging a Niagara falls of network data. (S)he would likely wants to split capture files into usable pieces (let's say 1Mbyte ones as I did in my dumpcap example above). Later (s)he pick up only those 1mbyte chunks which are really necessary, and left others intact. This will reduce greatly time spent on debugging this (esp. if this person knows scripting well).

Let's assume (s)he picks a few packets. One thing is certain - they need to be merged into one bigger dump. Ok, it's certainly possible with pcap-merging tool, right?

work ~/work/wireshark (git::master): /usr/sbin/mergecap -w ~/all.pcap ~/tcpdump_44599_20150130050529  ~/tcpdump_44600_20150130050538
mergecap: Error writing to outfile: Internal error
work ~/work/wireshark (git::master):

What the heck is this "Internal error"?

Unfortunately there is a great story behind this strange error message. Long story short there is no way to merge these "clever" pcapng chunks (and there are few more issues if someone indeed merges them with scripts or something). See these upstream issues for further links and details:

* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=7381
* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=8795
* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=10308
* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=11202

There were multiple attempts to fix/improve this situation (few of them were mine), and fortunately after a couple of years it has been fixed recently!

* https://code.wireshark.org/review/#/c/10058/

What Jeremy wants is now possible with this changeset. The problem is that this changeset for hydrabug (a bug, which causes even more bugs while trying to close it) is a hydrapatch itself (a patch which generates even more patches to be accepted) so I'm not sure if it's easy to backport it to 1.12.x branch.

We should wait for 1.99.x / 2.x.x branch I suppose. Or we should backport a really huge and invasive piece of code.
Comment 2 Jeremy Harris 2015-08-20 07:49:17 EDT
That sounds quite neat and something I could use.  However the issue is with
customer-gathered captures, and they (mostly) know and understand
"tcpdump -i any".  Is there any prospect of getting that to output a single file
with multiple IDBs?
Comment 3 Peter Lemenkov 2015-08-20 08:06:48 EDT
(In reply to Jeremy Harris from comment #2)
> That sounds quite neat and something I could use.  However the issue is with
> customer-gathered captures, and they (mostly) know and understand
> "tcpdump -i any".  Is there any prospect of getting that to output a single
> file
> with multiple IDBs?

Unfortunately no. 

I've heard that Mac OS X provides a different (heavily patched?) tcpdump with libcap which could save file to a pcapng format but I don't think "Macbook networking issues" is your customer's case.

See this:

* http://thread.gmane.org/gmane.network.tcpdump.devel/6797/focus=6798
Comment 4 Peter Lemenkov 2015-08-20 08:39:54 EDT
Ok, changing title, adding "FutureFeature", and reassigning to tcpdump.
Comment 5 Fedora Admin XMLRPC Client 2015-09-01 03:55:16 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 6 Pavel Šimerda (pavlix) 2015-09-01 08:03:53 EDT
As tcpdump is much more of a standard tool than wireshark, I would say we should keep it as usable as possible. I'm not commenting on the exact proposed solution, just that it would be IMO better to extend tcpdump than to make people use wireshark.
Comment 7 Peter Lemenkov 2015-10-29 10:58:32 EDT
See also:

* https://github.com/the-tcpdump-group/tcpdump/issues/480 (Support capturing on multiple interfaces)
Comment 8 Fedora Admin XMLRPC Client 2016-10-06 08:51:30 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 9 Fedora Admin XMLRPC Client 2016-10-26 03:28:32 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 10 Martin Sehnoutka 2017-02-03 05:18:35 EST
As mentioned before, the upstream developers know about this request. I'm closing this bug.

Note You need to log in before you can comment on or make changes to this bug.