Bug 1255343

Jeremy, that's an interesting question. Fortunately no further file specification expansion is required and I'd like to propose you something different.

Let me describe this issue in more details so other readers of this ticket will understand what you want to achieve.

Imagine a system where we have a lot of interfaces. I'm sitting in front of the system like this - it has few interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
5: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
6: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
11: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
12: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
15: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
38: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
43: vnet16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
44: vnet13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
45: vnet14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
50: vnet15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
53: vnet17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
54: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
86: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
87: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
88: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
91: vnet11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
92: vnet12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
93: vnet19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
94: vnet20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
100: vnet9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
101: vnet10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
102: vnet21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
103: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
104: vnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500

Let's say I'd like to capture traffic from br0, and virbr0. There are two ways to do so.

0. Dumb one (with tcpdump - I suspect Jeremy uses it): /ust/sbin/tcpdump -s 0 -i any -w ~/all_traffic_with_duplicates.log 
1. More complicate one (with dumpcap from Wireshark): 

/usr/sbin/dumpcap -t -n -b filesize:1024 -Z none -w /var/tmp/tcpdump/tcpdump -g -i br0 -f "port 80 or port 5060 or port 1812 or port 1813 or port 22222" -i virbr0 -f "port 5060 or port 1812 or port 1813 or port 22222"

See I've posted a slightly more complicated string. I just copied an actual command line and don't want to bother with shortening it right now.

The former (dumb way) captures the entire traffic both from br0 and virbr0 and from physical interfaces attached to it thus doubling packet's number. It's due to limitations of tcpdump itself.

So one has a huge chunks of data (50% loss btw) but easily manageable - merging, selecting, extracting with everything tool available.

The latter way is more clever, space-efficient, and it even saves an interface names which greatly helps debugging network issues later. Jeremy hope you're still reading this boring text! If you're indeed reading then I advise you to use this dumpcap trick rather than stick with good-old "tcpdump -i any". Just make yourself comfortable with uids/guids requirements of dumpcap (similar to tcpdump but slightly different).

One issue with the latter approach still remains though! Although dumpcap makes a perfect pcapng-packets with all the required information available, other tools lacks behind!

Imagine a person debugging a Niagara falls of network data. (S)he would likely wants to split capture files into usable pieces (let's say 1Mbyte ones as I did in my dumpcap example above). Later (s)he pick up only those 1mbyte chunks which are really necessary, and left others intact. This will reduce greatly time spent on debugging this (esp. if this person knows scripting well).

Let's assume (s)he picks a few packets. One thing is certain - they need to be merged into one bigger dump. Ok, it's certainly possible with pcap-merging tool, right?

work ~/work/wireshark (git::master): /usr/sbin/mergecap -w ~/all.pcap ~/tcpdump_44599_20150130050529  ~/tcpdump_44600_20150130050538
mergecap: Error writing to outfile: Internal error
work ~/work/wireshark (git::master):

What the heck is this "Internal error"?

Unfortunately there is a great story behind this strange error message. Long story short there is no way to merge these "clever" pcapng chunks (and there are few more issues if someone indeed merges them with scripts or something). See these upstream issues for further links and details:

* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=7381
* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=8795
* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=10308
* https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=11202

There were multiple attempts to fix/improve this situation (few of them were mine), and fortunately after a couple of years it has been fixed recently!

* https://code.wireshark.org/review/#/c/10058/

What Jeremy wants is now possible with this changeset. The problem is that this changeset for hydrabug (a bug, which causes even more bugs while trying to close it) is a hydrapatch itself (a patch which generates even more patches to be accepted) so I'm not sure if it's easy to backport it to 1.12.x branch.

We should wait for 1.99.x / 2.x.x branch I suppose. Or we should backport a really huge and invasive piece of code.

That sounds quite neat and something I could use.  However the issue is with
customer-gathered captures, and they (mostly) know and understand
"tcpdump -i any".  Is there any prospect of getting that to output a single file
with multiple IDBs?

(In reply to Jeremy Harris from comment #2)
> That sounds quite neat and something I could use.  However the issue is with
> customer-gathered captures, and they (mostly) know and understand
> "tcpdump -i any".  Is there any prospect of getting that to output a single
> file
> with multiple IDBs?

Unfortunately no. 

I've heard that Mac OS X provides a different (heavily patched?) tcpdump with libcap which could save file to a pcapng format but I don't think "Macbook networking issues" is your customer's case.

See this:

* http://thread.gmane.org/gmane.network.tcpdump.devel/6797/focus=6798

Ok, changing title, adding "FutureFeature", and reassigning to tcpdump.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

As tcpdump is much more of a standard tool than wireshark, I would say we should keep it as usable as possible. I'm not commenting on the exact proposed solution, just that it would be IMO better to extend tcpdump than to make people use wireshark.

See also:

* https://github.com/the-tcpdump-group/tcpdump/issues/480 (Support capturing on multiple interfaces)

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

As mentioned before, the upstream developers know about this request. I'm closing this bug.