Bug 557188

Summary: qemu mcast NIC type falsely returns sent packets to the sender (in promiscuous/bridged mode)
Product: [Fedora] Fedora Reporter: Need Real Name <karlcz>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 14CC: amit.shah, berrange, crobinso, dwmw2, ehabkost, itamar, jaswinder, jforbes, jfriesse, mst, neil, quintela, tburke, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-20 18:03:18 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Need Real Name 2010-01-20 12:14:20 EST
Description of problem:

Virtual NIC of type "mcast" receives copies of what it sent, resulting in many disastrous behaviors if you add one to a Linux software ethernet bridge.

Version-Release number of selected component (if applicable):


How reproducible:

Always.


Steps to Reproduce:
1. create a kvm/qemu guest with an mcast NIC as eth0
2. create a bridge br0 in the guest, enabling STP
3. add eth0 to br0 as its only slave
  
Actual results:

Once STP starts sending frames, the host starts reporting:

eth0: received packet with own address as source address

Expected results:

eth0 should not receive copies of what it sends.

Additional info:

This happens as above when the NIC goes into promiscuous mode. I have not bothered to verify whether it happens for multicast or broadcast traffic in non-promiscuous mode.

Things become more obvious and destructive if you add another slave to the bridge.  Any incoming broadcast or multicast traffic from the other slave gets bridged into the mcast NIC, reflected back, and creating a kind of psuedo-loop. This confuses the user, upsets everyone on the other LAN, and also confuses the Linux software bridge as it starts falsely discovering all the reflected MACs as if they are reachable behind the mcast NIC.

With a manually created list of the MAC addresses actually behind the mcast LAN, one can create draconian ebtables filters to drop all reflected packets via a command like:

ebtables -t nat -A PREROUTING -i ethX --among-src-file ! /etc/mcast-mac-addrs -j dnat --dnat-target DROP

This will drop the bad frames before they confuse the bridging code, and you end up with a working bridged mcast network.  This proves that the problem is reflected packets coming back to the sender via the mcast NIC.
Comment 1 David Woodhouse 2010-01-20 16:58:57 EST
This would presumably break all IPv6 duplicate address detection, so guests don't work with IPv6?
Comment 2 Need Real Name 2010-01-20 17:16:22 EST
I am not using IPv6 in my test environment, so I cannot confirm nor deny. The network goes haywire very rapidly if one does this experiment without the ebtables rule above, leading one to rip cables out of walls as fast as one can.

It is a great DoS attack on the victim LAN that you bridge into such an mcast tunnel, as it reflects all broadcast traffic such as mDNS and probably confuses hardware switches as well as the Linux software bridge.  As such, I won't be performing more detailed tests of this failure mode in my environment!
Comment 3 Daniel Berrange 2010-01-21 04:49:20 EST
Your initial description doesn't entirely make sense. The QEMU 'mcast' nic type is totally isolated from the host network, but in the steps to reproduce you're talking about setting up bridges in the host which implies the 'tap' nic type in QEMU. 

Please provide the exact QEMU command line you are invoking, and the output of 'ifconfig -a', 'brctl show' and 'route -n' on the host OS.
Comment 4 Need Real Name 2010-01-21 10:15:30 EST
No, I described setting up a bridge in the guest with the mcast NIC!

So the bug-triggering guest needs only one NIC, eth0, which is defined like this:

    <interface type='mcast'>
      <mac address='...'/>
      <source address='...' port='5558'/>
    </interface>

(with actual local MAC and multicast address assignments).

After booting that guest, use brctl to create a bridge br0, and simply slave that virtual eth0 to the bridge (as its only "physical" port).  This will exhibit the problem, with no real networks involved.

When I mentioned the host LAN, I had created a second NIC in a privileged guest, so it had one mcast NIC and one tap NIC already bridged into the host LAN by libvirt.  Then that guest bridged its two NICs, allowing other unprivileged guests to use an mcast NIC to participate in the host LAN.  Because of this mcast bug, you can only do this once this privileged guest has the severe ebtables workaround I reported.

But this extra step is unnecessary to demonstrate the bug.  It is only interesting in that it does something useful once the ebtables workaround is in place (allowing unprivileged guests to participate in a real LAN).
Comment 5 Fedora Admin XMLRPC Client 2010-03-09 11:53:57 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 6 Fedora Admin XMLRPC Client 2010-03-09 12:19:49 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 7 Bug Zapper 2010-03-15 10:08:33 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 8 Jan Friesse 2011-02-15 11:25:53 EST
I found this bug too.

Simple way how to reproduce:

Run kvm guest 1 and 2, with net something like this:
g1:-net nic,vlan=0,model=virtio,macaddr=52:54:00:00:01:01 -net socket,vlan=0,mcast=239.255.0.1:4097

g2: -net nic,vlan=0,model=virtio,macaddr=52:54:00:00:02:01 -net socket,vlan=0,mcast=239.255.0.1:4097

This should give you two running vm on same net.

Now in each guest configure ip addresses:
g1: ifconfig eth0 192.168.1.1
g2: ifconfig eth0 192.168.1.2

now on g2 ping 192.168.1.1

and on g1 run tcpdump -i eth0 icmp
and every second you will get:
time ip 192.168.1.2 ... ICMP echo request
time ip 192.168.1.1 ... ICMP echo reply
time ip 192.168.1.1 ... ICMP echo reply

There shouldn't be two echo replies, but only one.

Another effect is, that Duplicate Address Detection (for ipv6) simply doesn't work (this is what original bug is talking about).


Of course, solution is not to disable loopback for mcast socket, because there must be chance to run more then one VM on same host on same mcast address:port.

Solution may be to detect sending socket ip:port pair and simply drop packets received from this ip:port.
Comment 9 Jan Friesse 2011-02-15 11:41:39 EST
Just one extra note, I was able to reproduce this bug on  qemu-system-x86-0.13.0-1.fc14.i686
Comment 10 Neil Wilson 2011-04-15 13:57:50 EDT
Logged as Qemu bug on Launchpad:

https://bugs.launchpad.net/qemu/+bug/761469
Comment 11 Bug Zapper 2011-06-02 12:48:53 EDT
This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 12 Jan Friesse 2011-06-03 02:10:19 EDT
Bug is reproducible on fc14 too
Comment 13 Cole Robinson 2012-05-20 18:03:18 EDT
F14 is end-of-life now. If anyone is still affected by this bug with newer qemu versions, I'd recommend following up in the (still open) upstream bug report mentioned in Comment #10