Bug 1064963

Summary:	Firewall issue with virtual machines on different networks
Product:	[Fedora] Fedora	Reporter:	Melvin Jones <melvinj>
Component:	libvirt	Assignee:	Libvirt Maintainers <libvirt-maint>
Status:	CLOSED DUPLICATE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	20	CC:	berrange, clalancette, itamar, jforbes, jyang, laine, libvirt-maint, veillard, virt-maint
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-02-14 13:43:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Melvin Jones 2014-02-13 15:59:37 UTC

Description of problem:

I first noticed this problem on CentOS 6 while studying for the RHCSA exam and decided to try it out on my Fedora 20 box to see if it was still an issue and it is.

When you have two virtual networks created and you create a VM using each network, only one VM can ping or connect to the other.  One system pings fine and the other machine gets an error of "Destination Port Unreachable".  I took a look at the firwall rules on the host machine and see something like this.

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       35  2684 ACCEPT     all  --  *      virbr1  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
2       35  2684 ACCEPT     all  --  virbr1 *       192.168.122.0/24     0.0.0.0/0
3        0     0 ACCEPT     all  --  virbr1 virbr1  0.0.0.0/0            0.0.0.0/0
4        8   480 REJECT     all  --  *      virbr1  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
5        0     0 REJECT     all  --  virbr1 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
6    17699 3387K ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.100.0/24     ctstate RELATED,ESTABLISHED
7    32439   40M ACCEPT     all  --  virbr0 *       192.168.100.0/24     0.0.0.0/0
8        0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0
9        0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
10       0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable


I'm assuming these rules are added by libvirt and from these rules we can see that any traffic leaving virbr1 going to virbr0 will be accepted by rule #2; however, traffic leaving virbr0 going to virbr1 will be rejected by rule #4.  If rule #4+5 are moved below #8 then everything works fine or if you move rule #7 below rule #3 it also works.  I am not sure whether the correct behavior is to not have the two networks able to see each other by default or that they should both see each by default, but I assume that working one way but not the other is not the desired behavior.

Version-Release number of selected component (if applicable):

libvirt 1.1.3.3-5

How reproducible:

100%

Steps to Reproduce:
1. Create a second network using the Virtual Machine manager
2. Create two VMs, one on the default network and another on the new network.
3. Bring up both machines and try to ping between them.

Actual results:

Ping only works one way, which is the machine connected to the interface listed first in the FORWARD chain of the filter table.

Expected results:

Either both machines should be able to ping each other or neither machine should be able to ping the other.

Comment 1 Laine Stump 2014-02-14 11:46:35 UTC

This problem has existed for a *very* long time. It was first reported in Bug 453580. So far it hasn't been important enough to anyone to fix it, and so things have remained as they are for the last 5 years.

The *desired* behavior is that guests on different isolated and NATed networks should *not* be able to communicate with each other. The problem is that it's not possible to do that by merely re-ordering the rules, even if all the rules for all networks were added at the same time (and thus could be arbitrarily re-ordered) which anyway isn't the case (each network can and is started/stopped independently of all other networks). The presence of the nat table rules, which are apparently handled prior to the FORWARD rules, makes it even more complicated. Lacking any solution from someone more intelligent about iptables, the only way I can see to make this work would be to add a specific REJECT rule for "-i virbrX -o virbrY" for every pair of NATed/isolated networks, which is not at all scalable.

Fortunately, in the time since this was first reported, it hasn't been an important issue for anyone. But of course as always, that is subject to change, and patches are always welcome.

Comment 2 Melvin Jones 2014-02-14 13:42:53 UTC

(In reply to Laine Stump from comment #1)

Ok, thanks for the reply.  I tried to search for this problem before I posted it, but I guess I didn't do a good job of that.  I have a workaround for it at the moment so it's not big deal and I don't see myself trying to use this in any kind of production environment at the moment.  It was just something that came up in the book I was using to study for the RHCSA exam.

Comment 3 Melvin Jones 2014-02-14 13:43:39 UTC


*** This bug has been marked as a duplicate of bug 453580 ***

Comment 4 Daniel Berrangé 2018-11-01 13:33:25 UTC

FYI we believe we've finally solved the problem with this series

https://www.redhat.com/archives/libvir-list/2018-November/msg00018.html