Bug 458625

Summary: NetworkManager clobbers libvirt NAT rules
Product: [Fedora] Fedora Reporter: Andrew Bartlett <abartlet>
Component: NetworkManagerAssignee: Dan Williams <dcbw>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 9CC: berrange, dcbw, markmc, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-09-28 18:46:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 227011    
Bug Blocks:    
Attachments:
Description Flags
clobbered NAT rules (no virbr0 rules left)
none
Original NAT rules
none
/etc/sysconfig/iptables from boot none

Description Andrew Bartlett 2008-08-11 07:10:55 UTC
Created attachment 313911 [details]
clobbered NAT rules (no virbr0 rules left)

Description of problem:
NetworkManager clobbers libvirt NAT rules because it compleatly rewrites the iptables rules for the host, when sharing a connection.


Version-Release number of selected component (if applicable):
libvirt-0.4.4-2.fc9.i386
NetworkManager-0.7.0-0.10.svn3747.fc9.i386

How reproducible:
Every time

Steps to Reproduce:
1. Start libvirt at boot
2. Ensure ip forwarding is enabled
3. boot a virtual machine
4. ping external address from a VM
5. Enable a shared connection with the NetworkManager GUI

Actual results:
Ping to host successful, until 'connection sharing' is enabled.

Expected results:
Ping continues to succeed from the VM

Additional info:
The libvirt side of this is already being discussed in bug #227011 (requesting a sane way to add more iptables rules)

Comment 1 Andrew Bartlett 2008-08-11 07:13:03 UTC
Created attachment 313912 [details]
Original NAT rules

BTW, the libvirt rules seem better (not locked to sharing a particular interface, for example).

Comment 2 Andrew Bartlett 2008-08-11 07:14:25 UTC
Created attachment 313913 [details]
/etc/sysconfig/iptables from boot

Comment 3 Andrew Bartlett 2008-08-11 07:30:11 UTC
(sorry for the noise)

Comment 4 Dan Williams 2008-08-11 15:39:40 UTC
Yep; libvirt is currently incompatible with NetworkManager for this and a few other reasons; that's why libvirt sets up static networking rather than using NetworkManager.

Eventually we'll fix this when we start not using system() to do stuff with iptables.

Comment 5 Mark McLoughlin 2008-08-11 15:46:11 UTC
Why does NetworkManager clobber existing rules, though? Why not prepend/delete your own rules, leaving the existing ones intact?

(In reply to comment #4)
> Yep; libvirt is currently incompatible with NetworkManager for this and a few
> other reasons;

other reasons?

> that's why libvirt sets up static networking rather than using
> NetworkManager.

static networking ? We've no plans to use NetworkManager for any of the virtual networking functionality in libvirt ...

> Eventually we'll fix this when we start not using system() to do stuff with
> iptables.

How would that change the situation?

Comment 6 Dan Williams 2008-08-11 16:32:09 UTC
(In reply to comment #5)
> Why does NetworkManager clobber existing rules, though? Why not prepend/delete
> your own rules, leaving the existing ones intact?

For two reasons:

1) laziness: it wasn't clear to me during implementation how to get /sbin/iptables to remove the exact rule you specified; would be nice if the CLI tools returned a "rule ID" or something.  But it does allow you to specify a rule match which is what we'll use when I clean this up.

2) It wasn't clear to me how to completely clear the state without touching other stuff.  Since there's only one NAT table, and there's a lot of initial setup, how do you know that somebody already didn't set that stuff up when you're trying to tear it down?  Essentially, how do we back out this chunk of code:

	do_cmd ("echo \"1\" > /proc/sys/net/ipv4/ip_forward");
	do_cmd ("echo \"1\" > /proc/sys/net/ipv4/ip_dynaddr");
	do_cmd ("/sbin/modprobe ip_tables iptable_nat ip_nat_ftp ip_nat_irc");
	do_cmd ("/sbin/iptables -P INPUT ACCEPT");
	do_cmd ("/sbin/iptables -F INPUT");
	do_cmd ("/sbin/iptables -P OUTPUT ACCEPT");
	do_cmd ("/sbin/iptables -F OUTPUT");
	do_cmd ("/sbin/iptables -P FORWARD DROP");
	do_cmd ("/sbin/iptables -F FORWARD");
	do_cmd ("/sbin/iptables -t nat -F");

which is required to get the NAT up and running in the first place, without stomping all over libvirt.  You don't want to leave any of the NAT stuff set up if you're not actually NAT-ing, but of course how do you figure out something else is NAT-ing anyway?

> (In reply to comment #4)
> > Yep; libvirt is currently incompatible with NetworkManager for this and a few
> > other reasons;
> 
> other reasons?

Bridging really; though you can obviously tell NM to ignore the interface with some ifcfg-foo.

> > that's why libvirt sets up static networking rather than using
> > NetworkManager.
> 
> static networking ? We've no plans to use NetworkManager for any of the virtual
> networking functionality in libvirt ...
> 
> > Eventually we'll fix this when we start not using system() to do stuff with
> > iptables.
> 
> How would that change the situation?

More fine-grained and intelligent iptables manipulation can be achieved using something like libnl rather than trying to graft CLI tools like iptables into a C program when iptables clearly was not written to be plugged into other programs that need information about stuff.


All that said, the solution doesn't need to be perfect, and yes NM _should_ be trying to remove only the rules it has added.  Suggestions?  Should NM just back out the device-specific rules:

do_cmd ("/sbin/iptables -A FORWARD -i %s -o %s -m state --state ESTABLISHED,RELATED -j ACCEPT", extif, intif);
do_cmd ("/sbin/iptables -A FORWARD -i %s -o %s -j ACCEPT", extif, intif);
do_cmd ("/sbin/iptables -A FORWARD -i %s -o %s -j ACCEPT", intif, extif);
do_cmd ("/sbin/iptables -t nat -A POSTROUTING -o %s -j MASQUERADE", extif);

and leave the generic ones (listed near the top of this comment) in place even if there's no NAT-ing being done?

Comment 7 Daniel Berrangé 2008-08-12 09:38:44 UTC
> Yep; libvirt is currently incompatible with NetworkManager for this and a few
> other reasons; that's why libvirt sets up static networking rather than using
> NetworkManager.

This is *incorrect*.

Libvirt has two modes of networking

 - Bridging a guest to a physical device. This is not compatible with networkmanager, since bridged guests can't cope with switching between different host NICs

 - NAT based network. This was *explicitly* designed to work with networkmanager, so that virtualization can work out of the box. We create an isolated bridge device with no  physical NICs attached. Guests are connected to this bridge. NAT rules are then setup so that traffic from this bridge can be NAT'd to whichever physical NIC networkmanager has currently activated.

  http://wiki.libvirt.org/page/Networking

If new networkmanager has broken the second option, this is a show-stopper for release because this  means virtualization won't work on out of the box Fedora installs.  

NB, we have been very careful with libvirt IPtables rule to be as specific as possible when adding rules, and explicitly only remove rules we ourselves added. This is to ensure interoperability with other iptables rules & applications. 

If there are specific logic errors in the iptables rules libvirt adds, please file a bug against libvirt so we can address that.

Comment 8 Andrew Bartlett 2008-08-12 10:55:53 UTC
This is about NetworkManager breaking libvirt NAT networks when the new 'share connection' option (apparently a feature for F10, but also in F9) is enabled.

Comment 9 Dan Williams 2008-08-15 15:38:04 UTC
Upstream svn r3969 contains fixes that shouldn't stomp all over existing iptables config.

Comment 10 Andrew Bartlett 2008-08-15 23:19:43 UTC
Looking over the new rules in SVN (while I wait for packages to become available) I'm very pleased to see that they now only reference the shared device, so I should (I hope) be able to share my VPN connection to an IP phone (which was my original use case). 

Thanks!

Comment 11 Dan Williams 2008-08-18 13:44:01 UTC
Though you could use the iPhone's Cisco VPN client too :)  But yes, I believe this would allow VPN connections to be shared as well.   There may be security implications there but we'll handle that later because it'll take some UI work and user education.  Maybe something like VPN connections can only be re-shared over wifi if the wifi link is using encryption or the like.

Comment 12 Fedora Update System 2008-08-29 14:56:36 UTC
NetworkManager-openvpn-0.7.0-15.svn4027.fc9,NetworkManager-vpnc-0.7.0-0.10.svn4024.fc9,NetworkManager-0.7.0-0.11.svn4022.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/NetworkManager-openvpn-0.7.0-15.svn4027.fc9,NetworkManager-vpnc-0.7.0-0.10.svn4024.fc9,NetworkManager-0.7.0-0.11.svn4022.fc9

Comment 13 Fedora Update System 2008-09-02 20:41:49 UTC
NetworkManager-0.7.0-0.11.svn4022.fc8,NetworkManager-openvpn-0.7.0-15.svn4027.fc8,NetworkManager-vpnc-0.7.0-0.10.svn4024.fc8,NetworkManager-pptp-0.7.0-0.10.svn4027.fc8 has been submitted as an update for Fedora 8.
http://admin.fedoraproject.org/updates/NetworkManager-0.7.0-0.11.svn4022.fc8,NetworkManager-openvpn-0.7.0-15.svn4027.fc8,NetworkManager-vpnc-0.7.0-0.10.svn4024.fc8,NetworkManager-pptp-0.7.0-0.10.svn4027.fc8

Comment 14 Fedora Update System 2008-09-10 06:36:46 UTC
NetworkManager-pptp-0.7.0-0.10.svn4027.fc9, NetworkManager-openvpn-0.7.0-15.svn4027.fc9, NetworkManager-vpnc-0.7.0-0.10.svn4024.fc9, NetworkManager-0.7.0-0.11.svn4022.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update NetworkManager-pptp NetworkManager-openvpn NetworkManager-vpnc NetworkManager'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-7375

Comment 15 Fedora Update System 2008-09-10 06:54:20 UTC
NetworkManager-0.7.0-0.11.svn4022.fc8, NetworkManager-openvpn-0.7.0-15.svn4027.fc8, NetworkManager-vpnc-0.7.0-0.10.svn4024.fc8, NetworkManager-pptp-0.7.0-0.10.svn4027.fc8 has been pushed to the Fedora 8 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update NetworkManager NetworkManager-openvpn NetworkManager-vpnc NetworkManager-pptp'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F8/FEDORA-2008-7484

Comment 16 Fedora Update System 2008-09-28 18:42:33 UTC
NetworkManager-0.7.0-0.11.svn4022.fc8, NetworkManager-openvpn-0.7.0-15.svn4027.fc8, NetworkManager-vpnc-0.7.0-0.10.svn4024.fc8, NetworkManager-pptp-0.7.0-0.10.svn4027.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2008-09-28 18:45:33 UTC
NetworkManager-pptp-0.7.0-0.10.svn4027.fc9, NetworkManager-openvpn-0.7.0-15.svn4027.fc9, NetworkManager-vpnc-0.7.0-0.10.svn4024.fc9, NetworkManager-0.7.0-0.11.svn4022.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.