Bug 227011

Summary: RFE: Allow several sets of independently managed iptables rules
Product: [Fedora] Fedora Reporter: Mark McLoughlin <markmc>
Component: system-config-firewallAssignee: Thomas Woerner <twoerner>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: abartlet, andy, apevec, berrange, bill-bugzilla.redhat.com, bloch, donhoover, fschwarz, hbrock, jbainbri, jbastian, k.georgiou, mcepl, mcepl, mfuruta, purpleidea, rjones, rmusil, sanabria.d, scottt.tw, zing
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-06 19:33:55 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 458625    
Attachments:
Description Flags
iptables-init-script-separate-rules-config.patch none

Description Mark McLoughlin 2007-02-02 16:31:18 UTC
Currently in Fedora, there is no way for two seperate programs to maintain
different sets of iptables rules without trampling on one another.

As a case in point, we are adding "virtual networks" support to libvirt:

  http://www.gnome.org/~markmc/virtual-networking.html

As part of this, libvirt needs to be able to add iptables rules of its own:

----
    For each network we need the following iptables rules:

    1) Allow frames to be forwarded from the bridge interface to other
       bridge ports:

         -I FORWARD -m physdev --physdev-in $(iface) -j ACCEPT

    2) Allow IP packets from the bridge interface to be forwarded to
       other network interfaces:

         -I FORWARD -i $(iface) -j ACCEPT

    3) Masquerade any outgoing packets, with the exception of those
       frames which are just being forwarded across a bridge:

         -t nat -I POSTROUTING -m physdev ! --physdev-is-bridged -j MASQUERADE

    4) Allow DHCP and DNS requests in from the bridge interface so
       that dnsmasq can handle them:

         -I INPUT -i vnet0 -p tcp --dport 67 -j ACCEPT
         -I INPUT -i vnet0 -p tcp --dport 53 -j ACCEPT
         -I INPUT -i vnet0 -p udp --dport 53 -j ACCEPT

       Of course, we also need to enable IP forwarding via 
       /proc/sys/net/ipv4/ip_forward

       Also, for every TAP interface we enslave to a network's bridge, we
       must allow frames to be forwarded from that interface to other
       bridge ports as in (1).

    
http://www.gnome.org/~markmc/code/libvirt-networking/libvirt-network-iptables.patch
----

The problem, of couse, is that if someone does "service iptables restart" or
modifies the firewall config with system-config-securitylevel, all these rules
gets wiped out.

Attaching a patch which allows multiple programs to configure rules which will
be added by the iptables init script.

Basically, programs can put a list of rules for a chain in
/etc/iptables.d/filter or /etc/iptables.d/nat and the initscript will then
create that chain.

Also, /etc/sysconfig/iptables-config will have a list of chains which should be
added to e.g. the filter/INPUT or nat/PREROUTING chains, thus allowing the user
to configure whether or not a specific programs rules should be included.

The patch also handles details like retaining the past behaviour of
/etc/sysconfig/iptables for e.g. upgrades and not saving these chains when you
do "service iptables save" etc.

Work that still needs doing:

  1) Implement the --ignore parameter to iptables-save

  2) Make s-c-sl (or s-c-fw) output its config to /etc/iptables.d and remove
     /etc/sysconfig/iptables

  3) Add $IPTABLES_FILTER_INPUT_CHAINS etc. to iptables-config

  4) Add iptables.d/(nat|filter)

  5) Make sure this works for ip6tables

Comment 1 Mark McLoughlin 2007-02-02 16:31:18 UTC
Created attachment 147236 [details]
iptables-init-script-separate-rules-config.patch

Comment 2 Mark McLoughlin 2007-02-26 16:20:41 UTC
See what went into libvirt:

   http://www.redhat.com/archives/libvir-list/2007-February/msg00056.html

We can now build libvirt with:

  $> ./configure --with-iptables-prefix=RH-libvirt- 
                 --with-iptables-dir=/etc/init.d/iptables.d

in order to integrate with the iptables init script patch

Comment 3 Mark McLoughlin 2007-03-09 14:01:11 UTC
Oh, the default /etc/sysconfig/iptables would need to contain:

IPTABLES_FILTER_INPUT_CHAINS="RH-libvirt-INPUT RH-Firewall-1-INPUT"
IPTABLES_FILTER_FORWARD_CHAINS="RH-libvirt-FORWARD RH-Firewall-1-FORWARD"
IPTABLES_NAT_PREROUTING_CHAINS="RH-libvirt-PREROUTING"
IPTABLES_NAT_POSTROUTING_CHAINS="RH-libvirt-POSTROUTING"

Comment 4 Mark McLoughlin 2007-03-09 17:21:14 UTC
Just to re-iterate some points on this:

  - only chains listed in /etc/sysconfig/iptables are added from iptables.d

  - we need a --ignore parameter to iptables-save so that "service iptables
    save" doesn't save any of the chains that were loaded from iptables.d

  - we'll need to change s-c-fw to save its rules to iptables.d and delete
    /etc/sysconfig/iptables.

  - "service iptables save" now saves to /etc/sysconfig/iptables-saved

  - the initscript *either* loads /etc/sysconfig/iptables, if it exists,
    or first the chains from iptables.d and then the contents of
    /etc/sysconfig/iptables-saved.  This provides us with a sane migration
    path.

  - the order in which chains from iptables.d are loaded is specified in 
    iptables-config

  - all this is intended to address the problem of that all libvirt can do
    right now is insert (i.e. -I) it's rules to the INPUT etc. chains and
    those rules get summarily wiped out by "service iptables reload" and
    saved by "service iptables save"

  - one bug I see in the patch is that load_iptables() should load 
    /etc/sysconfig/iptables-saved"

Comment 5 Thomas Woerner 2007-08-23 14:23:15 UTC
How can the user configure, what each each virtual network can access locally
and in other networks (even external) and if the network is accessible from
other networks or from extern?

system-config-firewall in devel provides the ability to load custom rules.
Please have a look at "lokkit --help". You can add a file with your rules by
using "lokkit --custom-rules=ipv4:filter:/etc/sysconfig/my_rules_file".
lokkit also got an new option "--update" which regenerates the firewall rules
for ipv4 and ipv6. Use this if you are udating custom rules and want that the
changes are persistent.
The enhancements in lokkit does not require a change in the firewall handling
for iptables; it does not know anything about the includes. This solution is
transparent in use especially for updaters.

Please have a look at this solution and report problems if you have any.
Thanks

Comment 6 Thomas Woerner 2007-10-04 10:18:04 UTC
Assigning to system-config-firewall.

Comment 7 Mark McLoughlin 2008-01-04 16:14:31 UTC
twoerner: thanks, --custom-rules works just fine for us. See the libvirt patch
I've just posted:

  http://www.redhat.com/archives/libvir-list/2008-January/msg00025.html

Comment 8 David Lutterkort 2008-06-04 16:55:48 UTC
The approach taken here is not sufficient. I had edited /etc/sysconfig/iptables
manually, and an update of libvirt clobbered those manual changes. The change I
made was adding a rule to forward packets across a bridged device (to make DHCP
work across the bridge) The change was made before lokkit (or
system-config-firewall) was ever used.

lokkit should not clobber /etc/sysconfig/iptables if the file has been edited
manually. Barring that, it needs to back the original iptables file up.

Generally, the approach of turning what used to be the authoritative config file
(/etc/sysconfig/iptables) into a generated file is fatally flawed exactly
because it clobbers manual edits.

Comment 9 Mark McLoughlin 2008-06-04 17:10:59 UTC
lokkit --custom-rules behavior is fine - if a user explicitly chooses to  modify
your firewall configuration, then overwriting an existing
/etc/sysconfig/iptables is expected behaviour.

However, lokkit --custom-rules is not an appropriate integration point for libvirt.

Merely by *installing the libvirt package* a default virtual network is set up
which involves adding rules to allow traffic to be forwarded across a bridge and
NAT-ed to external physical networks.

It is not acceptable that installing libvirt should overwrite a users custom
iptables configuration.

This goes back to the original intention of this bug and the patch in comment #1.

What libvirt needs is a way for it to have its rules loaded by the iptables
initscript irrespective of whether the user has manually configured iptables or
used system-config-firewall

Comment 10 Mark McLoughlin 2008-06-04 17:16:16 UTC
See also bug #449996

Comment 11 Thomas Woerner 2008-06-04 18:09:52 UTC
See bug #449996#c4

Comment 12 Mark McLoughlin 2008-06-04 20:57:41 UTC
(In reply to comment #11)
> See bug #449996#c4

quoting:

    Use "lokkit --disabled" and lokkit will not touch the firewall 
    configuration files anymore. This is and was the way lokkit behaves 
    since some years.

So ... we should recommend to users that if they have custom rules in
/etc/sysconfig/iptables, they should either

 a) Add those custom rules to lokkit --with-custom-rules or

 b) Disable lokkit with lokkit --disabled, which will mean that libvirt's
    rules will be clobbered if they do "service iptables restart"

and they should choose which they want to do *before* installing libvirt.


This clearly is not a workable solution - users cannot know to make this
decision before installing libvirt and if they don't make this decision, users
with a working setup will find their setup broken by the mere act of installing
(or updating to a more recent) libvirt


lokkit is a tool for users to modify the firewall config, and libvirt should not
be clobbering user's configuration - I'm really failing to see how lokkit can be
made to be a suitable integration point for system software like libvirt

Hence the /etc/sysconfig/iptables.d idea so that libvirt can integrate in a way
that doesn't clobber user configuration

Comment 13 Daniel Berrangé 2008-06-04 21:03:56 UTC
Being able to just drop a file into /etc/sysconfig/iptables.d would also allow
us to have a more tightly controlled SELinux policy - libvirtd would merely need
to be able to drop files into that location, and not require ability to execute
lokkit with the extra privileges this entails.

Comment 14 Thomas Woerner 2008-06-05 08:45:50 UTC
Comment #12:
No they should choose before making custom /etc/sysconfig/ip*tables files.

The order or rules is very important, how do you want to make sure that your
custom rules or the ones from libvirtd or SElinux are placed into the right
position, especially if a user is adding files with own rules, too? 

"service iptables save" would not be usable with this anymore. This mechanism is
used in our training courses and would also mean a change in behavior for EL.

Comment #13: What will be more tight with /etc/sysconfig/iptables.d than it is
with current custom-rules option of lokkit? libvirt would need the ability to
restart the iptables and ip6tables services, so where is the difference?

Comment 15 Thomas Woerner 2008-06-05 09:32:42 UTC
This is the email I have sent to David about his question of parsing and
modifying ip*tables configurations:

---
I thought a lot about that in the past, but there are some problems with using
user-customized configurations:

1) There are iptables and ip6tables, the options for them are not the same,
there are even some major differences, like missing modules, other module
options, other arguments etc. Also there is no NAT for netfilter-v6. You have to
be able to modify both accordingly.

2) You could configure ip*tables in two different ways:
- Default policy: accept, reject/drop not wanted
- Default policy: deny, open up wanted
Both are in use out there.

3) The configuration system needs to understand every possible module
configuration and combination of modules and options to insert the changes into
the proper line or even worse into the proper chain. Please remember that you
can create your own chains, that could branch to other chains or return back.

4) netfilter and netfilter-v6 are moving targets. With every new kernel version
there are new modules and sometimes also removed or renamed modules or options
and also behaviour changes. Having a parser for iptables, which is able to parse
and also generate a valid configuration for every netfilter version is hard to
do. Having a parser for ip*tables will be lots of work to have it in sync with
netfilter in the various kernel versions.

5) Modifying the firewall configuration in a way that it breaks or that it is
opening the firewall up by mistake will be a major problem. This could happen,
if you put a rule into the wrong line. iptables-restore aborts on error.
Ignoring errors if you apply a new configuration could also result in a useless
firewall, either opened up or closed.

6) Limiting your desired configuration system to only a limited part of the
netfilter capabilities (modules, options, arguments, ..) is not an option.
Therefore the capabilities of this configuration system has to be identical to
the netfilter and netfilter-v6 capabilities in the kernel. In every kernel
version we ship for a release or even for all releases. Please also think of
updates: we often push a new kernel (with newer netfilter) into all current
Fedora releases.


I have had a parser, that was able to load nearly every netfilter configuration
some years ago, but the problem was that I had to build a netfilter-a'like
system to be able to get a system which could understand the configuration. Now,
this is a no-go in my opinion, because netfilter is moving too fast.

I thought about not building the firewall configuration files at all anymore.
This will be the best solution, I think. If the files are there, they are used
as is and the firewall is user-customized. If the files are missing (both of
them) then the firewall is generated dynamically. The problem with this is that
we are using the "service iptables save" command in our RH courses and this will
break it. This is also a change in behavior, which will be hard for EL.
---

Comment 16 Mark McLoughlin 2008-06-05 10:06:54 UTC
(In reply to comment #14)
> Comment #12:
> No they should choose before making custom /etc/sysconfig/ip*tables files.

That event is in the past. If people have a working setup involving custom
/etc/sysconfig/iptables, then we shouldn't break it.

> The order or rules is very important, how do you want to make sure that your
> custom rules or the ones from libvirtd or SElinux are placed into the right
> position, especially if a user is adding files with own rules, too? 

That is an issue irrespective of whether we have /etc/sysconfig/iptables.d or
lokkit --with-custom-rules

The way I look at it is that the default policy is REJECT and most everyone is
adding ACCEPT rules, so conflicts don't matter. That's a huge generalization,
but anyone adding rules would need to be mindful of the possibility of
conflicts. It's a shared resource with multiple users, I don't see how you can
avoid the issue.

> "service iptables save" would not be usable with this anymore.

Why not?

Do you mean that if you do "service iptables save" that all the rules form
/etc/sysconfig/iptables.d would be saved into /etc/sysconfig/iptables and you'd
have two copies of the rules? That's the issue I want to fix by adding
iptables-save --ignore

Or are you referring to something else ...?

I'm confident issues like this could be worked out. The proposed patch is merely
a proof-of-concept. If you agreed in principle that /etc/sysconfig/iptables.d
might be acceptable, we could work to resolve those issues.

> Comment #13: What will be more tight with /etc/sysconfig/iptables.d than it is
> with current custom-rules option of lokkit? 

It would not involve libvirt overwriting a users custom rules in
/etc/sysconfig/iptables

> libvirt would need the ability to
> restart the iptables and ip6tables services, so where is the difference?

No, libvirt would not need the ability to restart iptables. It adds the rules
itself *and* writes out a file to /etc/sysconfig/iptables.d

Comment 17 Mark McLoughlin 2008-06-05 10:11:11 UTC
(In reply to comment #15)
> This is the email I have sent to David about his question of parsing and
> modifying ip*tables configurations:

Which question? It sounds like this might have been related to augeas? I'm not
sure this is related to the question of how libvirt should integrate correctly
with iptables

(Are you conflating two issues just because the same people are invovled or ...?
Am I just missing some context here?)

Comment 18 Mark McLoughlin 2008-06-05 10:20:10 UTC
(In reply to comment #16)
> (In reply to comment #14)
> > The order or rules is very important, how do you want to make sure that your
> > custom rules or the ones from libvirtd or SElinux are placed into the right
> > position, especially if a user is adding files with own rules, too? 
> 
> That is an issue irrespective of whether we have /etc/sysconfig/iptables.d or
> lokkit --with-custom-rules
> 
> The way I look at it is that the default policy is REJECT and most everyone is
> adding ACCEPT rules, so conflicts don't matter. That's a huge generalization,
> but anyone adding rules would need to be mindful of the possibility of
> conflicts. It's a shared resource with multiple users, I don't see how you can
> avoid the issue.

Or to put it another way - libvirt needs to add these rules to offer the
functionality of Virtual Networks.

Currently we do add the rules, without calling out to lokkit, so there is
possibility of conflict between rules anyway.

libvirt using /etc/sysconfig/iptables.d would not increase the likelihood of a
conflict between rules.

If we conclude that the possibility of conflict means that we shouldn't be
adding rules, then we also must conclude that we cannot offer this functionality.

Comment 19 Thomas Woerner 2008-06-05 11:17:42 UTC
Two things first: 
1) We do not have one firewall, we have two: iptables for IPv4 and ip6tables for
IPv6. Please keep in mind, that we have to take care of both. 
2) lokkit is an abstraction layer for both firewalls. It takes care of adding
services etc. and there are no changes needed to iptables applications or the
behaviour of the init script at all.

Comment #16:
> That is an issue irrespective of whether we have /etc/sysconfig/iptables.d or
> lokkit --with-custom-rules
Correct, but lokkit is adding custom-rules at the end after the normal
configuration is done. Surely you could force to prepend rules with iptables
calls, but then you should know, what you do.

> The way I look at it is that the default policy is REJECT and most everyone is
> adding ACCEPT rules, so conflicts don't matter. That's a huge generalization,
> but anyone adding rules would need to be mindful of the possibility of
> conflicts. It's a shared resource with multiple users, I don't see how you can
> avoid the issue.
This is wrong. The default policy is accept for us, ports/rules to be opened up
are marked to be accepted, the remaining things are rejected with a special
reject type.

The order does matter. Please have a look at this configuration example:
  -A INPUT -t tcp -m tcp --dport 22 -s 10.32.0.0/24 -j DROP
  -A INPUT -t tcp -m tcp --dport 22 -j ACCEPT
First match first serve.

Which rules do you want to ignore with "iptables-save --ignore"? How do you want
to get the rules back into the right ordering when you apply the firewall again? 
In the example above the user inserted the rule "-A INPUT -t tcp -m tcp --dport
22 -s 10.32.0.0/24 -j DROP" by hand. It is not part of the normal configuration.
Then he uses "service iptables save". Where do you want to save this rule? Where
do you want to add it again? If you add it after the accept line, it is
completely useless.

Another thing that the user can do is to remove an existing rule from your
preconfigured configuration from /etc/sysconfig/ip*tables.d. If he uses the save
command, what would you save then?

> It would not involve libvirt overwriting a users custom rules in
> /etc/sysconfig/iptables
and
> No, libvirt would not need the ability to restart iptables. It adds the rules
> itself *and* writes out a file to /etc/sysconfig/iptables.d
I do not see, where the difference is: "It adds the rules
itself *and* writes out a file to /etc/sysconfig/iptables.d" and it adds the
rules itself and adds a custom file with using "lokkit --nostart
--custom-rules=XYZ.file". This is absolutely identical.

> If you agreed in principle that /etc/sysconfig/iptables.d might be acceptable, 
> we could work to resolve those issues.
Indeed, it would be nice to have a system like this, but how do you want to
resolve the problems? I have had a look at many configuration tools for
netfilter and netfilter-v6, and all limited the functionality in this area,
because ordering is the most important thing and we need to be able to save a
current configuration regardless if it matches preconfigured files or not.

Comment #17: I put it in the bugzilla, because there are some explanations in
the text. which I do not want to write again. Please also have a look at the
last section of comment #15.

Comment #18:
> If we conclude that the possibility of conflict means that we shouldn't be
> adding rules, then we also must conclude that we cannot offer this 
> functionality.
Correct, this is the problem with custom rules.

Comment 20 Andrew Bartlett 2008-08-11 07:19:16 UTC
This is more than just a libvirt problem.  NetworkManager's new connection sharing feature clobbers the libvirt rules.   See bug #458625.

Is there any way forward on this?  As both of the 'mess with iptables' contenders are actually after a very simple NAT networking setup, perhaps that could be accommodated more easily with a shared set of rules (that would step on fewer other toes?)

Comment 21 Andrew Bartlett 2008-08-12 04:30:28 UTC
Thinking about the 'NAT networks' rule idea, could we have a generic rule included in the default ruleset, that jumps to a 'consdier for NAT chain'.

Then programs such as libvirt and NetworkManager can simply add and remove rules that permit access to another 'NAT these packets' chain?  (being empty, would be harmless until needed)

That would seem to allow easy, deterministic modification of the NAT table, including for more traditional NAT setups (not using NetworkManger).

Thoughts?

Comment 22 Mark McLoughlin 2009-09-04 13:01:38 UTC
*** Bug 519439 has been marked as a duplicate of this bug. ***

Comment 23 Mark McLoughlin 2009-12-10 12:02:59 UTC
*** Bug 546064 has been marked as a duplicate of this bug. ***

Comment 25 Justin M. Forbes 2009-12-15 18:43:30 UTC
*** Bug 539744 has been marked as a duplicate of this bug. ***

Comment 26 Don Hoover 2010-11-02 15:29:27 UTC
I still like the idea of having a directory we can drop modular set of rules into.

We manage our systems centrally and push out configurations from a central repository.

Being able to push out say an /etc/sysconfig/iptables.d/kvm, /etc/sysconfig/iptables.d/ldapd, /etc/sysconfig/iptables.d/snmp etc..separately as needed depending on the functions that server is performing would be wonderfully intuitive for those of us that manage our systems in this way.

I would say that MOST systems are running as deny all, and opening ports/traffic as needed, ala using the firewall like tcpwrappers.

Comment 27 purpleidea 2011-03-20 04:08:50 UTC
FWIW, I'm a large user of the shorewall firewall [1] and I thought I'd mention that it's always good when things that play with iptables, play nicely with shorewall too.

My use case is that I use shorewall instead of the "iptables service" to manage iptables, and as such, when libvirt adds things to iptables, shorewall doesn't know they're added... Shorewall is quite "big" with servers, and this will be more relevant as more and more servers use kvm+libvirt, etc...

HTH

[1] http://www.shorewall.net/

Comment 28 Bill McGonigle 2011-03-21 01:39:51 UTC
Is this now FirewallD?

https://fedoraproject.org/wiki/FirewallD/

Comment 29 Thomas Woerner 2011-03-30 13:35:04 UTC
firewalld provides a dynamic firewall with a D-BUS interface. system-config-firewall on the other hand provides a static firewall.

With the D-BUS interface of firewalld it is possible to add firewall rules using the daemon. This way firewalld knows about the rules and can handle them for example in a reload or restart case. firewalld is also able to add and remove rules without the need to restart the whole firewall. This is very good for established connections.

For more information, please have a look at https://fedoraproject.org/wiki/FirewallD/

firewalld will not replace system-config-firewall, but it will be the default firewall solution in Fedora as soon as it is ready and stable. system-config-firewall will still be available for static firewall solutions, but it will be optional.

Comment 30 Daniel Berrangé 2011-03-31 15:20:51 UTC
While firewalld is a nice solution, AFAICT, the current impl has some limitations that prevent us using it, eg

http://lists.fedoraproject.org/pipermail/devel/2011-January/147613.html

Comment 31 Andrew Haveland-Robinson 2011-07-20 14:50:29 UTC
This problem has been driving us potty for years - is there no way to integrate this properly?

My main gripe is that libvirtd appears to INSERTS rules into the FORWARD chain:

# iptables -S FORWARD
-P FORWARD DROP
-A FORWARD -d 192.168.100.0/24 -i eth0 -o virbr1 -j ACCEPT
-A FORWARD -s 192.168.100.0/24 -i virbr1 -o eth0 -j ACCEPT
-A FORWARD -i virbr1 -o virbr1 -j ACCEPT
-A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable

and then icmp-port-unreachable rules clobber my following rules...

-A FORWARD -s 0.0.0.0/32 -d 224.0.0.1/32 -i virbr1 -j DROP
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.0.30/32 -d 192.168.0.7/32 -i tun+ -o eth0 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A FORWARD -s 192.168.0.7/32 -d 192.168.100.2/32 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A FORWARD -s 192.168.0.30/32 -d 192.168.100.2/32 -i tun+ -o virbr1 -p tcp -m tcp --dport 5280 -j ACCEPT
-A FORWARD -s 192.168.100.2/32 -d 192.168.0.30/32 -i virbr1 -o tun+ -p tcp -m tcp --sport 5280 -j ACCEPT
etc...

I would prefer that libvirtd doesn't meddle with iptables and let me manage it, although it would be nice if it just worked automatically as VMs are created/destroyed.

It would alleviate the problem a bit if 
FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
were appended to FORWARD instead of inserted.

I use /etc/init.d/iptables save for persistency. If libvirtd is active, then its rules get saved too, and then duplicated rules are added again when libvirtd boots. It needs to be more intelligent:

iptables is queryable using -S through grep - if the rule exists, then don't add the rule again. We can even work out how many times to remove a duplicated rule using tail -n +2:
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
...

# remove all but 1 duplicated rule:
IFS=$'\n'; for i in `iptables -t mangle -S POSTROUTING | grep ' -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill' | tail -n +2 | cut -b 4-`; do echo /sbin/iptables -t mangle -D $i > tmp.sh; bash tmp.sh; done; rm -f tmp.sh;

`/sbin/iptables -t mangle -D $i` should work but it doesn't - iptables thinks the whole command line is a too-long chain name and barfs, so have to work around it by squirting the commands to a shell script then executing that instead. :(

Append REJECT rules to the end of an existing FORWARD chain.
libvirtd could just advise on what rules to use, instead of blindly modifying.
Users should be able to manage iptables themselves instead of hacking hacks to undo the damage.

Comment 32 Andrew Haveland-Robinson 2011-07-20 19:41:06 UTC
Here's a fairly rough script I wrote to remove duplicate rules, and move icmp-port-unreachable rules to the end of the forward chain.
It's not perfect, but may help someone as a starting point.
Chances are that anyone using a FORWARD rule would already have this rule
FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
as the first one. This would forward all established connections anyway, so making libvirtd's per VM forward rules redundant.

#!/bin/bash

[[ -e tmp.sh ]] && rm -f tmp.sh;
[[ -e foundrules.txt ]] && rm -f foundrules.txt; touch foundrules.txt;

IFS=$'\n';
for i in `iptables -t mangle -S POSTROUTING | grep ' -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill' | tail -n +2 | cut -b 4-`; do
    echo /sbin/iptables -t mangle -D $i >> tmp.sh;
done;
[[ -e tmp.sh ]] && ( bash tmp.sh; rm -f tmp.sh; )

for rule in $(cat <<-!
FORWARD -d .* -o virbr.* -m state --state RELATED,ESTABLISHED -j ACCEPT
FORWARD -s .* -i virbr.* -j ACCEPT
FORWARD -i virbr.* -o virbr.* -j ACCEPT
!
); do
    `iptables -S FORWARD | egrep "$rule" | cut -b 4- | sort -m -u >> foundrules.txt`;
done;

for rule in $(cat foundrules.txt); do
    echo rule $rule;
    for i in `iptables -S FORWARD | grep "$rule" | tail -n +2 | cut -b 4-`; do
        echo /sbin/iptables -t filter -D $i >> tmp.sh;
    done;
done;

[[ -e foundrules.txt ]] && rm -f foundrules.txt
[[ -e tmp.sh ]] && ( bash tmp.sh; rm -f tmp.sh )

for i in `iptables -S FORWARD | grep 'FORWARD -. virbr. -j REJECT --reject-with icmp-port-unreachable' | cut -b 4-`; do
    echo /sbin/iptables -t filter -D $i >> tmp.sh;
    echo /sbin/iptables -t filter -A $i >> tosort.sh;
done;
[[ -e tmp.sh ]] && ( bash tmp.sh; rm -f tmp.sh; )
[[ -e tosort.sh ]] && ( cat tosort.sh | sort -u > tmp.sh; rm -f tosort.sh; )
[[ -e tmp.sh ]] && ( bash tmp.sh; rm -f tmp.sh; )

for rule in $(cat <<-!
INPUT -i virbr. -p udp -m udp --dport 53 -j ACCEPT
INPUT -i virbr. -p tcp -m tcp --dport 53 -j ACCEPT
INPUT -i virbr. -p udp -m udp --dport 67 -j ACCEPT
INPUT -i virbr. -p tcp -m tcp --dport 67 -j ACCEPT
!
); do
    for i in `iptables -S INPUT | grep " $rule" | cut -b 4-`; do
        echo /sbin/iptables -t filter -D $i >> tmp.sh;
        echo /sbin/iptables -t filter -I $i >> tosort.sh;
    done;
done;
[[ -e tmp.sh ]] && ( bash tmp.sh; rm -f tmp.sh; )
[[ -e tosort.sh ]] && ( cat tosort.sh | sort -m -u > tmp.sh; rm -f tosort.sh; )
[[ -e tmp.sh ]] && ( bash tmp.sh; rm -f tmp.sh; )

Comment 33 Jamie Bainbridge 2012-10-22 01:46:06 UTC
As per See Also private Bug 725094, this functionality is available in lokkit. Adding the closing comment from that bug, for public visibility.

> Lokkit as a frontend tool for our iptables configuration already provides
> such a mechanism, see lokkit --help:
> 
>   --custom-rules=[<type>:][<table>:]<filename>
>                         Specify a custom rules file for inclusion in the
>                         firewall, after the default rules. Default protocol
>                         type: ipv4, default table: filter. (Example:
>                         ipv4:filter:/etc/sysconfig/ipv4_filter_addon)
> 
> Example file: /usr/share/netcf/iptables-forward-bridged
> Content: -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT

Comment 34 Mark McLoughlin 2012-10-22 12:26:25 UTC
(In reply to comment #33)
> As per See Also private Bug 725094, this functionality is available in
> lokkit.

See 'lokkit --custom-rules is not an appropriate integration point for libvirt' in comment #9

Comment 35 Thomas Woerner 2013-11-06 19:33:55 UTC
Closing because there will not be big changes to system-config-firewall anymore.