Bug 846810 - Automagically iptables rules added by libvirt can't be avoided/disabled
Automagically iptables rules added by libvirt can't be avoided/disabled
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
All Linux
unspecified Severity high
: rc
: ---
Assigned To: Laine Stump
Virtualization Bugs
Yehuda Zimmerman
:
Depends On:
Blocks: 1113520 1203710 1305606 1313485 1137052 1288337
  Show dependency treegraph
 
Reported: 2012-08-08 14:57 EDT by Robert Scheck
Modified: 2016-11-03 14:04 EDT (History)
17 users (show)

See Also:
Fixed In Version: libvirt-2.0.0-8.el7
Doc Type: Release Note
Doc Text:
*libvirt* iptables rules can be manually managed for a specified network *libvirt* automatically generates and applies iptables rules appropriate for each type of network it creates. The rules are controlled by `forward mode` in the configuration of each network. Previously, there was no way for users to disable these automatically generated iptables rules and manually manage the iptables rules. In the current release, the `open` network `forward mode` was added. When specified for a network, *libvirt* does not generate any iptables rules for the network. As a result, iptables rules added outside the scope of *libvirt* are not disrupted and users can manually manage iptables rules.
Story Points: ---
Clone Of:
: 1137052 (view as bug list)
Environment:
Last Closed: 2016-11-03 14:04:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/messages on the host (14.16 KB, text/plain)
2016-09-11 23:33 EDT, yalzhang@redhat.com
no flags Details

  None (edit)
Description Robert Scheck 2012-08-08 14:57:27 EDT
Description of problem:
When using either the standard "default.xml" or the following one to create
a NAT network for virtual machines,

$ cat /etc/libvirt/qemu/networks/default.xml 
<network>
  <name>default</name>
  <uuid>7b53c807-fb93-87b9-67f2-06604dbfe3e6</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='01:23:45:67:89:AB'/>
  <ip address='192.168.0.33' netmask='255.255.255.240'>
  </ip>
</network>
$

some iptables rules are inserted into the regular chains. But unfortunately 
these libvirt-added iptables rules can not be avoided or disabled at all. At
least there is no documentation pointer and the suggestion by some libvirt
contributors on the Internet seems to be to use simply to configure a bridge
setup outside of libvirt only.

Having libvirt inserting random non-configurable iptables rules into some of
the chains disables maybe another firewall framework (like in our case) or it
just conflicts with it, say if the firewall framework resets all rules to set
the afterwards again, so the libvirt-added iptables rules are just gone.

Indeed, I might even add the same rules like libvirt is doing, but this is
not really necessarily the case. It also is a question of compliance if any
random daemon inserts iptables rules into the firewall settings. Specifically
in my today case, I did not want all of the enduser friendly default rules!

From what I got, there was already an attempt by Felix Schwarz to get this
broken behaviour solved, however I am unfortunately unable to see that this
patch has been accepted by you, Red Hat as the libvirt upstream.

I thought, libvirt was also developed for the Enterprise business, but when
I see such a feature really missing, it feels more like Enterprise, sorry...

Version-Release number of selected component (if applicable):
libvirt-0.9.10-21.el6_3.3.x86_64

How reproducible:
Everytime, see above and below.

Actual results:
Automagically iptables rules added by libvirt can't be avoided/disabled.

Expected results:
It can be configured that libvirt does not add it's own iptables rules to
the regular iptables chains, however the user has then to configure all of
the maybe required iptables rules completely himself on it's own.

Additional info:
- http://www.redhat.com/archives/libvir-list/2009-December/msg00631.html
- http://www.redhat.com/archives/libvir-list/2009-December/msg00632.html
- http://www.redhat.com/archives/libvir-list/2009-December/msg00637.html
Comment 2 Robert Scheck 2012-08-08 15:03:15 EDT
Cross-filed cases 00690348 and 00690349 on the Red Hat Customer portal.
Comment 3 Robert Scheck 2012-08-08 15:06:13 EDT
A really nasty workaround that is doing it's job for now is:

$ cat /etc/libvirt/hooks/qemu 
#!/bin/bash

if [ "$2" = "prepare" ]; then
  /sbin/iptables -D INPUT -i virbr0 -p udp --dport 53 -j ACCEPT || :
  /sbin/iptables -D INPUT -i virbr0 -p tcp --dport 53 -j ACCEPT || :
  /sbin/iptables -D INPUT -i virbr0 -p udp --dport 67 -j ACCEPT || :
  /sbin/iptables -D INPUT -i virbr0 -p tcp --dport 67 -j ACCEPT || :
  /sbin/iptables -D FORWARD -o virbr0 -d 192.168.0.32/28 -m state \
    --state RELATED,ESTABLISHED -j ACCEPT || :
  /sbin/iptables -D FORWARD -i virbr0 -s 192.168.0.32/28 -j ACCEPT || :
  /sbin/iptables -D FORWARD -i virbr0 -o virbr0 -j ACCEPT || :
  /sbin/iptables -D FORWARD -o virbr0 -j REJECT || :
  /sbin/iptables -D FORWARD -i virbr0 -j REJECT || :
  /sbin/iptables -t nat -D POSTROUTING -s 192.168.0.32/28 \
    ! -d 192.168.0.32/28 -p tcp -j MASQUERADE --to-ports 1024-65535 || :
  /sbin/iptables -t nat -D POSTROUTING -s 192.168.0.32/28 \
    ! -d 192.168.0.32/28 -p udp -j MASQUERADE --to-ports 1024-65535 || :
  /sbin/iptables -t nat -D POSTROUTING -s 192.168.0.32/28 \
    ! -d 192.168.0.32/28 -j MASQUERADE || :
fi
$

What does it do? It simply kicks the rules that just have been added a
few moments before.

Why not using a bridge setup on my own without libvirt involvement? This
is IMHO not why libvirt exists and would work around it. There is libvirt
and thus it should also allow appropriate configuration to get compliant
setups.
Comment 6 Laine Stump 2012-08-14 13:40:41 EDT
Note that it isn't practical to disable libvirt's iptables rules when forward mode='nat'. The entire purpose of forward mode='nat' is to add the iptables rules that perform nat.

What *is* reasonable is a mode for isolated (no <forward> specified) and forward mode='route' networks that suppresses the iptables rules, and we can keep this BZ open to track a potential patch to do that.

In the cases of isolated and routed networks, the rules are added in order to 1) allow traffic between guests on the given network, 2) prevent (isolated) or permit (routed) traffic between the given network and the outside, and 3) permit inbound dhcp and dns traffic to the instance of dnsmasq run on the host by libvirt.

In the end, a libvirt virtual network really is 1) a linux host bridge + 2) an IP address for that bridge, 3) an instance of dnsmasq specifically tailored to serve this network, + 4) a set of iptables rules to give proper access suitable for the higher level "mode" that is configured. (Note that only steps (3) and (4) have any mentionable complexity - steps (1) and (2) are a total of 10 lines or less in /etc/sysconfig/network-=scripts/ifcfg-*). The less of these 4 pieces of functionality you use, the more attractive it becomes to simply create a bridge yourself in the standard host network configuration, and define a libvirt "unmanaged" network to reference it, e.g:

  <network>
    <name>host-bridge</name>
    <forward mode='bridge'/>
    <bridge name='mybr0'/>
  </network>

In this case, libvirt won't create the bridge, won't run dnsmasq, and won't setup any iptables rules - it will simply connected guests on the host-bridge network to the pre-existing device "mybr0".


Finally, I should point out that one of the biggest problems with iptables rules is the part about one management application stepping all over another management application's rules. The problem is that there is no central controlling authority, and not even an agreed upon method of mitigating the problem (for example, if each rule had a priority, different applications could at least agree amongst themselves on the relative priority one application should have over another, and probably avoid most (if not all) of the problems. Instead, the reality is that every single management application simply inserts all of their rules at the beginning of the chain, because that's really the only practical way of reasonably assuring success).

The true solution to the problem of "no central controlling authority" isn't to just decide that one of the management applications will be the only one to add/remove rules, as that would be a management nightmare. The true solution is for the central controlling authority to be some lower level "broker" that takes requests to add/remove rules (possibly using a higher level language than that offered by iptables) and make sure that certain rules are followed to assure harmony. This is the objective of the firewalld project, which is currently included in Fedora 17, and will hopefully be turned on by default in Fedora 18. There are already patches submitted upstream that modify libvirt to use firewalld when it is available, rather than directly going to iptables.
Comment 7 Robert Scheck 2012-08-14 15:13:08 EDT
a) I don't want to create the bridge on my own
b) I don't want to have any iptables/route rules inserted
c) I don't care about what maybe reaches Fedora or not, this is RHEL
d) If you treat libvirt for the Enterprise, make it behaving like this

Which exactly did you not get in my initial description?
Comment 8 Robert Scheck 2012-08-14 15:18:02 EDT
The point is simply: It is always told that libvirt is *the* defacto standard
once it comes to an abstraction layer for virtualization. And now you suggest
to work around by having a bridge that is not managed by libvirt? Haha...
Comment 18 ITD27M01 2013-03-12 04:15:43 EDT
I have the same problem. I want to configure my iptables, but the rules is compiled into the libvirt driver. Maybe it needs to use a configuration file instead of hard-coded rules?!!
Comment 24 Jiri Denemark 2014-04-04 17:36:40 EDT
This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.
Comment 32 Laine Stump 2016-08-14 00:38:23 EDT
Patch posted upstream:

https://www.redhat.com/archives/libvir-list/2016-August/msg00640.html
Comment 33 Laine Stump 2016-08-19 21:41:27 EDT
Pushed upstream:

commit 25e8112d7c32ab271b9cae28f3ccbf5835206693

Author: Laine Stump <laine@laine.org>
Date:   Wed Aug 10 19:09:55 2016 -0400

    network: new network forward mode 'open'
Comment 41 yalzhang@redhat.com 2016-09-08 06:51:32 EDT
Test on libvirt-2.0.0-8.el7.x86_64, the results as expected.

1. specify a dev, will report error
# cat open1
<network>
  <name>open</name>
  <forward mode='open' dev='enp0s25'/>
<ip address='192.168.200.1' netmask='255.255.255.0'/>
</network>
# virsh net-define open1
error: Failed to define network from open1
error: XML error: forward dev not allowed for network 'open' with forward mode='open'

2. define and start a network with forward mode='open'
# cat open.xml
<network>
  <name>open</name>
  <uuid>81ff0d90-c91e-6742-64da-4a736edb9a9b</uuid>
  <bridge name="virbr8"/>
  <mac address='12:34:56:78:9A:BC'/>
  <forward mode="open"/>
  <ip address="192.168.200.1" netmask="255.255.255.0">
  <dhcp>
      <range start='192.168.200.2' end='192.168.200.254'/>
    </dhcp>
  </ip>
 <ip family='ipv6' address='2001:db8:ca2:2::1' prefix='64'>
    <dhcp>
      <range start='2001:db8:ca2:2::10' end='2001:db8:ca2:2::ff'/>
    </dhcp>
  </ip>
</network>
# virsh net-define open.xml
Network open defined from open.xml
# virsh net-start open
Network open started

3. after the network started, check no libvirt added rules for this network
# iptables -L -nv | grep virbr8   =====> virbr8 is the bridge which the network connected.
# service libvirtd restart
# iptables -L -nv | grep virbr8 ====> check again, no iptable rules add

4. start a guest connected to this network
on the guest
# dhclient 
# dhclient -6
# ifconfig
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.200.213  netmask 255.255.255.0  broadcast 192.168.200.255
        inet6 2001:db8:ca2:2::a3  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::5054:ff:fef6:14f3  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:f6:14:f3  txqueuelen 1000  (Ethernet)
        RX packets 12  bytes 1636 (1.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28  bytes 3824 (3.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
# ping 192.168.200.1
PING 192.168.200.1 (192.168.200.1) 56(84) bytes of data.
64 bytes from 192.168.200.1: icmp_seq=1 ttl=64 time=0.296 ms
64 bytes from 192.168.200.1: icmp_seq=2 ttl=64 time=0.267 ms

--- 192.168.200.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.267/0.281/0.296/0.022 ms
# ping6 2001:db8:ca2:2::1
PING 2001:db8:ca2:2::1(2001:db8:ca2:2::1) 56 data bytes
64 bytes from 2001:db8:ca2:2::1: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from 2001:db8:ca2:2::1: icmp_seq=2 ttl=64 time=0.239 ms

5. do save---> restore, the network on the guest works well as before.
6. migrate back and forth, it will be succeed.
7. hotplug works ok.
# virsh attach-interface fulls network open --persistent --liveInterface attached successfully
Comment 42 yalzhang@redhat.com 2016-09-11 23:33 EDT
Created attachment 1200060 [details]
/var/log/messages on the host
Comment 43 yalzhang@redhat.com 2016-09-11 23:37:09 EDT
Hi laine,

I found on libvirt-2.0.0-8.el7.x86_64, the guest with open type interface can not get ip address, Please refer to the attachment on comment 42. 


on the guest:
# dhclient ens8 -d
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/ens8/52:54:00:4d:2e:d9
Sending on   LPF/ens8/52:54:00:4d:2e:d9
Sending on   Socket/fallback
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 5 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 14 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 7 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 11 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 17 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 7 (xid=0x49cb6a61)
No DHCPOFFERS received.
No working leases in persistent database - sleeping.

# virsh net-dumpxml open1
<network connections='1'>
  <name>open1</name>
  <uuid>fa912360-3560-4df2-be7f-76c3b5b96626</uuid>
  <forward mode='open'/>
  <bridge name='virbr8' stp='on' delay='0'/>
  <mac address='12:34:56:78:9a:bc'/>
  <ip address='192.168.200.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.200.2' end='192.168.200.254'/>
    </dhcp>
  </ip>
  <ip family='ipv6' address='2001:db8:ca2:2::1' prefix='64'>
    <dhcp>
      <range start='2001:db8:ca2:2::10' end='2001:db8:ca2:2::ff'/>
    </dhcp>
  </ip>
</network>
Comment 44 Laine Stump 2016-09-12 08:01:31 EDT
Yes, that's a very likely possibility. With forward mode='open', it's the responsibility of the admin to open in the host firewall *any* port that is required for proper guest network operation. If the host doesn't allow incoming dhcp requests, then dhcp isn't going to work. If you want dhcp to work on the guest, you need to make sure your host's firewall allows it. That's the whole point of this mode.
Comment 45 yalzhang@redhat.com 2016-09-12 21:31:34 EDT
Many thanks to laine.
when the host firewalld service is active, the guest with open interface can not get ip as comment 43. After stop the firewalld service on the host, the guest can get dhcp ip address and ping the host as comment 41.
Move the bug to verified.
Comment 47 errata-xmlrpc 2016-11-03 14:04:59 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html

Note You need to log in before you can comment on or make changes to this bug.