Bug 846810

Cross-filed cases 00690348 and 00690349 on the Red Hat Customer portal.

A really nasty workaround that is doing it's job for now is:

$ cat /etc/libvirt/hooks/qemu 
#!/bin/bash

if [ "$2" = "prepare" ]; then
  /sbin/iptables -D INPUT -i virbr0 -p udp --dport 53 -j ACCEPT || :
  /sbin/iptables -D INPUT -i virbr0 -p tcp --dport 53 -j ACCEPT || :
  /sbin/iptables -D INPUT -i virbr0 -p udp --dport 67 -j ACCEPT || :
  /sbin/iptables -D INPUT -i virbr0 -p tcp --dport 67 -j ACCEPT || :
  /sbin/iptables -D FORWARD -o virbr0 -d 192.168.0.32/28 -m state \
    --state RELATED,ESTABLISHED -j ACCEPT || :
  /sbin/iptables -D FORWARD -i virbr0 -s 192.168.0.32/28 -j ACCEPT || :
  /sbin/iptables -D FORWARD -i virbr0 -o virbr0 -j ACCEPT || :
  /sbin/iptables -D FORWARD -o virbr0 -j REJECT || :
  /sbin/iptables -D FORWARD -i virbr0 -j REJECT || :
  /sbin/iptables -t nat -D POSTROUTING -s 192.168.0.32/28 \
    ! -d 192.168.0.32/28 -p tcp -j MASQUERADE --to-ports 1024-65535 || :
  /sbin/iptables -t nat -D POSTROUTING -s 192.168.0.32/28 \
    ! -d 192.168.0.32/28 -p udp -j MASQUERADE --to-ports 1024-65535 || :
  /sbin/iptables -t nat -D POSTROUTING -s 192.168.0.32/28 \
    ! -d 192.168.0.32/28 -j MASQUERADE || :
fi
$

What does it do? It simply kicks the rules that just have been added a
few moments before.

Why not using a bridge setup on my own without libvirt involvement? This
is IMHO not why libvirt exists and would work around it. There is libvirt
and thus it should also allow appropriate configuration to get compliant
setups.

Note that it isn't practical to disable libvirt's iptables rules when forward mode='nat'. The entire purpose of forward mode='nat' is to add the iptables rules that perform nat.

What *is* reasonable is a mode for isolated (no <forward> specified) and forward mode='route' networks that suppresses the iptables rules, and we can keep this BZ open to track a potential patch to do that.

In the cases of isolated and routed networks, the rules are added in order to 1) allow traffic between guests on the given network, 2) prevent (isolated) or permit (routed) traffic between the given network and the outside, and 3) permit inbound dhcp and dns traffic to the instance of dnsmasq run on the host by libvirt.

In the end, a libvirt virtual network really is 1) a linux host bridge + 2) an IP address for that bridge, 3) an instance of dnsmasq specifically tailored to serve this network, + 4) a set of iptables rules to give proper access suitable for the higher level "mode" that is configured. (Note that only steps (3) and (4) have any mentionable complexity - steps (1) and (2) are a total of 10 lines or less in /etc/sysconfig/network-=scripts/ifcfg-*). The less of these 4 pieces of functionality you use, the more attractive it becomes to simply create a bridge yourself in the standard host network configuration, and define a libvirt "unmanaged" network to reference it, e.g:

  <network>
    <name>host-bridge</name>
    <forward mode='bridge'/>
    <bridge name='mybr0'/>
  </network>

In this case, libvirt won't create the bridge, won't run dnsmasq, and won't setup any iptables rules - it will simply connected guests on the host-bridge network to the pre-existing device "mybr0".


Finally, I should point out that one of the biggest problems with iptables rules is the part about one management application stepping all over another management application's rules. The problem is that there is no central controlling authority, and not even an agreed upon method of mitigating the problem (for example, if each rule had a priority, different applications could at least agree amongst themselves on the relative priority one application should have over another, and probably avoid most (if not all) of the problems. Instead, the reality is that every single management application simply inserts all of their rules at the beginning of the chain, because that's really the only practical way of reasonably assuring success).

The true solution to the problem of "no central controlling authority" isn't to just decide that one of the management applications will be the only one to add/remove rules, as that would be a management nightmare. The true solution is for the central controlling authority to be some lower level "broker" that takes requests to add/remove rules (possibly using a higher level language than that offered by iptables) and make sure that certain rules are followed to assure harmony. This is the objective of the firewalld project, which is currently included in Fedora 17, and will hopefully be turned on by default in Fedora 18. There are already patches submitted upstream that modify libvirt to use firewalld when it is available, rather than directly going to iptables.

a) I don't want to create the bridge on my own
b) I don't want to have any iptables/route rules inserted
c) I don't care about what maybe reaches Fedora or not, this is RHEL
d) If you treat libvirt for the Enterprise, make it behaving like this

Which exactly did you not get in my initial description?

The point is simply: It is always told that libvirt is *the* defacto standard
once it comes to an abstraction layer for virtualization. And now you suggest
to work around by having a bridge that is not managed by libvirt? Haha...

I have the same problem. I want to configure my iptables, but the rules is compiled into the libvirt driver. Maybe it needs to use a configuration file instead of hard-coded rules?!!

This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.

Patch posted upstream:

https://www.redhat.com/archives/libvir-list/2016-August/msg00640.html

Pushed upstream:

commit 25e8112d7c32ab271b9cae28f3ccbf5835206693

Author: Laine Stump <laine>
Date:   Wed Aug 10 19:09:55 2016 -0400

    network: new network forward mode 'open'

Test on libvirt-2.0.0-8.el7.x86_64, the results as expected.

1. specify a dev, will report error
# cat open1
<network>
  <name>open</name>
  <forward mode='open' dev='enp0s25'/>
<ip address='192.168.200.1' netmask='255.255.255.0'/>
</network>
# virsh net-define open1
error: Failed to define network from open1
error: XML error: forward dev not allowed for network 'open' with forward mode='open'

2. define and start a network with forward mode='open'
# cat open.xml
<network>
  <name>open</name>
  <uuid>81ff0d90-c91e-6742-64da-4a736edb9a9b</uuid>
  <bridge name="virbr8"/>
  <mac address='12:34:56:78:9A:BC'/>
  <forward mode="open"/>
  <ip address="192.168.200.1" netmask="255.255.255.0">
  <dhcp>
      <range start='192.168.200.2' end='192.168.200.254'/>
    </dhcp>
  </ip>
 <ip family='ipv6' address='2001:db8:ca2:2::1' prefix='64'>
    <dhcp>
      <range start='2001:db8:ca2:2::10' end='2001:db8:ca2:2::ff'/>
    </dhcp>
  </ip>
</network>
# virsh net-define open.xml
Network open defined from open.xml
# virsh net-start open
Network open started

3. after the network started, check no libvirt added rules for this network
# iptables -L -nv | grep virbr8   =====> virbr8 is the bridge which the network connected.
# service libvirtd restart
# iptables -L -nv | grep virbr8 ====> check again, no iptable rules add

4. start a guest connected to this network
on the guest
# dhclient 
# dhclient -6
# ifconfig
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.200.213  netmask 255.255.255.0  broadcast 192.168.200.255
        inet6 2001:db8:ca2:2::a3  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::5054:ff:fef6:14f3  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:f6:14:f3  txqueuelen 1000  (Ethernet)
        RX packets 12  bytes 1636 (1.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28  bytes 3824 (3.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
# ping 192.168.200.1
PING 192.168.200.1 (192.168.200.1) 56(84) bytes of data.
64 bytes from 192.168.200.1: icmp_seq=1 ttl=64 time=0.296 ms
64 bytes from 192.168.200.1: icmp_seq=2 ttl=64 time=0.267 ms

--- 192.168.200.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.267/0.281/0.296/0.022 ms
# ping6 2001:db8:ca2:2::1
PING 2001:db8:ca2:2::1(2001:db8:ca2:2::1) 56 data bytes
64 bytes from 2001:db8:ca2:2::1: icmp_seq=1 ttl=64 time=0.483 ms
64 bytes from 2001:db8:ca2:2::1: icmp_seq=2 ttl=64 time=0.239 ms

5. do save---> restore, the network on the guest works well as before.
6. migrate back and forth, it will be succeed.
7. hotplug works ok.
# virsh attach-interface fulls network open --persistent --liveInterface attached successfully

Created attachment 1200060 [details]
/var/log/messages on the host

Hi laine,

I found on libvirt-2.0.0-8.el7.x86_64, the guest with open type interface can not get ip address, Please refer to the attachment on comment 42. 


on the guest:
# dhclient ens8 -d
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/ens8/52:54:00:4d:2e:d9
Sending on   LPF/ens8/52:54:00:4d:2e:d9
Sending on   Socket/fallback
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 5 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 14 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 7 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 11 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 17 (xid=0x49cb6a61)
DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 7 (xid=0x49cb6a61)
No DHCPOFFERS received.
No working leases in persistent database - sleeping.

# virsh net-dumpxml open1
<network connections='1'>
  <name>open1</name>
  <uuid>fa912360-3560-4df2-be7f-76c3b5b96626</uuid>
  <forward mode='open'/>
  <bridge name='virbr8' stp='on' delay='0'/>
  <mac address='12:34:56:78:9a:bc'/>
  <ip address='192.168.200.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.200.2' end='192.168.200.254'/>
    </dhcp>
  </ip>
  <ip family='ipv6' address='2001:db8:ca2:2::1' prefix='64'>
    <dhcp>
      <range start='2001:db8:ca2:2::10' end='2001:db8:ca2:2::ff'/>
    </dhcp>
  </ip>
</network>

Yes, that's a very likely possibility. With forward mode='open', it's the responsibility of the admin to open in the host firewall *any* port that is required for proper guest network operation. If the host doesn't allow incoming dhcp requests, then dhcp isn't going to work. If you want dhcp to work on the guest, you need to make sure your host's firewall allows it. That's the whole point of this mode.

Many thanks to laine.
when the host firewalld service is active, the guest with open interface can not get ip as comment 43. After stop the firewalld service on the host, the guest can get dhcp ip address and ping the host as comment 41.
Move the bug to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html