Bug 212245 - Routed networking for guests does not work
Routed networking for guests does not work
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: Herbert Xu
Depends On:
  Show dependency treegraph
Reported: 2006-10-25 15:52 EDT by Daniel Berrange
Modified: 2008-08-02 19:40 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-08-01 14:53:08 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Fix iptables for routed networking (2.61 KB, patch)
2006-10-26 22:27 EDT, Daniel Berrange
no flags Details | Diff

  None (edit)
Description Daniel Berrange 2006-10-25 15:52:15 EDT
Description of problem:
When setting up a guest using Xen's routed networking scripts instead of
bridging, no network traffic is ever routed off the host. The guest can talk to
Dom0, and Dom0 can talk to the guest, but the guest can't talk to the outside
world. There appear to be two problems breaking the outside world communication:

 - The iptables forwarding rule setup per-guest is placed *after* the rules from
the standard RHEL firewall scripts. Since the final catch all rule from the RHEL
firewall is 'REJECT', no guest <-> outside world traffic passes.
 - The proxy_arp  option on the host's public network interface is not
activated. Without this enabled, the guest can transmit traffic, but will never
receive any replies because extrernal hosts won't be able to do an ARP lookup
for the guest's IP.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Edit /etc/xend-config.sxp, changing networking to the following two settings:

  (network-script network-route)
  (vif-script     vif-route)

2. Create a guest config file and config its VIF with an explicit IP address
which is valid for the local LAN. eg, 

   vif = [ 'mac=00:16:3e:1e:7c:a6, ip=' ]

3. Boot the guest
4. Inside the guest, edit the eth0 configuration, giving it the same IP address
specified in its config file, eg

# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Xen Virtual Ethernet

5. Try and ping the outside world, eg google
Actual results:
No ping replies received

Expected results:
Ping replies normally

Additional info:

Here is the incorrect IPtable ruleset - notice the 'RH-Firewall-1-INPUT' rule
coming first in the FORWARD chain - the guest rules need to be before this.

# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --             

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --             
ACCEPT     all  --           PHYSDEV match
--physdev-in vif15.0 
ACCEPT     udp  --             PHYSDEV match
--physdev-in vif15.0 udp spt:68 dpt:67 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination         
ACCEPT     all  --             
ACCEPT     icmp --             icmp type 255 
ACCEPT     esp  --             
ACCEPT     ah   --             
ACCEPT     udp  --           udp dpt:5353 
ACCEPT     udp  --             udp dpt:631 
ACCEPT     tcp  --             tcp dpt:631 
ACCEPT     all  --             state
ACCEPT     tcp  --             state NEW tcp dpt:22 
REJECT     all  --             reject-with

Notice proxy arp is only set on the vif15.0  interface, not the eth0

# sysctl -a | grep proxy_arp
net.ipv4.conf.vif15/0.proxy_arp = 1
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.proxy_arp = 0
Comment 1 Stephen Tweedie 2006-10-26 07:56:18 EDT
I think we need both the INPUT and FORWARD chains to reflect the new IP, don't we?  

So what is needed may be:

in /etc/xen/scripts/vif-common.sh ---
  function frob_iptable() ---

    local c="-A"
should become
    local c="-I"


  iptables "$c" FORWARD -m physdev --physdev-in "$vif" "$@" -j ACCEPT \

should use "RH-Firewall-1-INPUT" instead of "FORWARD".

Can you please test with those changes?  Thanks.
Comment 2 Daniel Berrange 2006-10-26 10:46:54 EDT
You are correct that we need to have the rules in both INPUT & FORWARD chains to
correctly get all traffic, because without the INPUT rule, the dom0 services
firwall rules end up applying to the DomU too.

Unfortunately we can't just change the script to use RH-Firewall-1-INPUT,
because this breaks the firewalling for users who aren't using the
system-config-securitylevel tool for iptables.

Anyway, I tested the changes described above anyway, and discovered a further
problem. The rules added *never* match. Reading the IPtables does this is
because 'physdev' matches only apply to packets traversing a bridge device - it
allows matching based on devices which are members of the bridge. So while this
rule may work for the Xen network-bridge script, its useless for network-route
script.  Instead of '-m physdev --physdev-in $vif' we need to simply use '-i $vif'.

In summary, the changes we need to make to get a routed network config working are:

 - Input the rules directly to FORWARD & INPUT chains in position 1
 - Use '-i $vif' for matching devices
 - Set net.ipv4.conf.eth0.proxy_arp=1   or perhaps net.ipv4.conf.all.proxy_arp=1
Comment 3 Daniel Berrange 2006-10-26 22:27:00 EDT
Created attachment 139549 [details]
Fix iptables for routed networking

This patch attempts to fix the 3 issues listed above. It adds the iptables
rules to INPUT chain as well as FORWARD chain. It adds the rules at the start
instead of end of chain. It uses correct interface matching rules. It enables
proxy_arp for all interfaces.

I've tested it works with vif-route/network-route scripts. Need further testing
to verify operation of other network scripts (eg bridging) has not been
impacted negatively.
Comment 4 Daniel Berrange 2006-11-20 15:57:04 EST
Patch sent upstream. No response...

Comment 5 Herbert Xu 2006-11-23 21:59:02 EST
These scripts are highly distribution-specific.  They also depend on whether
firewall management software is in use (e.g., shorewall).  In fact the rules for
vif-route should simply be removed from upstream.  The iptables rules for
vif-brdige do not actually allow any packets to be received in dom0 or forwarded
to peth0.  All they do is allow the packets to traverse the bridge.  Once it
reaches the output device (be it dom0 or peth0) netfilter will process the
packets from scratch without using any physdev matches.  So I'm not sure whether
this patch is really suitable for upstream inclusion.

The upstream scripts are (as their location suggest) examples on a hypothetical
system with no or minimal firewall rules.

So I would suggest that we provide tailored alternate scripts for the default
RHEL configuration.

If we're concerned about having to maintain duplicate shell code then we should
modify the upstream scripts in such a way that they can invoke user-specified
scripts to perform system-specific tasks such as firewall configuration.

Let me know what you guys think.  Thanks.
Comment 6 Daniel Berrange 2007-08-01 14:53:08 EDT
Closing because libvirt provides a saner replacement for routed networking.

Note You need to log in before you can comment on or make changes to this bug.