Red Hat Bugzilla – Bug 212245
Routed networking for guests does not work
Last modified: 2008-08-02 19:40:33 EDT
Description of problem: When setting up a guest using Xen's routed networking scripts instead of bridging, no network traffic is ever routed off the host. The guest can talk to Dom0, and Dom0 can talk to the guest, but the guest can't talk to the outside world. There appear to be two problems breaking the outside world communication: - The iptables forwarding rule setup per-guest is placed *after* the rules from the standard RHEL firewall scripts. Since the final catch all rule from the RHEL firewall is 'REJECT', no guest <-> outside world traffic passes. - The proxy_arp option on the host's public network interface is not activated. Without this enabled, the guest can transmit traffic, but will never receive any replies because extrernal hosts won't be able to do an ARP lookup for the guest's IP. Version-Release number of selected component (if applicable): xen-3.0.3-2.el5 How reproducible: Always Steps to Reproduce: 1. Edit /etc/xend-config.sxp, changing networking to the following two settings: (network-script network-route) (vif-script vif-route) 2. Create a guest config file and config its VIF with an explicit IP address which is valid for the local LAN. eg, vif = [ 'mac=00:16:3e:1e:7c:a6, ip=10.13.4.220' ] 3. Boot the guest 4. Inside the guest, edit the eth0 configuration, giving it the same IP address specified in its config file, eg # cat /etc/sysconfig/network-scripts/ifcfg-eth0 # Xen Virtual Ethernet DEVICE=eth0 BOOTPROTO=static IPADDR=10.13.4.220 NETMASK=255.255.252.0 GATEWAY=10.13.4.221 HWADDR=00:16:3E:1E:7C:A6 ONBOOT=yes 5. Try and ping the outside world, eg google Actual results: No ping replies received Expected results: Ping replies normally Additional info: Here is the incorrect IPtable ruleset - notice the 'RH-Firewall-1-INPUT' rule coming first in the FORWARD chain - the guest rules need to be before this. # iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 10.13.4.220 0.0.0.0/0 PHYSDEV match --physdev-in vif15.0 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif15.0 udp spt:68 dpt:67 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain RH-Firewall-1-INPUT (2 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 255 ACCEPT esp -- 0.0.0.0/0 0.0.0.0/0 ACCEPT ah -- 0.0.0.0/0 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 224.0.0.251 udp dpt:5353 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:631 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:631 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Notice proxy arp is only set on the vif15.0 interface, not the eth0 # sysctl -a | grep proxy_arp net.ipv4.conf.vif15/0.proxy_arp = 1 net.ipv4.conf.eth0.proxy_arp = 0 net.ipv4.conf.lo.proxy_arp = 0 net.ipv4.conf.default.proxy_arp = 0 net.ipv4.conf.all.proxy_arp = 0
I think we need both the INPUT and FORWARD chains to reflect the new IP, don't we? So what is needed may be: in /etc/xen/scripts/vif-common.sh --- function frob_iptable() --- local c="-A" should become local c="-I" and iptables "$c" FORWARD -m physdev --physdev-in "$vif" "$@" -j ACCEPT \ should use "RH-Firewall-1-INPUT" instead of "FORWARD". Can you please test with those changes? Thanks.
You are correct that we need to have the rules in both INPUT & FORWARD chains to correctly get all traffic, because without the INPUT rule, the dom0 services firwall rules end up applying to the DomU too. Unfortunately we can't just change the script to use RH-Firewall-1-INPUT, because this breaks the firewalling for users who aren't using the system-config-securitylevel tool for iptables. Anyway, I tested the changes described above anyway, and discovered a further problem. The rules added *never* match. Reading the IPtables does this is because 'physdev' matches only apply to packets traversing a bridge device - it allows matching based on devices which are members of the bridge. So while this rule may work for the Xen network-bridge script, its useless for network-route script. Instead of '-m physdev --physdev-in $vif' we need to simply use '-i $vif'. In summary, the changes we need to make to get a routed network config working are: - Input the rules directly to FORWARD & INPUT chains in position 1 - Use '-i $vif' for matching devices - Set net.ipv4.conf.eth0.proxy_arp=1 or perhaps net.ipv4.conf.all.proxy_arp=1
Created attachment 139549 [details] Fix iptables for routed networking This patch attempts to fix the 3 issues listed above. It adds the iptables rules to INPUT chain as well as FORWARD chain. It adds the rules at the start instead of end of chain. It uses correct interface matching rules. It enables proxy_arp for all interfaces. I've tested it works with vif-route/network-route scripts. Need further testing to verify operation of other network scripts (eg bridging) has not been impacted negatively.
Patch sent upstream. No response... http://lists.xensource.com/archives/html/xen-devel/2006-11/msg00771.html
These scripts are highly distribution-specific. They also depend on whether firewall management software is in use (e.g., shorewall). In fact the rules for vif-route should simply be removed from upstream. The iptables rules for vif-brdige do not actually allow any packets to be received in dom0 or forwarded to peth0. All they do is allow the packets to traverse the bridge. Once it reaches the output device (be it dom0 or peth0) netfilter will process the packets from scratch without using any physdev matches. So I'm not sure whether this patch is really suitable for upstream inclusion. The upstream scripts are (as their location suggest) examples on a hypothetical system with no or minimal firewall rules. So I would suggest that we provide tailored alternate scripts for the default RHEL configuration. If we're concerned about having to maintain duplicate shell code then we should modify the upstream scripts in such a way that they can invoke user-specified scripts to perform system-specific tasks such as firewall configuration. Let me know what you guys think. Thanks.
Closing because libvirt provides a saner replacement for routed networking.