Bug 889868

Summary: dnsmasq DHCP request blocked by default firewall rules
Product: Red Hat OpenStack Reporter: Chris Wright <chrisw>
Component: openstack-novaAssignee: Gary Kotton <gkotton>
Status: CLOSED ERRATA QA Contact: Ofer Blaut <oblaut>
Severity: high Docs Contact:
Priority: high    
Version: 2.0 (Folsom)CC: apevec, gkotton, jhenner, ndipanov, pbrady, rbryant, rkukura, twilson, ykaul
Target Milestone: snapshot4Keywords: Triaged
Target Release: 2.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-2012.2.3-4.el6ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-21 14:14:22 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 892339    

Description Chris Wright 2012-12-23 12:40:35 EST
In some cases, guests are unable to obtain a DHCP lease.  The issue is caused by default host firewall rules:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

The default INPUT rule to reject is triggering on the DHCP request and is therefore never seen by dnsmasq.  Simple workaround is to disable that rule, however that has negative security implications.  This has been reproduced with both OVS and LinuxBridge.
Comment 2 Ofer Blaut 2012-12-26 02:18:38 EST
Issue apears on RHEL 6.4 beta with nova based firewall rules

the workaround i have used is to disable firewall rule in /etc/nova/nova/conf  & reboot 

#firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
firewall = nova.virt.firewall.NoopFirewallDriver

The Nova rules in my host:

instances i have used 10.10.1.5 & 10.10.1.4

iptables-save 
# Generated by iptables-save v1.4.7 on Wed Dec 26 10:50:19 2012
*nat
:PREROUTING ACCEPT [445522:23017910]
:POSTROUTING ACCEPT [8697:625989]
:OUTPUT ACCEPT [10947:1407803]
:nova-api-OUTPUT - [0:0]
:nova-api-POSTROUTING - [0:0]
:nova-api-PREROUTING - [0:0]
:nova-api-float-snat - [0:0]
:nova-api-snat - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-POSTROUTING - [0:0]
:nova-compute-PREROUTING - [0:0]
:nova-compute-float-snat - [0:0]
:nova-compute-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-compute-PREROUTING 
-A PREROUTING -j nova-api-PREROUTING 
-A POSTROUTING -j nova-compute-POSTROUTING 
-A POSTROUTING -j nova-api-POSTROUTING 
-A POSTROUTING -j nova-postrouting-bottom 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE 
-A OUTPUT -j nova-compute-OUTPUT 
-A OUTPUT -j nova-api-OUTPUT 
-A nova-api-snat -j nova-api-float-snat 
-A nova-compute-snat -j nova-compute-float-snat 
-A nova-postrouting-bottom -j nova-compute-snat 
-A nova-postrouting-bottom -j nova-api-snat 
COMMIT
# Completed on Wed Dec 26 10:50:19 2012
# Generated by iptables-save v1.4.7 on Wed Dec 26 10:50:19 2012
*mangle
:PREROUTING ACCEPT [701113:107906116]
:INPUT ACCEPT [258278:85280778]
:FORWARD ACCEPT [9740:3055008]
:OUTPUT ACCEPT [209363:102737571]
:POSTROUTING ACCEPT [219446:105071947]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill 
COMMIT
# Completed on Wed Dec 26 10:50:19 2012
# Generated by iptables-save v1.4.7 on Wed Dec 26 10:50:19 2012
*filter
:INPUT ACCEPT [258219:85261900]
:FORWARD ACCEPT [9740:3055008]
:OUTPUT ACCEPT [206968:101910933]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-compute-FORWARD - [0:0]
:nova-compute-INPUT - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-inst-10 - [0:0]
:nova-compute-inst-9 - [0:0]
:nova-compute-local - [0:0]
:nova-compute-provider - [0:0]
:nova-compute-sg-fallback - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j nova-compute-INPUT 
-A INPUT -j nova-api-INPUT 
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
-A FORWARD -j nova-filter-top 
-A FORWARD -j nova-compute-FORWARD 
-A FORWARD -j nova-api-FORWARD 
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT 
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A OUTPUT -j nova-filter-top 
-A OUTPUT -j nova-compute-OUTPUT 
-A OUTPUT -j nova-api-OUTPUT 
-A nova-api-INPUT -d 10.35.101.10/32 -p tcp -m tcp --dport 8775 -j ACCEPT 
-A nova-compute-inst-10 -m state --state INVALID -j DROP 
-A nova-compute-inst-10 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A nova-compute-inst-10 -j nova-compute-provider 
-A nova-compute-inst-10 -s 10.10.1.2/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT 
-A nova-compute-inst-10 -s 10.10.1.0/24 -j ACCEPT 
-A nova-compute-inst-10 -p icmp -j ACCEPT 
-A nova-compute-inst-10 -j nova-compute-sg-fallback 
-A nova-compute-inst-9 -m state --state INVALID -j DROP 
-A nova-compute-inst-9 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A nova-compute-inst-9 -j nova-compute-provider 
-A nova-compute-inst-9 -s 10.10.1.2/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT 
-A nova-compute-inst-9 -s 10.10.1.0/24 -j ACCEPT 
-A nova-compute-inst-9 -p icmp -j ACCEPT 
-A nova-compute-inst-9 -j nova-compute-sg-fallback 
-A nova-compute-local -d 10.10.1.5/32 -j nova-compute-inst-10 
-A nova-compute-local -d 10.10.1.4/32 -j nova-compute-inst-9 
-A nova-compute-sg-fallback -j DROP
Comment 3 Bob Kukura 2013-02-11 11:38:05 EST
I've found the following work-around to allow the DHCP lease to succeed:

$ sudo cp /etc/sysconfig/iptables .
$ sudo chmod 666 iptables
edit iptables
 add the following to INPUT chain before final REJECT:
  -A INPUT -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
 add the following to FORWARD chain before final REJECT:
  -A FORWARD -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
$ sudo cp iptables /etc/sysconfig/
$ sudo iptables-restore < iptables


There probably belong in nova-compute-INPUT and nova-compute-FORWARD instead of the main INPUT and FORWARD chains. We also need to determine whether either or both of these rules are needed on a quantum network node where quantum-dhcp-agent runs in addition to or instead of on the compute nodes.
Comment 14 Ofer Blaut 2013-03-06 07:07:45 EST
Tested on 
openstack-quantum-2012.2.3-5.el6ost.noarch
Comment 16 errata-xmlrpc 2013-03-21 14:14:22 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0657.html