Bug 889868

Summary:	dnsmasq DHCP request blocked by default firewall rules
Product:	Red Hat OpenStack	Reporter:	Chris Wright <chrisw>
Component:	openstack-nova	Assignee:	Gary Kotton <gkotton>
Status:	CLOSED ERRATA	QA Contact:	Ofer Blaut <oblaut>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.0 (Folsom)	CC:	apevec, gkotton, jhenner, ndipanov, pbrady, rkukura, twilson, ykaul
Target Milestone:	snapshot4	Keywords:	Triaged
Target Release:	2.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-nova-2012.2.3-4.el6ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-03-21 18:14:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	892339

Description Chris Wright 2012-12-23 17:40:35 UTC

In some cases, guests are unable to obtain a DHCP lease.  The issue is caused by default host firewall rules:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

The default INPUT rule to reject is triggering on the DHCP request and is therefore never seen by dnsmasq.  Simple workaround is to disable that rule, however that has negative security implications.  This has been reproduced with both OVS and LinuxBridge.

Comment 2 Ofer Blaut 2012-12-26 07:18:38 UTC

Issue apears on RHEL 6.4 beta with nova based firewall rules

the workaround i have used is to disable firewall rule in /etc/nova/nova/conf  & reboot 

#firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
firewall = nova.virt.firewall.NoopFirewallDriver

The Nova rules in my host:

instances i have used 10.10.1.5 & 10.10.1.4

iptables-save 
# Generated by iptables-save v1.4.7 on Wed Dec 26 10:50:19 2012
*nat
:PREROUTING ACCEPT [445522:23017910]
:POSTROUTING ACCEPT [8697:625989]
:OUTPUT ACCEPT [10947:1407803]
:nova-api-OUTPUT - [0:0]
:nova-api-POSTROUTING - [0:0]
:nova-api-PREROUTING - [0:0]
:nova-api-float-snat - [0:0]
:nova-api-snat - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-POSTROUTING - [0:0]
:nova-compute-PREROUTING - [0:0]
:nova-compute-float-snat - [0:0]
:nova-compute-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-compute-PREROUTING 
-A PREROUTING -j nova-api-PREROUTING 
-A POSTROUTING -j nova-compute-POSTROUTING 
-A POSTROUTING -j nova-api-POSTROUTING 
-A POSTROUTING -j nova-postrouting-bottom 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE 
-A OUTPUT -j nova-compute-OUTPUT 
-A OUTPUT -j nova-api-OUTPUT 
-A nova-api-snat -j nova-api-float-snat 
-A nova-compute-snat -j nova-compute-float-snat 
-A nova-postrouting-bottom -j nova-compute-snat 
-A nova-postrouting-bottom -j nova-api-snat 
COMMIT
# Completed on Wed Dec 26 10:50:19 2012
# Generated by iptables-save v1.4.7 on Wed Dec 26 10:50:19 2012
*mangle
:PREROUTING ACCEPT [701113:107906116]
:INPUT ACCEPT [258278:85280778]
:FORWARD ACCEPT [9740:3055008]
:OUTPUT ACCEPT [209363:102737571]
:POSTROUTING ACCEPT [219446:105071947]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill 
COMMIT
# Completed on Wed Dec 26 10:50:19 2012
# Generated by iptables-save v1.4.7 on Wed Dec 26 10:50:19 2012
*filter
:INPUT ACCEPT [258219:85261900]
:FORWARD ACCEPT [9740:3055008]
:OUTPUT ACCEPT [206968:101910933]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-compute-FORWARD - [0:0]
:nova-compute-INPUT - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-inst-10 - [0:0]
:nova-compute-inst-9 - [0:0]
:nova-compute-local - [0:0]
:nova-compute-provider - [0:0]
:nova-compute-sg-fallback - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j nova-compute-INPUT 
-A INPUT -j nova-api-INPUT 
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
-A FORWARD -j nova-filter-top 
-A FORWARD -j nova-compute-FORWARD 
-A FORWARD -j nova-api-FORWARD 
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT 
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A OUTPUT -j nova-filter-top 
-A OUTPUT -j nova-compute-OUTPUT 
-A OUTPUT -j nova-api-OUTPUT 
-A nova-api-INPUT -d 10.35.101.10/32 -p tcp -m tcp --dport 8775 -j ACCEPT 
-A nova-compute-inst-10 -m state --state INVALID -j DROP 
-A nova-compute-inst-10 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A nova-compute-inst-10 -j nova-compute-provider 
-A nova-compute-inst-10 -s 10.10.1.2/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT 
-A nova-compute-inst-10 -s 10.10.1.0/24 -j ACCEPT 
-A nova-compute-inst-10 -p icmp -j ACCEPT 
-A nova-compute-inst-10 -j nova-compute-sg-fallback 
-A nova-compute-inst-9 -m state --state INVALID -j DROP 
-A nova-compute-inst-9 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A nova-compute-inst-9 -j nova-compute-provider 
-A nova-compute-inst-9 -s 10.10.1.2/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT 
-A nova-compute-inst-9 -s 10.10.1.0/24 -j ACCEPT 
-A nova-compute-inst-9 -p icmp -j ACCEPT 
-A nova-compute-inst-9 -j nova-compute-sg-fallback 
-A nova-compute-local -d 10.10.1.5/32 -j nova-compute-inst-10 
-A nova-compute-local -d 10.10.1.4/32 -j nova-compute-inst-9 
-A nova-compute-sg-fallback -j DROP

Comment 3 Bob Kukura 2013-02-11 16:38:05 UTC

I've found the following work-around to allow the DHCP lease to succeed:

$ sudo cp /etc/sysconfig/iptables .
$ sudo chmod 666 iptables
edit iptables
 add the following to INPUT chain before final REJECT:
  -A INPUT -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
 add the following to FORWARD chain before final REJECT:
  -A FORWARD -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
$ sudo cp iptables /etc/sysconfig/
$ sudo iptables-restore < iptables


There probably belong in nova-compute-INPUT and nova-compute-FORWARD instead of the main INPUT and FORWARD chains. We also need to determine whether either or both of these rules are needed on a quantum network node where quantum-dhcp-agent runs in addition to or instead of on the compute nodes.

Comment 14 Ofer Blaut 2013-03-06 12:07:45 UTC

Tested on 
openstack-quantum-2012.2.3-5.el6ost.noarch

Comment 16 errata-xmlrpc 2013-03-21 18:14:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0657.html