Bug 877704 - Linux Bridge NAT doesn't work due to net.bridge.bridge-nf-call-iptables=1
Summary: Linux Bridge NAT doesn't work due to net.bridge.bridge-nf-call-iptables=1
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-quantum
Version: 2.0 (Folsom)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: snapshot3
: 2.1
Assignee: Terry Wilson
QA Contact: Ofer Blaut
URL:
Whiteboard:
: 877703 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-11-18 06:20 UTC by Etsuji Nakai
Modified: 2016-04-26 22:15 UTC (History)
3 users (show)

Fixed In Version: 2012.2.1-7.el6ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-03-21 19:03:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Add sysctl modules script for linuxbridge (2.89 KB, patch)
2013-02-01 00:27 UTC, Terry Wilson
apevec: review+
Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:0672 0 normal SHIPPED_LIVE Red Hat OpenStack 2.0 (Folsom) Preview bug fix and enhancement update 2013-03-21 23:02:46 UTC

Description Etsuji Nakai 2012-11-18 06:20:23 UTC
Description of problem:

When using Linux Bridge plugin, NAT function of the virtual router doesn't work, probably, due to "net.bridge.bridge-nf-call-iptables=1".

I'm not sure where this is set. My dirty workaround is to place the following code in /etc/rc.local.

-----------
rc=1
while [[ $rc -ne 0 ]]; do
	sysctl net.bridge.bridge-nf-call-iptables | grep "1"
	rc=$?
	sleep 5
done
sysctl -w net.bridge.bridge-nf-call-iptables=0
sysctl -w net.bridge.bridge-nf-call-ip6tables=0
sysctl -w net.bridge.bridge-nf-call-arptables=0
-----------

My setup is:
# rpm -qa | grep quantum
openstack-quantum-linuxbridge-2012.2-2.1.el6.noarch
python-quantumclient-2.1.1-0.el6.noarch
python-quantum-2012.2-2.1.el6.noarch
openstack-quantum-2012.2-2.1.el6.noarch

Comment 1 Etsuji Nakai 2012-11-18 06:30:40 UTC
My guess about the reason of this problem is, in the iptables's nat chain, before packets to be SNAT-ed reach the "quantum-l3-agent-snat" chain, it is ACCEPT-ed at the "quantum-l3-agent-POSTROUTING"'s following rule (if net.bridge.bridge-nf-call-iptables=1).

Chain quantum-l3-agent-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   600 ACCEPT     all  --  !qg-18ac7fca-0a !qg-18ac7fca-0a  0.0.0.0/0            0.0.0.0/0           ! ctstate DNAT 



The relevant parts of the nat tables is as below.
--------------
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   10   600 quantum-l3-agent-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 quantum-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 nova-compute-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 nova-api-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 nova-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 
Chain quantum-l3-agent-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   600 ACCEPT     all  --  !qg-18ac7fca-0a !qg-18ac7fca-0a  0.0.0.0/0            0.0.0.0/0           ! ctstate DNAT 

Chain quantum-postrouting-bottom (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 quantum-l3-agent-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0 

Chain quantum-l3-agent-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 quantum-l3-agent-float-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 SNAT       all  --  *      *       192.168.101.0/24     0.0.0.0/0           to:10.64.201.1 
-----------

Comment 3 Alan Pevec 2012-11-19 11:30:43 UTC
This is set in default sysctl.conf 
# grep bridge-nf /etc/sysctl.conf 
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# rpm -qf /etc/sysctl.conf
initscripts-9.03.31-2.el6.x86_64

So this shouldn't be an issue on default RHEL install, how do you install your machine?

Comment 4 Alan Pevec 2012-11-19 12:25:04 UTC
*** Bug 877703 has been marked as a duplicate of this bug. ***

Comment 5 Etsuji Nakai 2012-11-19 14:09:23 UTC
I'm using the default RHEL6.3 installation. However, at the boot time of RHEL6.3, because the bridge kernel module is not loaded, net.bridge.bridge-nf-call-* in sysctl.conf is simply ignored as below.

# sysctl -p
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key

I suspect that once Quantum L2 agent has started, it loads the bridge module and set net.bridge.bridge-nf-call-* = 1. I'm not sure why the agent requires them to be 1.

Comment 6 Etsuji Nakai 2012-11-19 14:17:50 UTC
Or probably, when the bridge module is loaded by the L2 agent, net.bridge.bridge-nf-call-* becomes 1 by default. If so, the L2 agent should explicitly set net.bridge.bridge-nf-call-* = 0 at that timing.

Comment 7 Etsuji Nakai 2012-11-19 14:38:59 UTC
I confirmed my guess by putting the following block in /etc/rc.local. (This is called after sysctl.conf is applied, and befer L2 agent starts.)

------
modprobe bridge
sysctl -w net.bridge.bridge-nf-call-iptables=0
sysctl -w net.bridge.bridge-nf-call-ip6tables=0
sysctl -w net.bridge.bridge-nf-call-arptables=0
------

And the result is:

# sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0

So the problem is that net.bridge.bridge-nf-call-* in sysctl.conf is ignored if the "brdige" module is not loaded at boot time.

Comment 8 Alan Pevec 2012-11-19 18:33:11 UTC
> So the problem is that net.bridge.bridge-nf-call-* in sysctl.conf is ignored if the "brdige" module is not loaded at boot time.

Indeed and I see the same on F17 too.
Packaging workaround in openstack-quantum-linuxbridge could be to include /etc/sysconfig/modules/openstack-quantum-linuxbridge.modules script which modprobes bridge.

Comment 9 Etsuji Nakai 2012-11-19 23:21:11 UTC
I see. The same workaround worked well for me, thanks.

# cat /etc/sysconfig/modules/openstack-quantum-linuxbridge.modules 
#!/bin/sh

modprobe -b bridge >/dev/null 2>&1

exit 0

Comment 10 Paul Robert Marino 2012-12-18 17:37:27 UTC
why not add a "/sbin/sysctl -p" to the end of init script for quantum? or even better quantum should be setting theses options based on need along with others.

Comment 11 Etsuji Nakai 2013-01-04 02:29:11 UTC
By the way, I confirmed that this affects only when netns support is disabled. 

When netns support is enabled, nat table is placed under a different namespace which doesn't contain the bridge interface inside it. So the value of net.bridge.bridge-nf-call-iptables doesn't affect the rule matching.

The bottom line is the requirement for net.bridge.bridge-nf-call-iptables could be different based on the configuration to be used by the customer. So hardcoding the parameter in Quantum's source code may be a little bit risky.

I think the best way to deal with it is:

- Use the package workaround of /etc/sysconfig/modules/openstack-quantum-linuxbridge.modules.
- Describe in the Deployment Guide what parameters in /etc/sysctl.conf should be set according to the intended configuration.

Comment 12 Terry Wilson 2013-02-01 00:27:23 UTC
Created attachment 691296 [details]
Add sysctl modules script for linuxbridge

apevec: does this patch look like a good start?

Comment 15 Ofer Blaut 2013-03-19 10:18:43 UTC
I have tested floating ip on quantum with linux bridge , it works

also setting is as expected using RHEL 6.4

[root@puma34 ~(keystone_admin_tenant1)]$ lsmod | grep bridge
bridge                 79078  0 
stp                     2218  2 garp,bridge
llc                     5546  3 garp,bridge,stp
[root@puma34 ~(keystone_admin_tenant1)]$ sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
[root@puma34 ~(keystone_admin_tenant1)]$ rpm -qa | grep quantum
openstack-quantum-2012.2.3-8.el6ost.noarch

Comment 17 errata-xmlrpc 2013-03-21 19:03:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0672.html


Note You need to log in before you can comment on or make changes to this bug.