Bug 877704 - Linux Bridge NAT doesn't work due to net.bridge.bridge-nf-call-iptables=1
Linux Bridge NAT doesn't work due to net.bridge.bridge-nf-call-iptables=1
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-quantum (Show other bugs)
2.0 (Folsom)
Unspecified Unspecified
medium Severity medium
: snapshot3
: 2.1
Assigned To: Terry Wilson
Ofer Blaut
: Triaged
: 877703 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-11-18 01:20 EST by Etsuji Nakai
Modified: 2016-04-26 18:15 EDT (History)
3 users (show)

See Also:
Fixed In Version: 2012.2.1-7.el6ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-21 15:03:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Add sysctl modules script for linuxbridge (2.89 KB, patch)
2013-01-31 19:27 EST, Terry Wilson
apevec: review+
Details | Diff

  None (edit)
Description Etsuji Nakai 2012-11-18 01:20:23 EST
Description of problem:

When using Linux Bridge plugin, NAT function of the virtual router doesn't work, probably, due to "net.bridge.bridge-nf-call-iptables=1".

I'm not sure where this is set. My dirty workaround is to place the following code in /etc/rc.local.

-----------
rc=1
while [[ $rc -ne 0 ]]; do
	sysctl net.bridge.bridge-nf-call-iptables | grep "1"
	rc=$?
	sleep 5
done
sysctl -w net.bridge.bridge-nf-call-iptables=0
sysctl -w net.bridge.bridge-nf-call-ip6tables=0
sysctl -w net.bridge.bridge-nf-call-arptables=0
-----------

My setup is:
# rpm -qa | grep quantum
openstack-quantum-linuxbridge-2012.2-2.1.el6.noarch
python-quantumclient-2.1.1-0.el6.noarch
python-quantum-2012.2-2.1.el6.noarch
openstack-quantum-2012.2-2.1.el6.noarch
Comment 1 Etsuji Nakai 2012-11-18 01:30:40 EST
My guess about the reason of this problem is, in the iptables's nat chain, before packets to be SNAT-ed reach the "quantum-l3-agent-snat" chain, it is ACCEPT-ed at the "quantum-l3-agent-POSTROUTING"'s following rule (if net.bridge.bridge-nf-call-iptables=1).

Chain quantum-l3-agent-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   600 ACCEPT     all  --  !qg-18ac7fca-0a !qg-18ac7fca-0a  0.0.0.0/0            0.0.0.0/0           ! ctstate DNAT 



The relevant parts of the nat tables is as below.
--------------
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   10   600 quantum-l3-agent-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 quantum-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 nova-compute-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 nova-api-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 nova-postrouting-bottom  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 
Chain quantum-l3-agent-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   600 ACCEPT     all  --  !qg-18ac7fca-0a !qg-18ac7fca-0a  0.0.0.0/0            0.0.0.0/0           ! ctstate DNAT 

Chain quantum-postrouting-bottom (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 quantum-l3-agent-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0 

Chain quantum-l3-agent-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 quantum-l3-agent-float-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 SNAT       all  --  *      *       192.168.101.0/24     0.0.0.0/0           to:10.64.201.1 
-----------
Comment 3 Alan Pevec 2012-11-19 06:30:43 EST
This is set in default sysctl.conf 
# grep bridge-nf /etc/sysctl.conf 
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# rpm -qf /etc/sysctl.conf
initscripts-9.03.31-2.el6.x86_64

So this shouldn't be an issue on default RHEL install, how do you install your machine?
Comment 4 Alan Pevec 2012-11-19 07:25:04 EST
*** Bug 877703 has been marked as a duplicate of this bug. ***
Comment 5 Etsuji Nakai 2012-11-19 09:09:23 EST
I'm using the default RHEL6.3 installation. However, at the boot time of RHEL6.3, because the bridge kernel module is not loaded, net.bridge.bridge-nf-call-* in sysctl.conf is simply ignored as below.

# sysctl -p
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key

I suspect that once Quantum L2 agent has started, it loads the bridge module and set net.bridge.bridge-nf-call-* = 1. I'm not sure why the agent requires them to be 1.
Comment 6 Etsuji Nakai 2012-11-19 09:17:50 EST
Or probably, when the bridge module is loaded by the L2 agent, net.bridge.bridge-nf-call-* becomes 1 by default. If so, the L2 agent should explicitly set net.bridge.bridge-nf-call-* = 0 at that timing.
Comment 7 Etsuji Nakai 2012-11-19 09:38:59 EST
I confirmed my guess by putting the following block in /etc/rc.local. (This is called after sysctl.conf is applied, and befer L2 agent starts.)

------
modprobe bridge
sysctl -w net.bridge.bridge-nf-call-iptables=0
sysctl -w net.bridge.bridge-nf-call-ip6tables=0
sysctl -w net.bridge.bridge-nf-call-arptables=0
------

And the result is:

# sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0

So the problem is that net.bridge.bridge-nf-call-* in sysctl.conf is ignored if the "brdige" module is not loaded at boot time.
Comment 8 Alan Pevec 2012-11-19 13:33:11 EST
> So the problem is that net.bridge.bridge-nf-call-* in sysctl.conf is ignored if the "brdige" module is not loaded at boot time.

Indeed and I see the same on F17 too.
Packaging workaround in openstack-quantum-linuxbridge could be to include /etc/sysconfig/modules/openstack-quantum-linuxbridge.modules script which modprobes bridge.
Comment 9 Etsuji Nakai 2012-11-19 18:21:11 EST
I see. The same workaround worked well for me, thanks.

# cat /etc/sysconfig/modules/openstack-quantum-linuxbridge.modules 
#!/bin/sh

modprobe -b bridge >/dev/null 2>&1

exit 0
Comment 10 Paul Robert Marino 2012-12-18 12:37:27 EST
why not add a "/sbin/sysctl -p" to the end of init script for quantum? or even better quantum should be setting theses options based on need along with others.
Comment 11 Etsuji Nakai 2013-01-03 21:29:11 EST
By the way, I confirmed that this affects only when netns support is disabled. 

When netns support is enabled, nat table is placed under a different namespace which doesn't contain the bridge interface inside it. So the value of net.bridge.bridge-nf-call-iptables doesn't affect the rule matching.

The bottom line is the requirement for net.bridge.bridge-nf-call-iptables could be different based on the configuration to be used by the customer. So hardcoding the parameter in Quantum's source code may be a little bit risky.

I think the best way to deal with it is:

- Use the package workaround of /etc/sysconfig/modules/openstack-quantum-linuxbridge.modules.
- Describe in the Deployment Guide what parameters in /etc/sysctl.conf should be set according to the intended configuration.
Comment 12 Terry Wilson 2013-01-31 19:27:23 EST
Created attachment 691296 [details]
Add sysctl modules script for linuxbridge

apevec: does this patch look like a good start?
Comment 15 Ofer Blaut 2013-03-19 06:18:43 EDT
I have tested floating ip on quantum with linux bridge , it works

also setting is as expected using RHEL 6.4

[root@puma34 ~(keystone_admin_tenant1)]$ lsmod | grep bridge
bridge                 79078  0 
stp                     2218  2 garp,bridge
llc                     5546  3 garp,bridge,stp
[root@puma34 ~(keystone_admin_tenant1)]$ sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
[root@puma34 ~(keystone_admin_tenant1)]$ rpm -qa | grep quantum
openstack-quantum-2012.2.3-8.el6ost.noarch
Comment 17 errata-xmlrpc 2013-03-21 15:03:20 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0672.html

Note You need to log in before you can comment on or make changes to this bug.