Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1056091 - Query of iptables after Floating IP creation takes 20+ second to return.
Summary: Query of iptables after Floating IP creation takes 20+ second to return.
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 4.0
Hardware: x86_64
OS: Linux
high
unspecified
Target Milestone: ---
: 5.0 (RHEL 7)
Assignee: Miguel Angel Ajo
QA Contact: Ofer Blaut
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-21 14:17 UTC by Joe Talerico
Modified: 2016-04-26 17:03 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-05 16:49:08 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Joe Talerico 2014-01-21 14:17:09 UTC
Description of problem:
After creating a Floating IP address pool, querying iptables for the nat rules takes 20+ seconds to return, seems to hang @ Chain neutron-l3-agent-PREROUTING and Chain neutron-l3-agent-snat.

Version-Release number of selected component (if applicable):
RHOS4

How reproducible:
Always

Steps to Reproduce:
Before : Note Date,
[root@pcloud13 ~(keystone_admin)]# date; ip netns exec qrouter-116d1145-a5b5-4cea-9716-02b5f838baae iptables -t nat -L; date;
Tue Jan 21 09:05:26 EST 2014
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere
neutron-postrouting-bottom  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere             169.254.169.254     tcp dpt:http redir ports 9697

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination
neutron-l3-agent-float-snat  all  --  anywhere             anywhere

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination
neutron-l3-agent-snat  all  --  anywhere             anywhere
Tue Jan 21 09:05:26 EST 2014

To Reproduce:
[root@pcloud13 ~(keystone_admin)]# neutron net-create public --router:external=True
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | d72cb780-626f-42a9-a697-eb512c7b22a0 |
| name                      | public                               |
| provider:network_type     | gre                                  |
| provider:physical_network |                                      |
| provider:segmentation_id  | 2                                    |
| router:external           | True                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | fc076705805a41388bd0c542246d5a60     |
+---------------------------+--------------------------------------+
[root@pcloud13 ~(keystone_admin)]# neutron subnet-create public 192.168.1.0/24 --name pub --enable_dhcp True
Created a new subnet:
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "192.168.1.2", "end": "192.168.1.254"} |
| cidr             | 192.168.1.0/24                                   |
| dns_nameservers  |                                                  |
| enable_dhcp      | True                                             |
| gateway_ip       | 192.168.1.1                                      |
| host_routes      |                                                  |
| id               | 544312b8-972a-43b4-a613-196421f31c52             |
| ip_version       | 4                                                |
| name             | pub                                              |
| network_id       | d72cb780-626f-42a9-a697-eb512c7b22a0             |
| tenant_id        | fc076705805a41388bd0c542246d5a60                 |
+------------------+--------------------------------------------------+
[root@pcloud13 ~(keystone_admin)]# neutron router-list
+--------------------------------------+--------+-----------------------+
| id                                   | name   | external_gateway_info |
+--------------------------------------+--------+-----------------------+
| 116d1145-a5b5-4cea-9716-02b5f838baae | Router | null                  |
+--------------------------------------+--------+-----------------------+
[root@pcloud13 ~(keystone_admin)]# neutron router-gateway-set 116d1145-a5b5-4cea-9716-02b5f838baae d72cb780-626f-42a9-a697-eb512c7b22a0
Set gateway for router 116d1145-a5b5-4cea-9716-02b5f838baae
[root@pcloud13 ~(keystone_admin)]# neutron floatingip-create public
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 192.168.1.3                          |
| floating_network_id | d72cb780-626f-42a9-a697-eb512c7b22a0 |
| id                  | 59c0b9e8-29a4-4f84-9c4b-74d584860c18 |
| port_id             |                                      |
| router_id           |                                      |
| tenant_id           | fc076705805a41388bd0c542246d5a60     |
+---------------------+--------------------------------------+
[root@pcloud13 ~(keystone_admin)]# neutron net-list
+--------------------------------------+--------+-----------------------------------------------------+
| id                                   | name   | subnets                                             |
+--------------------------------------+--------+-----------------------------------------------------+
| 5535e15d-19ff-41b9-87ff-785b70d2864a | 10Net  | 1022f90c-7754-4700-9c75-ed319c2ff2ed 10.0.0.0/24    |
| d72cb780-626f-42a9-a697-eb512c7b22a0 | public | 544312b8-972a-43b4-a613-196421f31c52 192.168.1.0/24 |
+--------------------------------------+--------+-----------------------------------------------------+
[root@pcloud13 ~(keystone_admin)]# date; ip netns exec qrouter-116d1145-a5b5-4cea-9716-02b5f838baae iptables -t nat -L; date;
Tue Jan 21 09:08:45 EST 2014
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere
neutron-postrouting-bottom  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere             169.254.169.254     tcp dpt:http redir ports 9697

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination
neutron-l3-agent-float-snat  all  --  anywhere             anywhere
SNAT       all  --  10.0.0.0/24          anywhere            to:192.168.1.2

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination
neutron-l3-agent-snat  all  --  anywhere             anywhere
Tue Jan 21 09:09:42 EST 2014


Expected results:
Return without the witnessed hesitation. 

Additional info:

Comment 2 Miguel Angel Ajo 2014-06-05 16:26:48 UTC
please try:

# iptables -L -t nat -n

I think it's probably related to name resolution, while
iptables tries to do inverse resolution of IPs on the lists.

Comment 3 Miguel Angel Ajo 2014-06-05 16:38:52 UTC
I tried to reproduce it here, but didn't work.

I really suspect is the DNS issue, I checked that I
have a DNS server, and that it's accessible from the namespace.

[root@rhos4-rhel65 ~]# time ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c iptables -L -t nat 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere            
neutron-postrouting-bottom  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere            

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination         
DNAT       all  --  anywhere             172.24.4.228        to:192.168.100.2 

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            ! ctstate DNAT 

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             169.254.169.254     tcp dpt:http redir ports 9697 
DNAT       all  --  anywhere             172.24.4.228        to:192.168.100.2 

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination         
SNAT       all  --  192.168.100.2        anywhere            to:172.24.4.228 

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination         
neutron-l3-agent-float-snat  all  --  anywhere             anywhere            
SNAT       all  --  192.168.100.0/24     anywhere            to:172.24.4.227 

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination         
neutron-l3-agent-snat  all  --  anywhere             anywhere            

real	0m0.025s
user	0m0.000s
sys	0m0.008s
[root@rhos4-rhel65 ~]# ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c cat /etc/resolv.conf 
; generated by /sbin/dhclient-script
search int_connected
nameserver 192.168.50.1
[root@rhos4-rhel65 ~]# ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c ping www.google.com
PING www.google.com (173.194.70.103) 56(84) bytes of data.
64 bytes from fa-in-f103.1e100.net (173.194.70.103): icmp_seq=1 ttl=43 time=49.2 ms

Comment 4 Miguel Angel Ajo 2014-06-05 16:49:08 UTC
Setting a wrong nameserver it reproduces, so it probably was that.

[root@rhos4-rhel65 ~]# ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c echo nameserver 192.168.50.166 > /etc/resolv.conf
[root@rhos4-rhel65 ~]# time ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c iptables -L -t nat 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere            
neutron-postrouting-bottom  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere            

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination         
DNAT       all  --  anywhere             172.24.4.228        to:192.168.100.2 

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            ! ctstate DNAT 

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             169.254.169.254     tcp dpt:http redir ports 9697 
DNAT       all  --  anywhere             172.24.4.228        to:192.168.100.2 

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination         
SNAT       all  --  192.168.100.2        anywhere            to:172.24.4.228 

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination         
neutron-l3-agent-float-snat  all  --  anywhere             anywhere            
SNAT       all  --  192.168.100.0/24     anywhere            to:172.24.4.227 

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination         
neutron-l3-agent-snat  all  --  anywhere             anywhere            

real	0m50.122s
user	0m0.005s
sys	0m0.042s


Note You need to log in before you can comment on or make changes to this bug.