Bug 1056091
| Summary: | Query of iptables after Floating IP creation takes 20+ second to return. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Joe Talerico <jtaleric> |
| Component: | openstack-neutron | Assignee: | Miguel Angel Ajo <mangelajo> |
| Status: | CLOSED NOTABUG | QA Contact: | Ofer Blaut <oblaut> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.0 | CC: | chrisw, jtaleric, lpeer, majopela, nyechiel, perfbz, yeylon |
| Target Milestone: | --- | ||
| Target Release: | 5.0 (RHEL 7) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-06-05 16:49:08 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
please try: # iptables -L -t nat -n I think it's probably related to name resolution, while iptables tries to do inverse resolution of IPs on the lists. I tried to reproduce it here, but didn't work. I really suspect is the DNS issue, I checked that I have a DNS server, and that it's accessible from the namespace. [root@rhos4-rhel65 ~]# time ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-PREROUTING all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-POSTROUTING all -- anywhere anywhere neutron-postrouting-bottom all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-l3-agent-OUTPUT all -- anywhere anywhere Chain neutron-l3-agent-OUTPUT (1 references) target prot opt source destination DNAT all -- anywhere 172.24.4.228 to:192.168.100.2 Chain neutron-l3-agent-POSTROUTING (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere ! ctstate DNAT Chain neutron-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- anywhere 169.254.169.254 tcp dpt:http redir ports 9697 DNAT all -- anywhere 172.24.4.228 to:192.168.100.2 Chain neutron-l3-agent-float-snat (1 references) target prot opt source destination SNAT all -- 192.168.100.2 anywhere to:172.24.4.228 Chain neutron-l3-agent-snat (1 references) target prot opt source destination neutron-l3-agent-float-snat all -- anywhere anywhere SNAT all -- 192.168.100.0/24 anywhere to:172.24.4.227 Chain neutron-postrouting-bottom (1 references) target prot opt source destination neutron-l3-agent-snat all -- anywhere anywhere real 0m0.025s user 0m0.000s sys 0m0.008s [root@rhos4-rhel65 ~]# ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c cat /etc/resolv.conf ; generated by /sbin/dhclient-script search int_connected nameserver 192.168.50.1 [root@rhos4-rhel65 ~]# ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c ping www.google.com PING www.google.com (173.194.70.103) 56(84) bytes of data. 64 bytes from fa-in-f103.1e100.net (173.194.70.103): icmp_seq=1 ttl=43 time=49.2 ms Setting a wrong nameserver it reproduces, so it probably was that. [root@rhos4-rhel65 ~]# ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c echo nameserver 192.168.50.166 > /etc/resolv.conf [root@rhos4-rhel65 ~]# time ip netns exec qrouter-fcf103bf-40e7-49a4-a2c3-f03fd589094c iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-PREROUTING all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-POSTROUTING all -- anywhere anywhere neutron-postrouting-bottom all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-l3-agent-OUTPUT all -- anywhere anywhere Chain neutron-l3-agent-OUTPUT (1 references) target prot opt source destination DNAT all -- anywhere 172.24.4.228 to:192.168.100.2 Chain neutron-l3-agent-POSTROUTING (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere ! ctstate DNAT Chain neutron-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- anywhere 169.254.169.254 tcp dpt:http redir ports 9697 DNAT all -- anywhere 172.24.4.228 to:192.168.100.2 Chain neutron-l3-agent-float-snat (1 references) target prot opt source destination SNAT all -- 192.168.100.2 anywhere to:172.24.4.228 Chain neutron-l3-agent-snat (1 references) target prot opt source destination neutron-l3-agent-float-snat all -- anywhere anywhere SNAT all -- 192.168.100.0/24 anywhere to:172.24.4.227 Chain neutron-postrouting-bottom (1 references) target prot opt source destination neutron-l3-agent-snat all -- anywhere anywhere real 0m50.122s user 0m0.005s sys 0m0.042s |
Description of problem: After creating a Floating IP address pool, querying iptables for the nat rules takes 20+ seconds to return, seems to hang @ Chain neutron-l3-agent-PREROUTING and Chain neutron-l3-agent-snat. Version-Release number of selected component (if applicable): RHOS4 How reproducible: Always Steps to Reproduce: Before : Note Date, [root@pcloud13 ~(keystone_admin)]# date; ip netns exec qrouter-116d1145-a5b5-4cea-9716-02b5f838baae iptables -t nat -L; date; Tue Jan 21 09:05:26 EST 2014 Chain PREROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-PREROUTING all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-POSTROUTING all -- anywhere anywhere neutron-postrouting-bottom all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-l3-agent-OUTPUT all -- anywhere anywhere Chain neutron-l3-agent-OUTPUT (1 references) target prot opt source destination Chain neutron-l3-agent-POSTROUTING (1 references) target prot opt source destination Chain neutron-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- anywhere 169.254.169.254 tcp dpt:http redir ports 9697 Chain neutron-l3-agent-float-snat (1 references) target prot opt source destination Chain neutron-l3-agent-snat (1 references) target prot opt source destination neutron-l3-agent-float-snat all -- anywhere anywhere Chain neutron-postrouting-bottom (1 references) target prot opt source destination neutron-l3-agent-snat all -- anywhere anywhere Tue Jan 21 09:05:26 EST 2014 To Reproduce: [root@pcloud13 ~(keystone_admin)]# neutron net-create public --router:external=True Created a new network: +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | d72cb780-626f-42a9-a697-eb512c7b22a0 | | name | public | | provider:network_type | gre | | provider:physical_network | | | provider:segmentation_id | 2 | | router:external | True | | shared | False | | status | ACTIVE | | subnets | | | tenant_id | fc076705805a41388bd0c542246d5a60 | +---------------------------+--------------------------------------+ [root@pcloud13 ~(keystone_admin)]# neutron subnet-create public 192.168.1.0/24 --name pub --enable_dhcp True Created a new subnet: +------------------+--------------------------------------------------+ | Field | Value | +------------------+--------------------------------------------------+ | allocation_pools | {"start": "192.168.1.2", "end": "192.168.1.254"} | | cidr | 192.168.1.0/24 | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 192.168.1.1 | | host_routes | | | id | 544312b8-972a-43b4-a613-196421f31c52 | | ip_version | 4 | | name | pub | | network_id | d72cb780-626f-42a9-a697-eb512c7b22a0 | | tenant_id | fc076705805a41388bd0c542246d5a60 | +------------------+--------------------------------------------------+ [root@pcloud13 ~(keystone_admin)]# neutron router-list +--------------------------------------+--------+-----------------------+ | id | name | external_gateway_info | +--------------------------------------+--------+-----------------------+ | 116d1145-a5b5-4cea-9716-02b5f838baae | Router | null | +--------------------------------------+--------+-----------------------+ [root@pcloud13 ~(keystone_admin)]# neutron router-gateway-set 116d1145-a5b5-4cea-9716-02b5f838baae d72cb780-626f-42a9-a697-eb512c7b22a0 Set gateway for router 116d1145-a5b5-4cea-9716-02b5f838baae [root@pcloud13 ~(keystone_admin)]# neutron floatingip-create public Created a new floatingip: +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | fixed_ip_address | | | floating_ip_address | 192.168.1.3 | | floating_network_id | d72cb780-626f-42a9-a697-eb512c7b22a0 | | id | 59c0b9e8-29a4-4f84-9c4b-74d584860c18 | | port_id | | | router_id | | | tenant_id | fc076705805a41388bd0c542246d5a60 | +---------------------+--------------------------------------+ [root@pcloud13 ~(keystone_admin)]# neutron net-list +--------------------------------------+--------+-----------------------------------------------------+ | id | name | subnets | +--------------------------------------+--------+-----------------------------------------------------+ | 5535e15d-19ff-41b9-87ff-785b70d2864a | 10Net | 1022f90c-7754-4700-9c75-ed319c2ff2ed 10.0.0.0/24 | | d72cb780-626f-42a9-a697-eb512c7b22a0 | public | 544312b8-972a-43b4-a613-196421f31c52 192.168.1.0/24 | +--------------------------------------+--------+-----------------------------------------------------+ [root@pcloud13 ~(keystone_admin)]# date; ip netns exec qrouter-116d1145-a5b5-4cea-9716-02b5f838baae iptables -t nat -L; date; Tue Jan 21 09:08:45 EST 2014 Chain PREROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-PREROUTING all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-POSTROUTING all -- anywhere anywhere neutron-postrouting-bottom all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-l3-agent-OUTPUT all -- anywhere anywhere Chain neutron-l3-agent-OUTPUT (1 references) target prot opt source destination Chain neutron-l3-agent-POSTROUTING (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere ! ctstate DNAT Chain neutron-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- anywhere 169.254.169.254 tcp dpt:http redir ports 9697 Chain neutron-l3-agent-float-snat (1 references) target prot opt source destination Chain neutron-l3-agent-snat (1 references) target prot opt source destination neutron-l3-agent-float-snat all -- anywhere anywhere SNAT all -- 10.0.0.0/24 anywhere to:192.168.1.2 Chain neutron-postrouting-bottom (1 references) target prot opt source destination neutron-l3-agent-snat all -- anywhere anywhere Tue Jan 21 09:09:42 EST 2014 Expected results: Return without the witnessed hesitation. Additional info: