Bug 1061750

Summary: Neutron Load Balancer VIP Creation Fails
Product: [Community] RDO Reporter: Nir Magnezi <nmagnezi>
Component: openstack-neutronAssignee: Ihar Hrachyshka <ihrachys>
Status: CLOSED CURRENTRELEASE QA Contact: Ofer Blaut <oblaut>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: chrisw, ihrachys, yeylon
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: ---Flags: nmagnezi: needinfo-
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-12 12:11:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1114261    
Bug Blocks:    

Description Nir Magnezi 2014-02-05 14:44:21 UTC
Created attachment 859680 [details]
lbaas-agent.log

Description of problem:
=======================
Neutron Load Balancer VIP Creation Fails.
This did used to work in Havana.
Exact reproduction steps will be detailed ahead.

Version-Release number of selected component (if applicable):
=============================================================
RDO Icehouse
# rpm -qa | grep neutron
python-neutronclient-2.3.3-1.el6.noarch
openstack-neutron-2014.1-0.5.b2.el6.noarch
openstack-neutron-openvswitch-2014.1-0.5.b2.el6.noarch
python-neutron-2014.1-0.5.b2.el6.noarch

How reproducible:
=================
Tested only once.

Steps to Reproduce:
===================
1. Create a Pool:
# neutron lb-pool-create --lb-method ROUND_ROBIN --name pool_vlan_208 --protocol HTTP --subnet-id bda73821-cca3-487b-994c-80144852e0b0 --tenant-id e62f88df0f1f4f5dbf8808b472ce3e7f
+------------------------+--------------------------------------+ 
| Field                  | Value                                | 
+------------------------+--------------------------------------+ 
| admin_state_up         | True                                 |
| description            |                                      |
| health_monitors        |                                      |
| health_monitors_status |                                      |
| id                     | 5fed1ffb-211e-430e-bbfb-243300faf2bb |
| lb_method              | ROUND_ROBIN                          |
| members                |                                      |
| name                   | pool_vlan_208                        |
| protocol               | HTTP                                 |
| provider               | haproxy                              |
| status                 | PENDING_CREATE                       |
| status_description     |                                      |
| subnet_id              | bda73821-cca3-487b-994c-80144852e0b0 |
| tenant_id              | e62f88df0f1f4f5dbf8808b472ce3e7f     |
| vip_id                 |                                      |
+------------------------+--------------------------------------+


2. Search for 2 instances for load balancing:
# nova list --all-tenant --tenant e62f88df0f1f4f5dbf8808b472ce3e7f | /bin/grep "=" | /usr/bin/cut -d" " -f 2' 
c1dd2a50-4257-496e-8b69-fab629e233f4
40adb81-13e0-45f5-8490-150f7dbe3794


3. Record the private (non-floating) IP of the two instances, and added them as pool members:
# neutron lb-member-create --address 192.168.208.2 --protocol-port 80 5fed1ffb-211e-430e-bbfb-243300faf2bb --tenant-id e62f88df0f1f4f5dbf8808b472ce3e7f
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| address            | 192.168.208.2                          |
| admin_state_up     | True                                 |
| id                 | dd410e81-f9c6-4978-ba50-17d16c7eaec8 |
| pool_id            | 5fed1ffb-211e-430e-bbfb-243300faf2bb |
| protocol_port      | 80                                   |
| status             | PENDING_CREATE                       |
| status_description |                                      |
| tenant_id          | e62f88df0f1f4f5dbf8808b472ce3e7f     |
| weight             | 1                                    |
+--------------------+--------------------------------------+

# neutron lb-member-create --address 192.168.208.4 --protocol-port 80 5fed1ffb-211e-430e-bbfb-243300faf2bb --tenant-id e62f88df0f1f4f5dbf8808b472ce3e7f
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| address            | 192.168.208.4                        |
| admin_state_up     | True                                 |
| id                 | 48f712f3-5dc4-45db-9be1-4f581a8b4231 |
| pool_id            | 5fed1ffb-211e-430e-bbfb-243300faf2bb |
| protocol_port      | 80                                   |
| status             | PENDING_CREATE                       |
| status_description |                                      |
| tenant_id          | e62f88df0f1f4f5dbf8808b472ce3e7f     |
| weight             | 1                                    |
+--------------------+--------------------------------------+


4. Create a VIP:
neutron lb-vip-create --name vip_vlan_208 --protocol-port 80 --protocol HTTP --subnet-id bda73821-cca3-487b-994c-80144852e0b0 --tenant-id e62f88df0f1f4f5dbf8808b472ce3e7f 5fed1ffb-211e-430e-bbfb-243300faf2bb
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| address             | 192.168.208.5                        |
| admin_state_up      | True                                 |
| connection_limit    | -1                                   |
| description         |                                      |
| id                  | ed0eb465-85ab-4fa3-b72a-c8930686087d |
| name                | vip_vlan_208                         |
| pool_id             | 5fed1ffb-211e-430e-bbfb-243300faf2bb |
| port_id             | 2137d78b-ea7a-451f-9b30-09ad3526c638 |
| protocol            | HTTP                                 |
| protocol_port       | 80                                   |
| session_persistence |                                      |
| status              | PENDING_CREATE                       |
| status_description  |                                      |
| subnet_id           | bda73821-cca3-487b-994c-80144852e0b0 |
| tenant_id           | e62f88df0f1f4f5dbf8808b472ce3e7f     |
+---------------------+--------------------------------------+


5. Create Health Monitor:
# neutron lb-healthmonitor-create --delay 3 --type HTTP --max-retries 1 --timeout 3 --tenant-id e62f88df0f1f4f5dbf8808b472ce3e7f
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| admin_state_up | True                                 |
| delay          | 3                                    |
| expected_codes | 200                                  |
| http_method    | GET                                  |
| id             | 7d78ff4b-637e-4291-97bf-ec4c4329fee4 |
| max_retries    | 1                                    |
|pools           |                                      |
| tenant_id      | e62f88df0f1f4f5dbf8808b472ce3e7f     |
| timeout        | 3                                    |
| type           | HTTP                                 |
| url_path       | /                                    |
+----------------+--------------------------------------+


6. Associate Health Monitor:
# neutron lb-healthmonitor-associate 7d78ff4b-637e-4291-97bf-ec4c4329fee4 5fed1ffb-211e-430e-bbfb-243300faf2bb --tenant-id e62f88df0f1f4f5dbf8808b472ce3e7f
Associated health monitor 7d78ff4b-637e-4291-97bf-ec4c4329fee4


7. Create Floating IP:
# neutron floatingip-create --tenant-id e62f88df0f1f4f5dbf8808b472ce3e7f ext_net
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 192.168.170.8                        |
| floating_network_id | fc1337bd-784b-4ead-8c5c-682176a8a874 |
| id                  | c39ffa32-334c-4450-8941-7493b63f0263 |
| port_id             |                                      |
| router_id           |                                      |
| tenant_id           | e62f88df0f1f4f5dbf8808b472ce3e7f     |
+---------------------+--------------------------------------+


8. Associate Floating Ip To Vip:
# neutron floatingip-associate c39ffa32-334c-4450-8941-7493b63f0263 2137d78b-ea7a-451f-9b30-09ad3526c638
Associated floatingip c39ffa32-334c-4450-8941-7493b63f0263


9. Verify Pool List Status:
# neutron lb-pool-list
+--------------------------------------+---------------+----------+-------------+----------+----------------+--------+
| id                                   | name          | provider | lb_method   | protocol | admin_state_up | status |
+--------------------------------------+---------------+----------+-------------+----------+----------------+--------+
| 5fed1ffb-211e-430e-bbfb-243300faf2bb | pool_vlan_208 | haproxy  | ROUND_ROBIN | HTTP     | True           | ACTIVE |
+--------------------------------------+---------------+----------+-------------+----------+----------------+--------+


10. Verify The VIP status:
# neutron lb-vip-list
+--------------------------------------+--------------+---------------+----------+----------------+--------+
| id                                   | name         | address       | protocol | admin_state_up | status |
+--------------------------------------+--------------+---------------+----------+----------------+--------+
| ed0eb465-85ab-4fa3-b72a-c8930686087d | vip_vlan_208 | 192.168.208.5 | HTTP     | True           | ERROR  |
+--------------------------------------+--------------+---------------+----------+----------------+--------+

Actual results:
===============
VIP indicated status = ERROR
Errors I found in the lbaas-agent.log file:
ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
ERROR neutron.services.loadbalancer.drivers.haproxy.agent_manager [req-cb8fd5b0-1683-4e1d-ac3b-c4fdedbb6272 None] Unable to retrieve ready devices

Expected results:
=================
VIP indicated status = ACTIVE

Additional info:
================
Find lbaas-agent.log attached.

Comment 2 Ihar Hrachyshka 2014-06-26 15:57:16 UTC
Looks like lbaas agent failed to contact neutron server. Could you please attach neutron.log?

Comment 3 Nir Magnezi 2014-06-29 12:13:36 UTC
(In reply to Ihar Hrachyshka from comment #2)
> Looks like lbaas agent failed to contact neutron server. Could you please
> attach neutron.log?

That setup is no longer live.
I will do by best to reproduce it as soon as Bug #1114261 will be resolved.

Comment 4 Ihar Hrachyshka 2014-07-17 13:00:52 UTC
@Nir, can I ask you to retest it?

Comment 5 Ihar Hrachyshka 2014-08-12 12:11:35 UTC
Ok, there was enough time to retest the bug. Closing the bug as CURRENTRELEASE.