1327931 – LBaasV2 pool creation with session persistence [HTTP_COOKIE] is getting failed

Bug 1327931 - LBaasV2 pool creation with session persistence [HTTP_COOKIE] is getting failed

Summary: LBaasV2 pool creation with session persistence [HTTP_COOKIE] is getting failed

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron-lbaas
Sub Component:
Version:	7.0 (Kilo)
Hardware:	All
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	async
Target Release:	7.0 (Kilo)
Assignee:	Nir Magnezi
QA Contact:	Alexander Stafeyev
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-04-18 03:58 UTC by VIKRANT
Modified:	2019-10-10 11:54 UTC (History)
CC List:	9 users (show)
Fixed In Version:	openstack-neutron-lbaas-2015.1.2-2.el7ost
Doc Type:	Bug Fix
Doc Text:	Cause: Neutron LBaaS v2 was using the 'session_persistence' and 'sessionpersistence' strings inconsistently throughout the neutron-server and LBaaS v2 agent, resulting in the creation of a load balancing pool to return success, meanwhile the agent was raising an exception and failing to configure haproxy. Consequence: The inability to create a load balancing pool with session persistence defined. Fix: Eliminate the source of the exception by using the 'session_persistence' and 'sessionpersistence' strings consistently. Result: The session persistency may now be used when creating a load balancing pool via the LBaaS v2 API.
Clone Of:
Environment:
Last Closed:	2016-05-24 14:54:41 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	306998	0	None	None	None	2016-04-18 13:41:10 UTC
Red Hat Product Errata	RHBA-2016:1103	0	normal	SHIPPED_LIVE	openstack-neutron bug fix advisory	2016-05-24 18:53:50 UTC

Description VIKRANT 2016-04-18 03:58:09 UTC

Description of problem:
LBaasV2 pool creation with session persistence [HTTP_COOKIE] is getting failed. 

Version-Release number of selected component (if applicable):
RHEL OSP 7 (Kilo)

How reproducible:
Everytime.

Steps to Reproduce:

Step 1 : Created a load balancer and listener.

[root@allinone-7 ~(keystone_admin)]# neutron lbaas-loadbalancer-create --name Snet_test_1 9bed29a5-8cb3-436a-89fc-6ca6a8467c03
Created a new loadbalancer:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| description         |                                      |
| id                  | 9455e883-2fb2-49d8-8468-2b24003de808 |
| listeners           |                                      |
| name                | Snet_test_1                          |
| operating_status    | OFFLINE                              |
| provider            | haproxy                              |
| provisioning_status | PENDING_CREATE                       |
| tenant_id           | 90686d89a72143179f7608cb9b6d0898     |
| vip_address         | 10.10.1.10                           |
| vip_port_id         | 74229957-e7b9-4c65-8d23-b04c85dcc693 |
| vip_subnet_id       | 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 |
+---------------------+--------------------------------------+

[root@allinone-7 ~(keystone_admin)]# neutron lbaas-listener-create --loadbalancer 9455e883-2fb2-49d8-8468-2b24003de808 --protocol TCP --protocol-port 80 --name Snet_test_1_80
Created a new listener:
+--------------------------+------------------------------------------------+
| Field                    | Value                                          |
+--------------------------+------------------------------------------------+
| admin_state_up           | True                                           |
| connection_limit         | -1                                             |
| default_pool_id          |                                                |
| default_tls_container_id |                                                |
| description              |                                                |
| id                       | 11e48ded-bb5d-47ec-afbe-0ce441c57171           |
| loadbalancers            | {"id": "9455e883-2fb2-49d8-8468-2b24003de808"} |
| name                     | Snet_test_1_80                                 |
| protocol                 | TCP                                            |
| protocol_port            | 80                                             |
| sni_container_ids        |                                                |
| tenant_id                | 90686d89a72143179f7608cb9b6d0898               |
+--------------------------+------------------------------------------------+

Step 2 : Creating pool with session persistence.

[root@allinone-7 ~(keystone_admin)]# neutron lbaas-pool-create --session-persistence type=HTTP_COOKIE --lb-algorithm ROUND_ROBIN --listener Snet_test_1_80 --protocol TCP --name Snet_test_1_pool80
Created a new pool:
+---------------------+------------------------------------------------+
| Field               | Value                                          |
+---------------------+------------------------------------------------+
| admin_state_up      | True                                           |
| description         |                                                |
| healthmonitor_id    |                                                |
| id                  | ec557aef-4f02-41e2-871c-e9c84158ee11           |
| lb_algorithm        | ROUND_ROBIN                                    |
| listeners           | {"id": "11e48ded-bb5d-47ec-afbe-0ce441c57171"} |
| members             |                                                |
| name                | Snet_test_1_pool80                             |
| protocol            | TCP                                            |
| session_persistence | {"cookie_name": null, "type": "HTTP_COOKIE"}   |
| tenant_id           | 90686d89a72143179f7608cb9b6d0898               |
+---------------------+------------------------------------------------+

Call trace seen while creating pool.

~~~
2016-04-15 10:05:07.139 11565 ERROR neutron_lbaas.agent.agent_manager [req-4fa46075-5567-4149-b389-0ce5393724f9 ] Create pool ec557aef-4f02-41e2-871c-e9c84158ee11 failed on device driver haproxy_ns
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager Traceback (most recent call last):
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/agent/agent_manager.py", line 328, in create_pool
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     driver.pool.create(pool)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py", line 419, in create
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     self.driver.loadbalancer.refresh(pool.listener.loadbalancer)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py", line 364, in refresh
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     if (not self.driver.deploy_instance(loadbalancer) and
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     return f(*args, **kwargs)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap
roxy/namespace_driver.py", line 172, in deploy_instance
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     self.update(loadbalancer)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap
roxy/namespace_driver.py", line 181, in update
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     self._spawn(loadbalancer, extra_args)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap
roxy/namespace_driver.py", line 347, in _spawn
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     haproxy_base_dir)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/lo
adbalancer/drivers/haproxy/jinja_cfg.py", line 89, in save_config
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     haproxy_base_dir)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/lo
adbalancer/drivers/haproxy/jinja_cfg.py", line 221, in render_loadbalancer_obj
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     loadbalancer = _transform_loadbalancer(loadbalancer, haproxy_base_dir)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 236, in _transform_loadbalancer
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     x, haproxy_base_dir) for x in loadbalancer.listeners]
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 261, in _transform_listener
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     ret_value['default_pool'] = _transform_pool(listener.default_pool)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 299, in _transform_pool
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     pool.sessionpersistence)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 310, in _transform_session_persistence
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager     'type': persistence.type,
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager AttributeError: 'dict' object has no attribute 'type'
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager 
~~~

Step 3 : Creating members in the pool.

~~~
[root@allinone-7 ~(keystone_admin)]# neutron lbaas-member-create --subnet 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 --address 10.10.1.5 --protocol-port 80 Snet_test_1_pool80

[root@allinone-7 ~(keystone_admin)]# neutron lbaas-member-create --subnet 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 --address 10.10.1.6 --protocol-port 80 Snet_test_1_pool80
~~~

Error message seen while creating members in pool.

~~~
2016-04-15 10:05:29.479 11565 ERROR neutron_lbaas.agent.agent_manager [req-0e2fa075-dac9-45e4-a0c6-c54b206cc960 ] Create member 0e25ce7e-4e8e-4360-9b83-4ba416afbd56 failed on device driver haproxy_ns
~~~

Step 4 : Curl is getting failed.

~~~
[root@allinone-7 ~(keystone_admin)]# ip netns exec qdhcp-049b58b3-716f-4445-ae24-32a23f8523dd bash
[root@allinone-7 ~(keystone_admin)]# for i in {1..5} ; do curl  10.10.1.10 ; done
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
~~~

Actual results:
LbaaS v2 with session persistence is getting failed.

Expected results:
It should not get fail.

Additional info:

It look like this issue is similar to upstream Bug [1]

[1] https://bugs.launchpad.net/neutron/+bug/1469597

We are able to create LbaaSV2 without session persistence successfully and it's working fine.

I am able to reproduce this issue in my packstack Kilo setup easily.

Comment 26 Alexander Stafeyev 2016-05-17 10:34:02 UTC

Tested it on my environment (KILO AIO setup - packstack installation) 

persistence works well. 

Can see the Set-Cookie in the first response, the cookie in the following requests. The requests go only to 1 member ( configured 2 ) 

BR

Comment 29 Alexander Stafeyev 2016-05-19 05:05:33 UTC

openstack-neutron-lbaas-2015.1.2-2.el7ost.noarch.rpm

Comment 32 errata-xmlrpc 2016-05-24 14:54:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1103.html

Note You need to log in before you can comment on or make changes to this bug.