Cause:
Neutron LBaaS v2 was using the 'session_persistence' and 'sessionpersistence' strings inconsistently throughout the neutron-server and LBaaS v2 agent, resulting in the creation of a load balancing pool to return success, meanwhile the agent was raising an exception and failing to configure haproxy.
Consequence:
The inability to create a load balancing pool with session persistence defined.
Fix:
Eliminate the source of the exception by using the 'session_persistence' and 'sessionpersistence' strings consistently.
Result:
The session persistency may now be used when creating a load balancing pool via the LBaaS v2 API.
Description of problem:
LBaasV2 pool creation with session persistence [HTTP_COOKIE] is getting failed.
Version-Release number of selected component (if applicable):
RHEL OSP 7 (Kilo)
How reproducible:
Everytime.
Steps to Reproduce:
Step 1 : Created a load balancer and listener.
[root@allinone-7 ~(keystone_admin)]# neutron lbaas-loadbalancer-create --name Snet_test_1 9bed29a5-8cb3-436a-89fc-6ca6a8467c03
Created a new loadbalancer:
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| description | |
| id | 9455e883-2fb2-49d8-8468-2b24003de808 |
| listeners | |
| name | Snet_test_1 |
| operating_status | OFFLINE |
| provider | haproxy |
| provisioning_status | PENDING_CREATE |
| tenant_id | 90686d89a72143179f7608cb9b6d0898 |
| vip_address | 10.10.1.10 |
| vip_port_id | 74229957-e7b9-4c65-8d23-b04c85dcc693 |
| vip_subnet_id | 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 |
+---------------------+--------------------------------------+
[root@allinone-7 ~(keystone_admin)]# neutron lbaas-listener-create --loadbalancer 9455e883-2fb2-49d8-8468-2b24003de808 --protocol TCP --protocol-port 80 --name Snet_test_1_80
Created a new listener:
+--------------------------+------------------------------------------------+
| Field | Value |
+--------------------------+------------------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| default_pool_id | |
| default_tls_container_id | |
| description | |
| id | 11e48ded-bb5d-47ec-afbe-0ce441c57171 |
| loadbalancers | {"id": "9455e883-2fb2-49d8-8468-2b24003de808"} |
| name | Snet_test_1_80 |
| protocol | TCP |
| protocol_port | 80 |
| sni_container_ids | |
| tenant_id | 90686d89a72143179f7608cb9b6d0898 |
+--------------------------+------------------------------------------------+
Step 2 : Creating pool with session persistence.
[root@allinone-7 ~(keystone_admin)]# neutron lbaas-pool-create --session-persistence type=HTTP_COOKIE --lb-algorithm ROUND_ROBIN --listener Snet_test_1_80 --protocol TCP --name Snet_test_1_pool80
Created a new pool:
+---------------------+------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------+
| admin_state_up | True |
| description | |
| healthmonitor_id | |
| id | ec557aef-4f02-41e2-871c-e9c84158ee11 |
| lb_algorithm | ROUND_ROBIN |
| listeners | {"id": "11e48ded-bb5d-47ec-afbe-0ce441c57171"} |
| members | |
| name | Snet_test_1_pool80 |
| protocol | TCP |
| session_persistence | {"cookie_name": null, "type": "HTTP_COOKIE"} |
| tenant_id | 90686d89a72143179f7608cb9b6d0898 |
+---------------------+------------------------------------------------+
Call trace seen while creating pool.
~~~
2016-04-15 10:05:07.139 11565 ERROR neutron_lbaas.agent.agent_manager [req-4fa46075-5567-4149-b389-0ce5393724f9 ] Create pool ec557aef-4f02-41e2-871c-e9c84158ee11 failed on device driver haproxy_ns
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager Traceback (most recent call last):
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/agent/agent_manager.py", line 328, in create_pool
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager driver.pool.create(pool)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py", line 419, in create
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager self.driver.loadbalancer.refresh(pool.listener.loadbalancer)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py", line 364, in refresh
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager if (not self.driver.deploy_instance(loadbalancer) and
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager return f(*args, **kwargs)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap
roxy/namespace_driver.py", line 172, in deploy_instance
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager self.update(loadbalancer)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap
roxy/namespace_driver.py", line 181, in update
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager self._spawn(loadbalancer, extra_args)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap
roxy/namespace_driver.py", line 347, in _spawn
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager haproxy_base_dir)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/lo
adbalancer/drivers/haproxy/jinja_cfg.py", line 89, in save_config
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager haproxy_base_dir)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/lo
adbalancer/drivers/haproxy/jinja_cfg.py", line 221, in render_loadbalancer_obj
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager loadbalancer = _transform_loadbalancer(loadbalancer, haproxy_base_dir)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 236, in _transform_loadbalancer
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager x, haproxy_base_dir) for x in loadbalancer.listeners]
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 261, in _transform_listener
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager ret_value['default_pool'] = _transform_pool(listener.default_pool)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 299, in _transform_pool
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager pool.sessionpersistence)
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 310, in _transform_session_persistence
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager 'type': persistence.type,
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager AttributeError: 'dict' object has no attribute 'type'
2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager
~~~
Step 3 : Creating members in the pool.
~~~
[root@allinone-7 ~(keystone_admin)]# neutron lbaas-member-create --subnet 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 --address 10.10.1.5 --protocol-port 80 Snet_test_1_pool80
[root@allinone-7 ~(keystone_admin)]# neutron lbaas-member-create --subnet 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 --address 10.10.1.6 --protocol-port 80 Snet_test_1_pool80
~~~
Error message seen while creating members in pool.
~~~
2016-04-15 10:05:29.479 11565 ERROR neutron_lbaas.agent.agent_manager [req-0e2fa075-dac9-45e4-a0c6-c54b206cc960 ] Create member 0e25ce7e-4e8e-4360-9b83-4ba416afbd56 failed on device driver haproxy_ns
~~~
Step 4 : Curl is getting failed.
~~~
[root@allinone-7 ~(keystone_admin)]# ip netns exec qdhcp-049b58b3-716f-4445-ae24-32a23f8523dd bash
[root@allinone-7 ~(keystone_admin)]# for i in {1..5} ; do curl 10.10.1.10 ; done
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
~~~
Actual results:
LbaaS v2 with session persistence is getting failed.
Expected results:
It should not get fail.
Additional info:
It look like this issue is similar to upstream Bug [1]
[1] https://bugs.launchpad.net/neutron/+bug/1469597
We are able to create LbaaSV2 without session persistence successfully and it's working fine.
I am able to reproduce this issue in my packstack Kilo setup easily.
Comment 26Alexander Stafeyev
2016-05-17 10:34:02 UTC
Tested it on my environment (KILO AIO setup - packstack installation)
persistence works well.
Can see the Set-Cookie in the first response, the cookie in the following requests. The requests go only to 1 member ( configured 2 )
BR
Comment 29Alexander Stafeyev
2016-05-19 05:05:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2016-1103.html
Description of problem: LBaasV2 pool creation with session persistence [HTTP_COOKIE] is getting failed. Version-Release number of selected component (if applicable): RHEL OSP 7 (Kilo) How reproducible: Everytime. Steps to Reproduce: Step 1 : Created a load balancer and listener. [root@allinone-7 ~(keystone_admin)]# neutron lbaas-loadbalancer-create --name Snet_test_1 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 Created a new loadbalancer: +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | description | | | id | 9455e883-2fb2-49d8-8468-2b24003de808 | | listeners | | | name | Snet_test_1 | | operating_status | OFFLINE | | provider | haproxy | | provisioning_status | PENDING_CREATE | | tenant_id | 90686d89a72143179f7608cb9b6d0898 | | vip_address | 10.10.1.10 | | vip_port_id | 74229957-e7b9-4c65-8d23-b04c85dcc693 | | vip_subnet_id | 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 | +---------------------+--------------------------------------+ [root@allinone-7 ~(keystone_admin)]# neutron lbaas-listener-create --loadbalancer 9455e883-2fb2-49d8-8468-2b24003de808 --protocol TCP --protocol-port 80 --name Snet_test_1_80 Created a new listener: +--------------------------+------------------------------------------------+ | Field | Value | +--------------------------+------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | default_pool_id | | | default_tls_container_id | | | description | | | id | 11e48ded-bb5d-47ec-afbe-0ce441c57171 | | loadbalancers | {"id": "9455e883-2fb2-49d8-8468-2b24003de808"} | | name | Snet_test_1_80 | | protocol | TCP | | protocol_port | 80 | | sni_container_ids | | | tenant_id | 90686d89a72143179f7608cb9b6d0898 | +--------------------------+------------------------------------------------+ Step 2 : Creating pool with session persistence. [root@allinone-7 ~(keystone_admin)]# neutron lbaas-pool-create --session-persistence type=HTTP_COOKIE --lb-algorithm ROUND_ROBIN --listener Snet_test_1_80 --protocol TCP --name Snet_test_1_pool80 Created a new pool: +---------------------+------------------------------------------------+ | Field | Value | +---------------------+------------------------------------------------+ | admin_state_up | True | | description | | | healthmonitor_id | | | id | ec557aef-4f02-41e2-871c-e9c84158ee11 | | lb_algorithm | ROUND_ROBIN | | listeners | {"id": "11e48ded-bb5d-47ec-afbe-0ce441c57171"} | | members | | | name | Snet_test_1_pool80 | | protocol | TCP | | session_persistence | {"cookie_name": null, "type": "HTTP_COOKIE"} | | tenant_id | 90686d89a72143179f7608cb9b6d0898 | +---------------------+------------------------------------------------+ Call trace seen while creating pool. ~~~ 2016-04-15 10:05:07.139 11565 ERROR neutron_lbaas.agent.agent_manager [req-4fa46075-5567-4149-b389-0ce5393724f9 ] Create pool ec557aef-4f02-41e2-871c-e9c84158ee11 failed on device driver haproxy_ns 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager Traceback (most recent call last): 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/agent/agent_manager.py", line 328, in create_pool 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager driver.pool.create(pool) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py", line 419, in create 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager self.driver.loadbalancer.refresh(pool.listener.loadbalancer) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py", line 364, in refresh 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager if (not self.driver.deploy_instance(loadbalancer) and 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager return f(*args, **kwargs) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap roxy/namespace_driver.py", line 172, in deploy_instance 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager self.update(loadbalancer) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap roxy/namespace_driver.py", line 181, in update 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager self._spawn(loadbalancer, extra_args) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/hap roxy/namespace_driver.py", line 347, in _spawn 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager haproxy_base_dir) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/lo adbalancer/drivers/haproxy/jinja_cfg.py", line 89, in save_config 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager haproxy_base_dir) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/lo adbalancer/drivers/haproxy/jinja_cfg.py", line 221, in render_loadbalancer_obj 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager loadbalancer = _transform_loadbalancer(loadbalancer, haproxy_base_dir) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 236, in _transform_loadbalancer 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager x, haproxy_base_dir) for x in loadbalancer.listeners] 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 261, in _transform_listener 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager ret_value['default_pool'] = _transform_pool(listener.default_pool) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 299, in _transform_pool 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager pool.sessionpersistence) 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/drivers/haproxy/jinja_cfg.py", line 310, in _transform_session_persistence 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager 'type': persistence.type, 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager AttributeError: 'dict' object has no attribute 'type' 2016-04-15 10:05:07.139 11565 TRACE neutron_lbaas.agent.agent_manager ~~~ Step 3 : Creating members in the pool. ~~~ [root@allinone-7 ~(keystone_admin)]# neutron lbaas-member-create --subnet 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 --address 10.10.1.5 --protocol-port 80 Snet_test_1_pool80 [root@allinone-7 ~(keystone_admin)]# neutron lbaas-member-create --subnet 9bed29a5-8cb3-436a-89fc-6ca6a8467c03 --address 10.10.1.6 --protocol-port 80 Snet_test_1_pool80 ~~~ Error message seen while creating members in pool. ~~~ 2016-04-15 10:05:29.479 11565 ERROR neutron_lbaas.agent.agent_manager [req-0e2fa075-dac9-45e4-a0c6-c54b206cc960 ] Create member 0e25ce7e-4e8e-4360-9b83-4ba416afbd56 failed on device driver haproxy_ns ~~~ Step 4 : Curl is getting failed. ~~~ [root@allinone-7 ~(keystone_admin)]# ip netns exec qdhcp-049b58b3-716f-4445-ae24-32a23f8523dd bash [root@allinone-7 ~(keystone_admin)]# for i in {1..5} ; do curl 10.10.1.10 ; done curl: (52) Empty reply from server curl: (52) Empty reply from server curl: (52) Empty reply from server curl: (52) Empty reply from server curl: (52) Empty reply from server ~~~ Actual results: LbaaS v2 with session persistence is getting failed. Expected results: It should not get fail. Additional info: It look like this issue is similar to upstream Bug [1] [1] https://bugs.launchpad.net/neutron/+bug/1469597 We are able to create LbaaSV2 without session persistence successfully and it's working fine. I am able to reproduce this issue in my packstack Kilo setup easily.