Bug 1434901

Summary:

Bad loadbalancer creation/deletion - loadbalncer stuck in pending delte after delete trial.

Product:

Red Hat OpenStack

Reporter:

Alexander Stafeyev <astafeye>

Component:

openstack-tripleo-heat-templates

Assignee:

Brent Eagles <beagles>

Status:

CLOSED DUPLICATE

QA Contact:

Alexander Stafeyev <astafeye>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

11.0 (Ocata)

CC:

amuller, ihrachys, lpeer, majopela, mburns, nyechiel, rhel-osp-director-maint

Target Milestone:

Target Release:

11.0 (Ocata)

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-04-06 16:33:35 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1378993

Attachments:

Description	Flags
octavia conf	none

Description Alexander Stafeyev 2017-03-22 14:50:34 UTC

Created attachment 1265416 [details]
octavia conf

Description of problem:
Created a LB - there is a connection refusal between lbaas and the contoller. 

When I wanted to delete the LB it remained stuck in pending delete
[root@controller-0 ~]# neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+------------------+----------------------------------+-------------+---------------------+----------+
| id                                   | name             | tenant_id                        | vip_address | provisioning_status | provider |
+--------------------------------------+------------------+----------------------------------+-------------+---------------------+----------+
| 46836e91-5244-4f60-9681-ce3de15527e5 | First_Octavia_LB | e58aab6d1699430c8eb7c7d0ea7ea389 | 192.168.2.4 | ERROR               | octavia  |
+--------------------------------------+------------------+----------------------------------+-------------+---------------------+----------+

Version-Release number of selected component (if applicable):
[root@controller-0 ~]# rpm -qa | grep octavia
python-octavia-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-worker-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
puppet-octavia-10.3.0-0.20170211033053.ad861e4.el7ost.noarch
openstack-octavia-health-manager-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-diskimage-create-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-api-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-common-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-housekeeping-0.10.1-0.20170303212414.4afa76b.el7ost.noarch


How reproducible:

100%

Steps to Reproduce:
1. See the conf file attached 
2. Create LB network , flavor, image
3. Try to create LB 

Actual results:


Expected results:


Additional info:
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin [req-c09e24f5-a762-4085-8dd4-500ab017992e - - - - -] There was an error in the driver
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin Traceback (most recent call last):
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 177, in _call_driver_operation
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     driver_method(context, db_entity, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 118, in func_wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     args[0].failed_completion(args[1], args[2])
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 108, in func_wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     r = func(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 242, in create
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     self.driver.req.post(self._url(lb), args)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 150, in post
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.request('POST', url, args)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 131, in request
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     token = self.auth_session.get_token()
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 840, in get_token
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return (self.get_auth_headers(auth) or {}).get('X-Auth-Token')
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return auth.get_headers(self, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     token = self.get_token(session)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.get_access(session).auth_token
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     self.auth_ref = self.get_auth_ref(session)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     authenticated=False, log=False)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.request(url, 'POST', **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return wrapped(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     resp = send(**kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 690, in _send_request
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     raise exceptions.ConnectFailure(msg)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin ConnectFailure: Unable to establish connection to http://127.0.0.1:5000/v2.0/tokens: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x9afe2d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin 
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource [req-c09e24f5-a762-4085-8dd4-500ab017992e - - - - -] create failed: No details.
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource Traceback (most recent call last):
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 79, in resource
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     result = method(request=request, **args)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 436, in create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 95, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     traceback.format_exc())
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 549, in _create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     obj = do_create(body)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 531, in do_create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 524, in do_create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 381, in create_loadbalancer
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self._call_driver_operation(context, create_method, lb_db)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 185, in _call_driver_operation
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     raise loadbalancerv2.DriverError(msg=e)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource DriverError: Driver error: Unable to establish connection to http://127.0.0.1:5000/v2.0/tokens: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x9afe2d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource

Comment 1 Nir Yechiel 2017-03-27 13:35:23 UTC

I am assuming this is a THT issue?

Comment 2 Brent Eagles 2017-04-06 16:33:35 UTC

This was due to incorrect configuration in the post deployment steps so it should be resolved now.

*** This bug has been marked as a duplicate of bug 1433728 ***