Bug 1434904

Summary:	Bad loadbalancer creation
Product:	Red Hat OpenStack	Reporter:	Alexander Stafeyev <astafeye>
Component:	openstack-tripleo-heat-templates	Assignee:	Brent Eagles <beagles>
Status:	CLOSED DUPLICATE	QA Contact:	Alexander Stafeyev <astafeye>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	11.0 (Ocata)	CC:	aschultz, mburns, rhel-osp-director-maint
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-04-06 16:39:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1378993

Description Alexander Stafeyev 2017-03-22 14:55:52 UTC

Description of problem:

Lb cannot be created after deployment - connection refuse on port 5000


Version-Release number of selected component (if applicable):
[root@controller-0 ~]# rpm -qa | grep octavia
python-octavia-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-worker-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
puppet-octavia-10.3.0-0.20170211033053.ad861e4.el7ost.noarch
openstack-octavia-health-manager-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-diskimage-create-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-api-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-common-0.10.1-0.20170303212414.4afa76b.el7ost.noarch
openstack-octavia-housekeeping-0.10.1-0.20170303212414.4afa76b.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. See the conf file attached 
2. Create LB network , flavor, image
3. see logs (neutron)

Actual results:
LB created in error state 

Expected results:
LB should be created - the connection should not be refused after ospd deployment with octavia yamls. 

Additional info:
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin [req-c09e24f5-a762-4085-8dd4-500ab017992e - - - - -] There was an error in the driver
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin Traceback (most recent call last):
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 177, in _call_driver_operation
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     driver_method(context, db_entity, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 118, in func_wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     args[0].failed_completion(args[1], args[2])
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 108, in func_wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     r = func(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 242, in create
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     self.driver.req.post(self._url(lb), args)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 150, in post
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.request('POST', url, args)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 131, in request
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     token = self.auth_session.get_token()
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 840, in get_token
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return (self.get_auth_headers(auth) or {}).get('X-Auth-Token')
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return auth.get_headers(self, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     token = self.get_token(session)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.get_access(session).auth_token
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     self.auth_ref = self.get_auth_ref(session)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     authenticated=False, log=False)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.request(url, 'POST', **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     return wrapped(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     resp = send(**kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 690, in _send_request
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin     raise exceptions.ConnectFailure(msg)
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin ConnectFailure: Unable to establish connection to http://127.0.0.1:5000/v2.0/tokens: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x9afe2d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
/var/log/neutron/server.log:2017-03-22 14:35:20.456 464812 ERROR neutron_lbaas.services.loadbalancer.plugin 
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource [req-c09e24f5-a762-4085-8dd4-500ab017992e - - - - -] create failed: No details.
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource Traceback (most recent call last):
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 79, in resource
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     result = method(request=request, **args)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 436, in create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 95, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     traceback.format_exc())
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 549, in _create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     obj = do_create(body)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 531, in do_create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self.force_reraise()
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 524, in do_create
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 381, in create_loadbalancer
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     self._call_driver_operation(context, create_method, lb_db)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 185, in _call_driver_operation
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource     raise loadbalancerv2.DriverError(msg=e)
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource DriverError: Driver error: Unable to establish connection to http://127.0.0.1:5000/v2.0/tokens: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x9afe2d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
/var/log/neutron/server.log:2017-03-22 14:35:20.528 464812 ERROR neutron.api.v2.resource

Comment 1 Brent Eagles 2017-04-06 16:39:08 UTC

This was caused by bad configuration due to errors in documented post deployment steps here https://bugzilla.redhat.com/show_bug.cgi?id=1433728. Closing as a duplicate.

*** This bug has been marked as a duplicate of bug 1433728 ***