Bug 1298598

Summary:	AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate
Product:	Red Hat OpenStack	Reporter:	Robin Cernin <rcernin>
Component:	openstack-neutron	Assignee:	Jakub Libosvar <jlibosva>
Status:	CLOSED ERRATA	QA Contact:	Toni Freger <tfreger>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.0 (Juno)	CC:	amuller, chrisw, dmaley, jlibosva, mschuppe, nyechiel, tfreger, yeylon
Target Milestone:	async	Keywords:	ZStream
Target Release:	6.0 (Juno)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-2014.2.3-33.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1298873 (view as bug list)		Environment:
Last Closed:	2016-02-08 14:18:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1298873

Description Robin Cernin 2016-01-14 14:10:48 UTC

Description of problem:

neutron]$ zgrep AssertionError server.log* 
2016-01-06 17:27:50.589 33160 INFO neutron.openstack.common.service [-] Caught SIGTERM, stopping children
2016-01-06 17:27:50.589 33160 INFO neutron.openstack.common.service [-] Waiting on 32 children to exit
2016-01-06 17:27:50.589 33207 INFO neutron.openstack.common.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Child caught SIGTERM, exiting
2016-01-06 17:27:50.589 33208 INFO neutron.openstack.common.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Child caught SIGTERM, exiting
...
2016-01-06 17:27:50.609 33185 ERROR neutron.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Unrecoverable error: please check log for details.
2016-01-06 17:27:50.609 33185 TRACE neutron.service Traceback (most recent call last):
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 102, in serve_wsgi
2016-01-06 17:27:50.609 33185 TRACE neutron.service     service.start()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 73, in start
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self.wsgi_app = _run_wsgi(self.app_name)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 174, in _run_wsgi
2016-01-06 17:27:50.609 33185 TRACE neutron.service     workers=cfg.CONF.api_workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 224, in start
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._launch(application, workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 238, in _launch
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._server.launch_service(service, workers=workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 341, in launch_service
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._start_child(wrap)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 322, in _start_child
2016-01-06 17:27:50.609 33185 TRACE neutron.service     status, signo = self._child_wait_for_exit_or_signal(launcher)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 280, in _child_wait_for_exit_or_signal
2016-01-06 17:27:50.609 33185 TRACE neutron.service     launcher.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 128, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self.services.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 470, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     service.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 117, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._server.kill()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 240, in kill
2016-01-06 17:27:50.609 33185 TRACE neutron.service     return kill(self, *throw_args)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 294, in kill
2016-01-06 17:27:50.609 33185 TRACE neutron.service     g.throw(*throw_args)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-01-06 17:27:50.609 33185 TRACE neutron.service     result = function(*args, **kwargs)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 263, in _run
2016-01-06 17:27:50.609 33185 TRACE neutron.service     socket_timeout=self.client_socket_timeout)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 842, in server
2016-01-06 17:27:50.609 33185 TRACE neutron.service     pool.waitall()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 117, in waitall
2016-01-06 17:27:50.609 33185 TRACE neutron.service     "Calling waitall() from within one of the " \
2016-01-06 17:27:50.609 33185 TRACE neutron.service AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-06 17:27:50.609 33185 TRACE neutron.service 

Version-Release number of selected component (if applicable):

openstack-neutron-2014.2.3-23.el7ost.noarch
openstack-neutron-common-2014.2.3-23.el7ost.noarch
openstack-neutron-ml2-2014.2.3-23.el7ost.noarch
openstack-neutron-openvswitch-2014.2.3-23.el7ost.noarch
python-neutron-2014.2.3-23.el7ost.noarch
python-neutronclient-2.3.9-1.el7ost.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:



Expected results:


Additional info:

Comment 2 Robin Cernin 2016-01-14 14:19:35 UTC

Hello,

Could we request possible backport of the fix to OSP6 / Juno ?

Thank you,
Regards,
Robin Černín

Comment 3 Robin Cernin 2016-01-14 15:58:25 UTC

Hello,

Also affects 

2016-01-13 16:36:02.116 48087 TRACE keystone.common.environment.eventlet_server AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-13 16:36:02.116 48089 TRACE keystone.common.environment.eventlet_server AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.

Thank you,
Regards,
Robin Černín

Comment 4 Jakub Libosvar 2016-01-15 08:19:45 UTC

(In reply to Robin Cernin from comment #3)
> Hello,
> 
> Also affects 
> 
> 2016-01-13 16:36:02.116 48087 TRACE
> keystone.common.environment.eventlet_server AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 2016-01-13 16:36:02.116 48089 TRACE
> keystone.common.environment.eventlet_server AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 
> Thank you,
> Regards,
> Robin Černín

You should clone this bug to keystone and ask them to backport https://review.openstack.org/#/c/160720/

Comment 5 Robin Cernin 2016-01-15 10:58:52 UTC

(In reply to Jakub Libosvar from comment #4)
> (In reply to Robin Cernin from comment #3)
> > Hello,
> > 
> > Also affects 
> > 
> > 2016-01-13 16:36:02.116 48087 TRACE
> > keystone.common.environment.eventlet_server AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 2016-01-13 16:36:02.116 48089 TRACE
> > keystone.common.environment.eventlet_server AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 
> > Thank you,
> > Regards,
> > Robin Černín
> 
> You should clone this bug to keystone and ask them to backport
> https://review.openstack.org/#/c/160720/

Cloned BZ for keystone to https://bugzilla.redhat.com/1298873

Comment 8 Toni Freger 2016-02-03 14:43:03 UTC

The code exists within the release.
openstack-neutron-2014.2.3-33.el7ost.noarch

Comment 10 errata-xmlrpc 2016-02-08 14:18:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0130.html