Bug 1298598 - AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate
Summary: AssertionError: Calling waitall() from within one of the GreenPool's greenthr...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: async
: 6.0 (Juno)
Assignee: Jakub Libosvar
QA Contact: Toni Freger
URL:
Whiteboard:
Depends On:
Blocks: 1298873
TreeView+ depends on / blocked
 
Reported: 2016-01-14 14:10 UTC by Robin Cernin
Modified: 2019-09-12 09:45 UTC (History)
8 users (show)

Fixed In Version: openstack-neutron-2014.2.3-33.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1298873 (view as bug list)
Environment:
Last Closed: 2016-02-08 14:18:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1423250 0 None None None 2016-01-14 14:10:48 UTC
OpenStack gerrit 157320 0 None None None 2016-01-15 08:12:20 UTC
Red Hat Product Errata RHBA-2016:0130 0 normal SHIPPED_LIVE openstack-neutron bug fix and enhancement advisory 2016-02-08 19:16:42 UTC

Description Robin Cernin 2016-01-14 14:10:48 UTC
Description of problem:

neutron]$ zgrep AssertionError server.log* 
2016-01-06 17:27:50.589 33160 INFO neutron.openstack.common.service [-] Caught SIGTERM, stopping children
2016-01-06 17:27:50.589 33160 INFO neutron.openstack.common.service [-] Waiting on 32 children to exit
2016-01-06 17:27:50.589 33207 INFO neutron.openstack.common.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Child caught SIGTERM, exiting
2016-01-06 17:27:50.589 33208 INFO neutron.openstack.common.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Child caught SIGTERM, exiting
...
2016-01-06 17:27:50.609 33185 ERROR neutron.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Unrecoverable error: please check log for details.
2016-01-06 17:27:50.609 33185 TRACE neutron.service Traceback (most recent call last):
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 102, in serve_wsgi
2016-01-06 17:27:50.609 33185 TRACE neutron.service     service.start()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 73, in start
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self.wsgi_app = _run_wsgi(self.app_name)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 174, in _run_wsgi
2016-01-06 17:27:50.609 33185 TRACE neutron.service     workers=cfg.CONF.api_workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 224, in start
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._launch(application, workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 238, in _launch
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._server.launch_service(service, workers=workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 341, in launch_service
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._start_child(wrap)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 322, in _start_child
2016-01-06 17:27:50.609 33185 TRACE neutron.service     status, signo = self._child_wait_for_exit_or_signal(launcher)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 280, in _child_wait_for_exit_or_signal
2016-01-06 17:27:50.609 33185 TRACE neutron.service     launcher.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 128, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self.services.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 470, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     service.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 117, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._server.kill()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 240, in kill
2016-01-06 17:27:50.609 33185 TRACE neutron.service     return kill(self, *throw_args)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 294, in kill
2016-01-06 17:27:50.609 33185 TRACE neutron.service     g.throw(*throw_args)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-01-06 17:27:50.609 33185 TRACE neutron.service     result = function(*args, **kwargs)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 263, in _run
2016-01-06 17:27:50.609 33185 TRACE neutron.service     socket_timeout=self.client_socket_timeout)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 842, in server
2016-01-06 17:27:50.609 33185 TRACE neutron.service     pool.waitall()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 117, in waitall
2016-01-06 17:27:50.609 33185 TRACE neutron.service     "Calling waitall() from within one of the " \
2016-01-06 17:27:50.609 33185 TRACE neutron.service AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-06 17:27:50.609 33185 TRACE neutron.service 

Version-Release number of selected component (if applicable):

openstack-neutron-2014.2.3-23.el7ost.noarch
openstack-neutron-common-2014.2.3-23.el7ost.noarch
openstack-neutron-ml2-2014.2.3-23.el7ost.noarch
openstack-neutron-openvswitch-2014.2.3-23.el7ost.noarch
python-neutron-2014.2.3-23.el7ost.noarch
python-neutronclient-2.3.9-1.el7ost.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:



Expected results:


Additional info:

Comment 2 Robin Cernin 2016-01-14 14:19:35 UTC
Hello,

Could we request possible backport of the fix to OSP6 / Juno ?

Thank you,
Regards,
Robin Černín

Comment 3 Robin Cernin 2016-01-14 15:58:25 UTC
Hello,

Also affects 

2016-01-13 16:36:02.116 48087 TRACE keystone.common.environment.eventlet_server AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-13 16:36:02.116 48089 TRACE keystone.common.environment.eventlet_server AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.

Thank you,
Regards,
Robin Černín

Comment 4 Jakub Libosvar 2016-01-15 08:19:45 UTC
(In reply to Robin Cernin from comment #3)
> Hello,
> 
> Also affects 
> 
> 2016-01-13 16:36:02.116 48087 TRACE
> keystone.common.environment.eventlet_server AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 2016-01-13 16:36:02.116 48089 TRACE
> keystone.common.environment.eventlet_server AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 
> Thank you,
> Regards,
> Robin Černín

You should clone this bug to keystone and ask them to backport https://review.openstack.org/#/c/160720/

Comment 5 Robin Cernin 2016-01-15 10:58:52 UTC
(In reply to Jakub Libosvar from comment #4)
> (In reply to Robin Cernin from comment #3)
> > Hello,
> > 
> > Also affects 
> > 
> > 2016-01-13 16:36:02.116 48087 TRACE
> > keystone.common.environment.eventlet_server AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 2016-01-13 16:36:02.116 48089 TRACE
> > keystone.common.environment.eventlet_server AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 
> > Thank you,
> > Regards,
> > Robin Černín
> 
> You should clone this bug to keystone and ask them to backport
> https://review.openstack.org/#/c/160720/

Cloned BZ for keystone to https://bugzilla.redhat.com/1298873

Comment 8 Toni Freger 2016-02-03 14:43:03 UTC
The code exists within the release.
openstack-neutron-2014.2.3-33.el7ost.noarch

Comment 10 errata-xmlrpc 2016-02-08 14:18:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0130.html


Note You need to log in before you can comment on or make changes to this bug.