Bug 1298598 - AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate
AssertionError: Calling waitall() from within one of the GreenPool's greenthr...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
6.0 (Juno)
Unspecified Unspecified
high Severity high
: async
: 6.0 (Juno)
Assigned To: Jakub Libosvar
Toni Freger
: ZStream
Depends On:
Blocks: 1298873
  Show dependency treegraph
 
Reported: 2016-01-14 09:10 EST by Robin Cernin
Modified: 2016-04-27 01:22 EDT (History)
8 users (show)

See Also:
Fixed In Version: openstack-neutron-2014.2.3-33.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1298873 (view as bug list)
Environment:
Last Closed: 2016-02-08 09:18:47 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1423250 None None None 2016-01-14 09:10 EST
OpenStack gerrit 157320 None None None 2016-01-15 03:12 EST

  None (edit)
Description Robin Cernin 2016-01-14 09:10:48 EST
Description of problem:

neutron]$ zgrep AssertionError server.log* 
2016-01-06 17:27:50.589 33160 INFO neutron.openstack.common.service [-] Caught SIGTERM, stopping children
2016-01-06 17:27:50.589 33160 INFO neutron.openstack.common.service [-] Waiting on 32 children to exit
2016-01-06 17:27:50.589 33207 INFO neutron.openstack.common.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Child caught SIGTERM, exiting
2016-01-06 17:27:50.589 33208 INFO neutron.openstack.common.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Child caught SIGTERM, exiting
...
2016-01-06 17:27:50.609 33185 ERROR neutron.service [req-b98fe8e2-7ef2-435e-ac2d-bf94a4cab49b None] Unrecoverable error: please check log for details.
2016-01-06 17:27:50.609 33185 TRACE neutron.service Traceback (most recent call last):
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 102, in serve_wsgi
2016-01-06 17:27:50.609 33185 TRACE neutron.service     service.start()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 73, in start
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self.wsgi_app = _run_wsgi(self.app_name)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 174, in _run_wsgi
2016-01-06 17:27:50.609 33185 TRACE neutron.service     workers=cfg.CONF.api_workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 224, in start
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._launch(application, workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 238, in _launch
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._server.launch_service(service, workers=workers)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 341, in launch_service
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._start_child(wrap)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 322, in _start_child
2016-01-06 17:27:50.609 33185 TRACE neutron.service     status, signo = self._child_wait_for_exit_or_signal(launcher)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 280, in _child_wait_for_exit_or_signal
2016-01-06 17:27:50.609 33185 TRACE neutron.service     launcher.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 128, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self.services.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 470, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     service.stop()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 117, in stop
2016-01-06 17:27:50.609 33185 TRACE neutron.service     self._server.kill()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 240, in kill
2016-01-06 17:27:50.609 33185 TRACE neutron.service     return kill(self, *throw_args)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 294, in kill
2016-01-06 17:27:50.609 33185 TRACE neutron.service     g.throw(*throw_args)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-01-06 17:27:50.609 33185 TRACE neutron.service     result = function(*args, **kwargs)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/wsgi.py", line 263, in _run
2016-01-06 17:27:50.609 33185 TRACE neutron.service     socket_timeout=self.client_socket_timeout)
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 842, in server
2016-01-06 17:27:50.609 33185 TRACE neutron.service     pool.waitall()
2016-01-06 17:27:50.609 33185 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 117, in waitall
2016-01-06 17:27:50.609 33185 TRACE neutron.service     "Calling waitall() from within one of the " \
2016-01-06 17:27:50.609 33185 TRACE neutron.service AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-06 17:27:50.609 33185 TRACE neutron.service 

Version-Release number of selected component (if applicable):

openstack-neutron-2014.2.3-23.el7ost.noarch
openstack-neutron-common-2014.2.3-23.el7ost.noarch
openstack-neutron-ml2-2014.2.3-23.el7ost.noarch
openstack-neutron-openvswitch-2014.2.3-23.el7ost.noarch
python-neutron-2014.2.3-23.el7ost.noarch
python-neutronclient-2.3.9-1.el7ost.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:



Expected results:


Additional info:
Comment 2 Robin Cernin 2016-01-14 09:19:35 EST
Hello,

Could we request possible backport of the fix to OSP6 / Juno ?

Thank you,
Regards,
Robin Černín
Comment 3 Robin Cernin 2016-01-14 10:58:25 EST
Hello,

Also affects 

2016-01-13 16:36:02.116 48087 TRACE keystone.common.environment.eventlet_server AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-13 16:36:02.116 48089 TRACE keystone.common.environment.eventlet_server AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.

Thank you,
Regards,
Robin Černín
Comment 4 Jakub Libosvar 2016-01-15 03:19:45 EST
(In reply to Robin Cernin from comment #3)
> Hello,
> 
> Also affects 
> 
> 2016-01-13 16:36:02.116 48087 TRACE
> keystone.common.environment.eventlet_server AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 2016-01-13 16:36:02.116 48089 TRACE
> keystone.common.environment.eventlet_server AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling
> waitall() from within one of the GreenPool's greenthreads will never
> terminate.
> 
> Thank you,
> Regards,
> Robin Černín

You should clone this bug to keystone and ask them to backport https://review.openstack.org/#/c/160720/
Comment 5 Robin Cernin 2016-01-15 05:58:52 EST
(In reply to Jakub Libosvar from comment #4)
> (In reply to Robin Cernin from comment #3)
> > Hello,
> > 
> > Also affects 
> > 
> > 2016-01-13 16:36:02.116 48087 TRACE
> > keystone.common.environment.eventlet_server AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 2016-01-13 16:36:02.116 48089 TRACE
> > keystone.common.environment.eventlet_server AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 2016-01-13 16:36:02.119 48087 CRITICAL keystone [-] AssertionError: Calling
> > waitall() from within one of the GreenPool's greenthreads will never
> > terminate.
> > 
> > Thank you,
> > Regards,
> > Robin Černín
> 
> You should clone this bug to keystone and ask them to backport
> https://review.openstack.org/#/c/160720/

Cloned BZ for keystone to https://bugzilla.redhat.com/1298873
Comment 8 Toni Freger 2016-02-03 09:43:03 EST
The code exists within the release.
openstack-neutron-2014.2.3-33.el7ost.noarch
Comment 10 errata-xmlrpc 2016-02-08 09:18:47 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0130.html

Note You need to log in before you can comment on or make changes to this bug.