Bug 832250 - beaker-provision has a race between creating and reaping child processes
beaker-provision has a race between creating and reaping child processes
Status: CLOSED CURRENTRELEASE
Product: Beaker
Classification: Community
Component: lab controller (Show other bugs)
0.9
Unspecified Unspecified
unspecified Severity unspecified (vote)
: 0.9.0
: ---
Assigned To: Dan Callaghan
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-14 21:41 EDT by Dan Callaghan
Modified: 2012-06-26 02:40 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-26 02:40:55 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan Callaghan 2012-06-14 21:41:30 EDT
The following exceptions were observed:

2012-06-14 14:07:05 [ERROR   ] { 9879} bkr.labcontroller.provision:85 Error processing command 3323
Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line
73, in handle
     handle_power(dict(command.items() + [('action', u'off')]))
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 168, in handle_power
     timeout=300)
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/async.py", line 78, in __init__
     super(MonitoredSubprocess, self).__init__(*args, **kwargs)
   File "/usr/lib64/python2.6/subprocess.py", line 639, in __init__
     errread, errwrite)
   File "/usr/lib64/python2.6/subprocess.py", line 1138, in _execute_child
     self.pid = os.fork()
   File "/usr/lib64/python2.6/site-packages/gevent/hub.py", line 162, in fork
     result = _original_fork()
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/async.py", line 90, in _sigchld_handler
     if child.poll() is not None:
   File "/usr/lib64/python2.6/subprocess.py", line 755, in poll
     return self._internal_poll()
   File "/usr/lib64/python2.6/subprocess.py", line 1256, in _internal_poll
     pid, sts = _waitpid(self.pid, _WNOHANG)
TypeError: an integer is required

2012-06-14 11:22:21 [ERROR   ] {10593} bkr.labcontroller.provision:85 Error processing command 3346
Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line
74, in handle
     handle_power(dict(command.items() + [('action', u'on')]))
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line
168, in handle_power
     timeout=300)
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/async.py", line 78, in __init__
     super(MonitoredSubprocess, self).__init__(*args, **kwargs)
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/async.py", line 90, in _sigchld_handler
     if child.poll() is not None:
   File "/usr/lib64/python2.6/subprocess.py", line 755, in poll
     return self._internal_poll()
   File "/usr/lib64/python2.6/subprocess.py", line 1254, in _internal_poll
     if self.returncode is None:
AttributeError: 'MonitoredSubprocess' object has no attribute 'returncode'

In both cases it is because we are calling .poll() on a MonitoredSubprocess instance which has not been fully initialised yet.
Comment 1 Dan Callaghan 2012-06-14 22:48:02 EDT
I have a test which reliably reproduces both of those exceptions (you get one or the other at random).

I think the best fix is to move the actual work out of the SIGCHLD handler and into a separate greenlet that is scheduled as normal. Then it does not race with or interfere with anything else. And if it does raise for some reason, it won't screw up the whole gevent hub. In retrospect, I should have done it that way originally.

http://gerrit.beaker-project.org/1138

Ultimately (when gevent 1.0 is released) we should get away from installing our own SIGCHLD handler at all, and use gevent's child handling which is properly tied in to the libev event loop etc.
Comment 3 Dan Callaghan 2012-06-26 02:40:55 EDT
Beaker 0.9.0 has been released.

Note You need to log in before you can comment on or make changes to this bug.