Bug 832328 - beaker-provision does not reliably shut down
beaker-provision does not reliably shut down
Status: CLOSED CURRENTRELEASE
Product: Beaker
Classification: Community
Component: lab controller (Show other bugs)
0.9
Unspecified Unspecified
unspecified Severity unspecified (vote)
: 0.9.0
: ---
Assigned To: Dan Callaghan
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-15 03:52 EDT by Dan Callaghan
Modified: 2012-06-26 02:41 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-26 02:41:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan Callaghan 2012-06-15 03:52:47 EDT
If beaker-provision is sent SIGTERM (which is how the init script attempts to shut it down) while under heavy load, the signal handler could raise ShutdownException in the wrong greenlet:

2012-06-15 01:50:53 [ERROR   ] {19685} bkr.labcontroller.provision:85 Error processing command 42442
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 78, in handle
    handle_configure_netboot(command)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 126, in handle_configure_netboot
    command['netboot']['initrd_url'], command['fqdn'])
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 153, in fetch_images
    siphon(urllib2.urlopen(initrd_url), dest)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 45, in siphon
    dest.write(chunk)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 182, in shutdown_handler
    raise ShutdownException() # gevent prints this to stderr, just ignore it
ShutdownException

Normally it is expected to be raised in the main greenlet, where it is handled correctly. If it raises in the wrong greenlet then the daemon will never shut down cleanly.

The fix is to use an event to signal shutdown, rather than raising an exception.
Comment 1 Dan Callaghan 2012-06-15 03:55:56 EDT
On Gerrit: http://gerrit.beaker-project.org/1142
Comment 3 Dan Callaghan 2012-06-26 02:41:23 EDT
Beaker 0.9.0 has been released.

Note You need to log in before you can comment on or make changes to this bug.