Bug 720890

Summary:	beakerd threads should log a traceback if they die
Product:	[Retired] Beaker	Reporter:	Dan Callaghan <dcallagh>
Component:	scheduler	Assignee:	Dan Callaghan <dcallagh>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	0.6	CC:	bpeck, dcallagh, mcsontos, rmancy, stl
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-07-28 23:50:16 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Dan Callaghan 2011-07-13 07:00:14 UTC

There is nothing supervising the various worker threads which run inside beakerd, so that if any of them dies due to an unhandled exception the traceback will be written to stderr (so, lost forever) and we will never know what went wrong.

Kobo has a decorator (kobo.decorators.log_traceback) which will write a traceback to a log file if an exception is raised out of the decorated function. We don't want to depend on kobo in Beaker server, but we could adapt the idea and write our own decorator for the beakerd worker threads.

Comment 1 Dan Callaghan 2011-07-14 00:47:07 UTC

Another interesting idea from Kobo is to override sys.excepthook (I didn't realise that was possible). See kobo.tback.set_except_hook. We already use this in lab controller code.

Unfortunately I don't think we can use sys.excepthook for beakerd's worker threads, since apparently it's not called for uncaught exceptions in threads [1].

[1] http://bugs.python.org/issue1230540

Comment 3 Raymond Mancy 2011-07-26 23:48:34 UTC

2011-07-26 19:36:06,287 beakerd ERROR Uncaught exception in new_recipes_loop
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/bkr/server/util.py", line 140, in decorated
    func(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bkr/server/tools/beakerd.py", line 603, in new_recipes_loop
    if not new_recipes():
  File "/usr/lib/python2.4/site-packages/bkr/server/tools/beakerd.py", line 79, in new_recipes
    raise Exception()
Exception