Hide Forgot
There is nothing supervising the various worker threads which run inside beakerd, so that if any of them dies due to an unhandled exception the traceback will be written to stderr (so, lost forever) and we will never know what went wrong. Kobo has a decorator (kobo.decorators.log_traceback) which will write a traceback to a log file if an exception is raised out of the decorated function. We don't want to depend on kobo in Beaker server, but we could adapt the idea and write our own decorator for the beakerd worker threads.
Another interesting idea from Kobo is to override sys.excepthook (I didn't realise that was possible). See kobo.tback.set_except_hook. We already use this in lab controller code. Unfortunately I don't think we can use sys.excepthook for beakerd's worker threads, since apparently it's not called for uncaught exceptions in threads [1]. [1] http://bugs.python.org/issue1230540
2011-07-26 19:36:06,287 beakerd ERROR Uncaught exception in new_recipes_loop Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/bkr/server/util.py", line 140, in decorated func(*args, **kwargs) File "/usr/lib/python2.4/site-packages/bkr/server/tools/beakerd.py", line 603, in new_recipes_loop if not new_recipes(): File "/usr/lib/python2.4/site-packages/bkr/server/tools/beakerd.py", line 79, in new_recipes raise Exception() Exception