When a recipe is in Reserved state, its tasks are all already completed and no new results can be recorded. The server currently enforces this. However, if beaker-watchdog detects a panic while the recipe is Reserved it will try to report that to the server (appending a Panic result to the final task, and then reducing the watchdog to 10 minutes). That fails due to some server-side sanity checking: bkr.labcontroller.proxy ERROR Failed to run monitor for dev-kvm-guest-08.rhts.eng.bos.redhat.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 552, in run updated |= monitor.run() File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 617, in run return self.console_watch.update() File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 320, in update self.process_log(block) File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 277, in process_log self.proxy.report_panic(self.watchdog, panic_found) File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 639, in report_panic self.task_result(task()['id'], 'panic', '/', 0, panic_message) File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 155, in task_result result_summary) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 563, in request result = transport_class.request(self, *args, **kwargs) File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 419, in _request return self.parse_response(h.getfile()) File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response return self._parse_response(file, None) File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response return u.close() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.ValueError'>:Cannot record result for finished task T:133753"> We just need to tidy up beaker-watchdog so that it doesn't try and fail to report the panic when the recipe is Reserved.
I think the code path in beaker-watchdog which handles this should already have access to the recipe status. If status == Reserved then ignore the panic. Not sure if there is any cleaner way to handle it besides just special-casing the Reserved status like that...
I'm not sure if I have understood this correctly. But I dont think that we have access to the recipe status from report_panic in proxy.py (I'm not sure if this is what you mean by "the code path in beaker-watchdog which handles this"). The recipe element is an XML element and the tasks status is retrievable, but I'm unsure about how to get the overall recipe status. I put in a patch for review nevertheless as it might help see what I was attempting to do: https://gerrit.beaker-project.org/#/c/5796/
Beaker 24.4 has been released.