Bug 1340566

Summary: beaker-watchdog might try (and fail) to report a panic for a Reserved recipe
Product: [Retired] Beaker Reporter: Dan Callaghan <dcallagh>
Component: generalAssignee: Anwesha Chatterjee <achatter>
Status: CLOSED CURRENTRELEASE QA Contact: Dan Callaghan <dcallagh>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: achatter, dcallagh, dowang, mjia, rjoost
Target Milestone: 24.4Keywords: EasyFix, Patch, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-03 03:57:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Callaghan 2016-05-27 22:27:45 UTC
When a recipe is in Reserved state, its tasks are all already completed and no new results can be recorded. The server currently enforces this.

However, if beaker-watchdog detects a panic while the recipe is Reserved it will try to report that to the server (appending a Panic result to the final task, and then reducing the watchdog to 10 minutes).

That fails due to some server-side sanity checking:

bkr.labcontroller.proxy ERROR Failed to run monitor for dev-kvm-guest-08.rhts.eng.bos.redhat.com
 Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 552, in run
     updated |= monitor.run()
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 617, in run
     return self.console_watch.update()
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 320, in update
     self.process_log(block)
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 277, in process_log
     self.proxy.report_panic(self.watchdog, panic_found)
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 639, in report_panic
     self.task_result(task()['id'], 'panic', '/', 0, panic_message)
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 155, in task_result
     result_summary)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
     return self.__send(self.__name, args)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
     verbose=self.__verbose
   File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 563, in request
     result = transport_class.request(self, *args, **kwargs)
   File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 419, in _request
     return self.parse_response(h.getfile())
   File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response
     return self._parse_response(file, None)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
     return u.close()
   File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
     raise Fault(**self._stack[0])
 Fault: <Fault 1: "<type 'exceptions.ValueError'>:Cannot record result for finished task T:133753">

We just need to tidy up beaker-watchdog so that it doesn't try and fail to report the panic when the recipe is Reserved.

Comment 1 Dan Callaghan 2017-01-12 03:16:49 UTC
I think the code path in beaker-watchdog which handles this should already have access to the recipe status. If status == Reserved then ignore the panic. Not sure if there is any cleaner way to handle it besides just special-casing the Reserved status like that...

Comment 2 Anwesha Chatterjee 2017-08-28 06:28:42 UTC
I'm not sure if I have understood this correctly. But I dont think that we have access to the recipe status from report_panic in proxy.py (I'm not sure if this is what you mean by "the code path in beaker-watchdog which handles this"). 

The recipe element is an XML element and the tasks status is retrievable, but I'm unsure about how to get the overall recipe status.

I put in a patch for review nevertheless as it might help see what I was attempting to do:

https://gerrit.beaker-project.org/#/c/5796/

Comment 5 Dan Callaghan 2017-10-03 03:57:38 UTC
Beaker 24.4 has been released.