Description of problem: During automated kernel testing, we hit an External Watchdog Expired due to "backend simple_recipe: ERROR Encoutnered problem while running task" Version-Release number of selected component (if applicable): Beaker Version 0.15.5 How reproducible: Unknown Although there were two instances at same time for separate recipes in same job. Steps to Reproduce: 1. Issue seen while running /kernel/RHEL6/drivers/3rd-party teask This task,by design, reboots the host multiple times Actual results: https://beaker.engineering.redhat.com/recipes/1246996#task19509926 http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/recipes/1246+/1246996/console.log -----<-snip->----- 2014-02-28 00:23:40,043 beah buildProtocol: INFO BackendListener: Connected. Address: UNIXSocket('') [-- MARK -- Fri Feb 28 05:25:00 2014] 2014-02-28 00:26:13,082 backend simple_recipe: ERROR Encoutnered problem while running task '19509926'. Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 518, in simple_recipe result = task.getResult() File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 605, in _deferGenerator result = g.next() File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 560, in _run_task if not thingy.getResult(): File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib64/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse response = xmlrpclib.loads(contents)[0][0] File "/usr/lib64/python2.6/xmlrpclib.py", line 1184, in loads return u.close(), u.getmethodname() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.AttributeError'>:SafeCookieTransport instance has no attribute 'close'"> -----<-snip->----- Expected results: no External Watchdog Additional info: Just an FYI... There is a typo in the error message echoed out to the console. *Encoutnered should be Encountered ;)
All, Second instance at same time frame seen here: https://beaker.engineering.redhat.com/recipes/1246996#task19509926 http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/recipes/1246+/1246996/console.log -----<-snip->----- 2014-02-28 00:23:40,043 beah buildProtocol: INFO BackendListener: Connected. Address: UNIXSocket('') [-- MARK -- Fri Feb 28 05:25:00 2014] 2014-02-28 00:26:13,082 backend simple_recipe: ERROR Encoutnered problem while running task '19509926'. Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 518, in simple_recipe result = task.getResult() File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 605, in _deferGenerator result = g.next() File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 560, in _run_task if not thingy.getResult(): File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib64/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse response = xmlrpclib.loads(contents)[0][0] File "/usr/lib64/python2.6/xmlrpclib.py", line 1184, in loads return u.close(), u.getmethodname() File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.AttributeError'>:SafeCookieTransport instance has no attribute 'close'"> -----<-snip->----- Best, -pbunyan
fwiw, I previously intermittently seen something similar when testing with RHEL4 and /kernel/drivers/3rd-party. Reference: https://bugzilla.redhat.com/show_bug.cgi?id=922252 Best, -pbunyan
(In reply to PaulB from comment #1) > Fault: <Fault 1: "<type 'exceptions.AttributeError'>:SafeCookieTransport > instance has no attribute 'close'"> The stack trace from /var/log/beaker/proxy.log on the lab controller confirms that this is indeed the same bug that was fixed with this patch: http://git.beaker-project.org/cgit/beaker/commit/?h=release-0.15&id=9efbef2f8e074e91b4cf7414e704c05e18179ace We expect this should be happening very rarely, because it's only a problem in a code path that is already handling unexpected errors. The fact that we're seeing it at all is probably due to the ongoing network problems in the BOS lab (connections from LC -> server being terminated mid-stream).
Beaker 0.16.0 has been released.