Description of problem: During automated testing, task successfully completed but aborted with external watchdog due to: Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>"> Version-Release number of selected component (if applicable): Beaker 0.18.4 How reproducible: Unknown Steps to Reproduce: 1. Actual results: https://beaker.engineering.redhat.com/recipes/1669050 https://beaker.engineering.redhat.com/recipes/1669050#task25946249 http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2014/11/8014/801427/1669050/console.log -----<-snip->----- 11/13/14 11:29:16 testID:25946249 start: /mnt/tests/kernel/RHEL6/buildcheck /mnt/tests/kernel/RHEL6/buildcheck -----<-snip->----- 2014-11-13 11:39:23,572 rhts_task checkin_finish: INFO resetting nohup 11/13/14 11:39:23 testID:25946249 finish: -----<-snip->----- 2014-11-13 11:39:24,686 beah task_finished: INFO Task 5950eaa8-9b41-4b1c-81c1-9c65878cee1f has finished. Marking it as done. 2014-11-13 11:39:24,688 backend async_proc: INFO Task 25946249 done. Submitting logs... 2014-11-13 11:39:24,984 backend _send_cmd: INFO Command ['Command', 'forward', '57ee12b2-9b40-4feb-b10f-2581afea7af1', {'event': ['Event', 'extend_watchdog', '08e8ca6c-10b3-4a65-b50f-43d360cfba62', {'source': 'beah_beaker_backend', 'id': '5950eaa8-9b41-4b1c-81c1-9c65878cee1f'}, None, {'timeout': 14394}]}] sent. [-- MARK -- Thu Nov 13 16:40:00 2014] 2014-11-13 11:40:46,322 backend _run_task: ERROR Exception raised when running task. Setting to finished Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 598, in _run_task thingy.getResult() File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse response = xmlrpclib.loads(contents)[0][0] File "/usr/lib/python2.6/xmlrpclib.py", line 1184, in loads return u.close(), u.getmethodname() File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>"> 2014-11-13 11:40:46,370 backend simple_recipe: ERROR Encountered problem while running task '25946249'. Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 517, in simple_recipe result = task.getResult() File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 605, in _deferGenerator result = g.next() File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 598, in _run_task thingy.getResult() File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse response = xmlrpclib.loads(contents)[0][0] File "/usr/lib/python2.6/xmlrpclib.py", line 1184, in loads return u.close(), u.getmethodname() File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>"> -----<-snip->----- Expected results: No "500 INTERNAL SERVER ERROR" expected. Additional info:
All, Issue also seen here: RS:1321304 https://beaker.engineering.redhat.com/recipes/1669060 https://beaker.engineering.redhat.com/recipes/1669061 /kernel/filesystems/cifs/connectathon http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2014/11/8014/801427/1669061/console.log -----<-snip->----- 11/13/14 11:36:00 JobID:801427 Test:/kernel/filesystems/cifs/connectathon Response:1 logger: /usr/bin/rhts-test-runner.sh rhts-test-update 127.0.0.1:7095 25946484 start kernel-kernel-filesystems-cifs-connectathon 2014-11-13 11:36:01,053 rhts_task checkin_start: INFO setting nohup 11/13/14 11:36:01 testID:25946484 start: -----<-snip->----- 2014-11-13 11:38:26,569 beah task_finished: INFO Task 589b3243-0788-44b6-af36-43ca1de6d9cd has finished. Marking it as done. 2014-11-13 11:38:26,570 backend async_proc: INFO Task 25946484 done. Submitting logs... 2014-11-13 11:38:26,936 backend _send_cmd: INFO Command ['Command', 'forward', 'f00bf415-c458-4fa2-87c5-e9e96b49412c', {'event': ['Event', 'extend_watchdog', '374d34e9-4a6a-4a56-b542-8f1e4a5bae08', {'source': 'beah_beaker_backend', 'id': '589b3243-0788-44b6-af36-43ca1de6d9cd'}, None, {'timeout': 14394}]}] sent. 2014-11-13 11:39:49,477 backend _run_task: ERROR Exception raised when running task. Setting to finished Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 598, in _run_task thingy.getResult() File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult self.result.raiseException() File "/usr/lib/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse response = xmlrpclib.loads(contents)[0][0] File "/usr/lib/python2.6/xmlrpclib.py", line 1184, in loads return u.close(), u.getmethodname() File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>"> -----<-snip->----- Best, -pbunyan