Bug 1164291

Summary: [BEAKER] Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>">
Product: [Retired] Beaker Reporter: PaulB <pbunyan>
Component: lab controllerAssignee: beaker-dev-list
Status: CLOSED DUPLICATE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 0.18CC: aigao, asaha, asavkov, bpeck, dcallagh, jburke, jstancek, pbunyan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-18 22:57:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description PaulB 2014-11-14 14:56:54 UTC 
Description of problem:
 During automated testing, task successfully completed but aborted with external watchdog due to:
Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>"> 

Version-Release number of selected component (if applicable):
 Beaker 0.18.4 

How reproducible:
 Unknown

Steps to Reproduce:
1.

Actual results:
https://beaker.engineering.redhat.com/recipes/1669050
https://beaker.engineering.redhat.com/recipes/1669050#task25946249
http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2014/11/8014/801427/1669050/console.log
-----<-snip->----- 
11/13/14 11:29:16  testID:25946249 start:  
/mnt/tests/kernel/RHEL6/buildcheck /mnt/tests/kernel/RHEL6/buildcheck  
-----<-snip->-----
2014-11-13 11:39:23,572 rhts_task checkin_finish: INFO resetting nohup  
11/13/14 11:39:23  testID:25946249 finish:  
-----<-snip->-----
2014-11-13 11:39:24,686 beah task_finished: INFO Task 5950eaa8-9b41-4b1c-81c1-9c65878cee1f has finished. Marking it as done.  
2014-11-13 11:39:24,688 backend async_proc: INFO Task 25946249 done. Submitting logs...  
2014-11-13 11:39:24,984 backend _send_cmd: INFO Command ['Command', 'forward', '57ee12b2-9b40-4feb-b10f-2581afea7af1', {'event': ['Event', 'extend_watchdog', '08e8ca6c-10b3-4a65-b50f-43d360cfba62', {'source': 'beah_beaker_backend', 'id': '5950eaa8-9b41-4b1c-81c1-9c65878cee1f'}, None, {'timeout': 14394}]}] sent.  
[-- MARK -- Thu Nov 13 16:40:00 2014] 
2014-11-13 11:40:46,322 backend _run_task: ERROR Exception raised when running task. Setting to finished  
Traceback (most recent call last):  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 598, in _run_task  
    thingy.getResult()  
  File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse  
    response = xmlrpclib.loads(contents)[0][0]  
  File "/usr/lib/python2.6/xmlrpclib.py", line 1184, in loads  
    return u.close(), u.getmethodname()  
  File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close  
    raise Fault(**self._stack[0])  
Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>">  
2014-11-13 11:40:46,370 backend simple_recipe: ERROR Encountered problem while running task '25946249'.  
Traceback (most recent call last):  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 517, in simple_recipe  
    result = task.getResult()  
  File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 605, in _deferGenerator  
    result = g.next()  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 598, in _run_task  
    thingy.getResult()  
  File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse  
    response = xmlrpclib.loads(contents)[0][0]  
  File "/usr/lib/python2.6/xmlrpclib.py", line 1184, in loads  
    return u.close(), u.getmethodname()  
  File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close  
    raise Fault(**self._stack[0])  
Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>">  
-----<-snip->-----

Expected results:
 No "500 INTERNAL SERVER ERROR" expected.

Additional info:
Comment 1 PaulB 2014-11-14 15:12:20 UTC 
All,
Issue also seen here:
RS:1321304
https://beaker.engineering.redhat.com/recipes/1669060

https://beaker.engineering.redhat.com/recipes/1669061 	
/kernel/filesystems/cifs/connectathon
http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2014/11/8014/801427/1669061/console.log
-----<-snip->-----
11/13/14 11:36:00  JobID:801427 Test:/kernel/filesystems/cifs/connectathon Response:1  
logger: /usr/bin/rhts-test-runner.sh rhts-test-update 127.0.0.1:7095 25946484 start  kernel-kernel-filesystems-cifs-connectathon  
2014-11-13 11:36:01,053 rhts_task checkin_start: INFO setting nohup  
11/13/14 11:36:01  testID:25946484 start: 
-----<-snip->-----
2014-11-13 11:38:26,569 beah task_finished: INFO Task 589b3243-0788-44b6-af36-43ca1de6d9cd has finished. Marking it as done.  
2014-11-13 11:38:26,570 backend async_proc: INFO Task 25946484 done. Submitting logs...  
2014-11-13 11:38:26,936 backend _send_cmd: INFO Command ['Command', 'forward', 'f00bf415-c458-4fa2-87c5-e9e96b49412c', {'event': ['Event', 'extend_watchdog', '374d34e9-4a6a-4a56-b542-8f1e4a5bae08', {'source': 'beah_beaker_backend', 'id': '589b3243-0788-44b6-af36-43ca1de6d9cd'}, None, {'timeout': 14394}]}] sent.  
2014-11-13 11:39:49,477 backend _run_task: ERROR Exception raised when running task. Setting to finished  
Traceback (most recent call last):  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 598, in _run_task  
    thingy.getResult()  
  File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse  
    response = xmlrpclib.loads(contents)[0][0]  
  File "/usr/lib/python2.6/xmlrpclib.py", line 1184, in loads  
    return u.close(), u.getmethodname()  
  File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close  
    raise Fault(**self._stack[0])  
Fault: <Fault 1: "<class 'xmlrpclib.ProtocolError'>:<ProtocolError for beaker.engineering.redhat.com/client/: 500 INTERNAL SERVER ERROR>">  
-----<-snip->-----

Best,
-pbunyan