Bug 1071389

Summary:	beah fails to run task due to XML-RPC fault: AttributeError: SafeCookieTransport instance has no attribute 'close'
Product:	[Retired] Beaker	Reporter:	PaulB <pbunyan>
Component:	scheduler	Assignee:	Dan Callaghan <dcallagh>
Status:	CLOSED CURRENTRELEASE	QA Contact:	tools-bugs <tools-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	0.15	CC:	aigao, asaha, dcallagh, jburke, llim, pbunyan, rmancy, xjia
Target Milestone:	0.16	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-03-17 03:01:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description PaulB 2014-02-28 16:20:58 UTC

Description of problem:
 During automated kernel testing, we hit an External Watchdog Expired 
due to "backend simple_recipe: ERROR Encoutnered problem while running task"

Version-Release number of selected component (if applicable):
 Beaker Version 0.15.5

How reproducible:
 Unknown
 Although there were two instances at same time for 
 separate recipes in same job. 

Steps to Reproduce:
1. Issue seen while running /kernel/RHEL6/drivers/3rd-party teask
   This task,by design, reboots the host multiple times

Actual results:
https://beaker.engineering.redhat.com/recipes/1246996#task19509926
http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/recipes/1246+/1246996/console.log
-----<-snip->-----
2014-02-28 00:23:40,043 beah buildProtocol: INFO BackendListener: Connected.  Address: UNIXSocket('')  
[-- MARK -- Fri Feb 28 05:25:00 2014] 
2014-02-28 00:26:13,082 backend simple_recipe: ERROR Encoutnered problem while running task '19509926'.  
Traceback (most recent call last):  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 518, in simple_recipe  
    result = task.getResult()  
  File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 605, in _deferGenerator  
    result = g.next()  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 560, in _run_task  
    if not thingy.getResult():  
  File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib64/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse  
    response = xmlrpclib.loads(contents)[0][0]  
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1184, in loads  
    return u.close(), u.getmethodname()  
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close  
    raise Fault(**self._stack[0])  
Fault: <Fault 1: "<type 'exceptions.AttributeError'>:SafeCookieTransport instance has no attribute 'close'">  
-----<-snip->-----

Expected results:
 no External Watchdog 

Additional info:
 Just an FYI...
 There is a typo in the error message echoed out to the console.
 *Encoutnered should be Encountered ;)

Comment 1 PaulB 2014-02-28 16:22:03 UTC

All,

Second instance at same time frame seen here:
 https://beaker.engineering.redhat.com/recipes/1246996#task19509926
 http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/recipes/1246+/1246996/console.log
-----<-snip->-----
2014-02-28 00:23:40,043 beah buildProtocol: INFO BackendListener: Connected.  Address: UNIXSocket('')  
[-- MARK -- Fri Feb 28 05:25:00 2014] 
2014-02-28 00:26:13,082 backend simple_recipe: ERROR Encoutnered problem while running task '19509926'.  
Traceback (most recent call last):  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 518, in simple_recipe  
    result = task.getResult()  
  File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 605, in _deferGenerator  
    result = g.next()  
  File "/usr/lib/python2.6/site-packages/beah/backends/beakerlc.py", line 560, in _run_task  
    if not thingy.getResult():  
  File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 584, in getResult  
    self.result.raiseException()  
  File "/usr/lib64/python2.6/site-packages/twisted/web/xmlrpc.py", line 307, in parseResponse  
    response = xmlrpclib.loads(contents)[0][0]  
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1184, in loads  
    return u.close(), u.getmethodname()  
  File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close  
    raise Fault(**self._stack[0])  
Fault: <Fault 1: "<type 'exceptions.AttributeError'>:SafeCookieTransport instance has no attribute 'close'">  
-----<-snip->-----

Best,
-pbunyan

Comment 2 PaulB 2014-02-28 16:23:53 UTC

fwiw,
I previously intermittently seen something similar when testing with RHEL4 and /kernel/drivers/3rd-party.

Reference:
https://bugzilla.redhat.com/show_bug.cgi?id=922252

Best,
-pbunyan

Comment 4 Dan Callaghan 2014-03-04 22:18:46 UTC

(In reply to PaulB from comment #1)
> Fault: <Fault 1: "<type 'exceptions.AttributeError'>:SafeCookieTransport
> instance has no attribute 'close'">  

The stack trace from /var/log/beaker/proxy.log on the lab controller confirms that this is indeed the same bug that was fixed with this patch:

http://git.beaker-project.org/cgit/beaker/commit/?h=release-0.15&id=9efbef2f8e074e91b4cf7414e704c05e18179ace

We expect this should be happening very rarely, because it's only a problem in a code path that is already handling unexpected errors.

The fact that we're seeing it at all is probably due to the ongoing network problems in the BOS lab (connections from LC -> server being terminated mid-stream).

Comment 8 Dan Callaghan 2014-03-17 03:01:30 UTC

Beaker 0.16.0 has been released.