Bug 998714

Summary: python: pycurl not deallocated properly during interpreter exit
Product: [Fedora] Fedora Reporter: Tomas Hoger <thoger>
Component: pythonAssignee: Bohuslav "Slavek" Kabrda <bkabrda>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: bkabrda, dmalcolm, ivazqueznet, jonathansteffan, kdudka, mstuchli, tomspur
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-18 19:49:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
xmlrpclib pycurl transport
none
test that does not trigger the problem
none
better small tet none

Description Tomas Hoger 2013-08-19 20:31:10 UTC 
Description of problem:
pycurl is not always deallocated correctly when program using pycurl exits.  That causes curl_easy_cleanup() not to get called, which results in cookies not getting written to a specified COOKIEJAR file.

Attaching a script which reproduces the problem.  It implements xmlrpclib transport using pycurl.  It talks to RH bugzilla to log in, but the authentication cookie is not saved to the COOKIEJAR file.  I verified with gdb that do_curl_dealloc() does not get called.

I noticed that disabling HEADERFUNCTION callback fixes the problem.  I do not believe the callback function in the script is incorrect.  Using the same in a different / more simple script does not break module deallocation.

Version-Release number of selected component (if applicable):
python-pycurl-7.19.0-15.1.fc19.x86_64
libcurl-7.29.0-7.fc19.x86_64
python-2.7.5-3.fc19.x86_64
Comment 1 Tomas Hoger 2013-08-19 20:33:13 UTC 
Created attachment 788197 [details]
xmlrpclib pycurl transport

This script demonstrates the issue.  Pass your BZ user name as argument.  Check cookies.txt file content after it exits (and empty it before running the test).
Comment 2 Tomas Hoger 2013-08-19 20:43:31 UTC 
Created attachment 788198 [details]
test that does not trigger the problem
Comment 3 Tomas Hoger 2013-08-19 21:38:28 UTC 
Created attachment 788208 [details]
better small tet

I realized that my minimal test case did not match xmlrpclib one very well.  This one should be better, and it does trigger the problem.

I suspect the problem here actually comes from the script.  It seems to create a loop in python object references (CURLHTTPRequest refers to pycurl.Curl instance and that instance refers back to the CURLHTTPRequest instance via reference to its method), which probably blocks garbage collection and proper module cleanup.

Yeah, I won't be surprised if this is closed:notabug.
Comment 4 Kamil Dudka 2013-08-27 15:02:15 UTC 
(In reply to Tomas Hoger from comment #3)
> I suspect the problem here actually comes from the script.  It seems to
> create a loop in python object references (CURLHTTPRequest refers to
> pycurl.Curl instance and that instance refers back to the CURLHTTPRequest
> instance via reference to its method), which probably blocks garbage
> collection and proper module cleanup.

This is an issue with the python interpreter.  Please consider calling the close() method on the curl object explicitly to work around the problem.
Comment 5 Tomas Hoger 2013-09-18 19:49:54 UTC 
(In reply to Kamil Dudka from comment #4)
> This is an issue with the python interpreter.  Please consider calling the
> close() method on the curl object explicitly to work around the problem.

Calling close() is not really practical, as that would require changing several scripts that use a module that uses pycurl.  I worked around by avoiding mentioned loop, which fixed the problem for me.  I don't believe this is the right place to request python garbage collector change, so I'm closing this.