Bug 998714 - python: pycurl not deallocated properly during interpreter exit
Summary: python: pycurl not deallocated properly during interpreter exit
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Fedora
Classification: Fedora
Component: python
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Bohuslav "Slavek" Kabrda
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-19 20:31 UTC by Tomas Hoger
Modified: 2013-09-18 19:49 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-09-18 19:49:54 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
xmlrpclib pycurl transport (2.82 KB, text/plain)
2013-08-19 20:33 UTC, Tomas Hoger
no flags Details
test that does not trigger the problem (1.18 KB, text/plain)
2013-08-19 20:43 UTC, Tomas Hoger
no flags Details
better small tet (1.25 KB, text/plain)
2013-08-19 21:38 UTC, Tomas Hoger
no flags Details

Description Tomas Hoger 2013-08-19 20:31:10 UTC
Description of problem:
pycurl is not always deallocated correctly when program using pycurl exits.  That causes curl_easy_cleanup() not to get called, which results in cookies not getting written to a specified COOKIEJAR file.

Attaching a script which reproduces the problem.  It implements xmlrpclib transport using pycurl.  It talks to RH bugzilla to log in, but the authentication cookie is not saved to the COOKIEJAR file.  I verified with gdb that do_curl_dealloc() does not get called.

I noticed that disabling HEADERFUNCTION callback fixes the problem.  I do not believe the callback function in the script is incorrect.  Using the same in a different / more simple script does not break module deallocation.

Version-Release number of selected component (if applicable):
python-pycurl-7.19.0-15.1.fc19.x86_64
libcurl-7.29.0-7.fc19.x86_64
python-2.7.5-3.fc19.x86_64

Comment 1 Tomas Hoger 2013-08-19 20:33:13 UTC
Created attachment 788197 [details]
xmlrpclib pycurl transport

This script demonstrates the issue.  Pass your BZ user name as argument.  Check cookies.txt file content after it exits (and empty it before running the test).

Comment 2 Tomas Hoger 2013-08-19 20:43:31 UTC
Created attachment 788198 [details]
test that does not trigger the problem

Comment 3 Tomas Hoger 2013-08-19 21:38:28 UTC
Created attachment 788208 [details]
better small tet

I realized that my minimal test case did not match xmlrpclib one very well.  This one should be better, and it does trigger the problem.

I suspect the problem here actually comes from the script.  It seems to create a loop in python object references (CURLHTTPRequest refers to pycurl.Curl instance and that instance refers back to the CURLHTTPRequest instance via reference to its method), which probably blocks garbage collection and proper module cleanup.

Yeah, I won't be surprised if this is closed:notabug.

Comment 4 Kamil Dudka 2013-08-27 15:02:15 UTC
(In reply to Tomas Hoger from comment #3)
> I suspect the problem here actually comes from the script.  It seems to
> create a loop in python object references (CURLHTTPRequest refers to
> pycurl.Curl instance and that instance refers back to the CURLHTTPRequest
> instance via reference to its method), which probably blocks garbage
> collection and proper module cleanup.

This is an issue with the python interpreter.  Please consider calling the close() method on the curl object explicitly to work around the problem.

Comment 5 Tomas Hoger 2013-09-18 19:49:54 UTC
(In reply to Kamil Dudka from comment #4)
> This is an issue with the python interpreter.  Please consider calling the
> close() method on the curl object explicitly to work around the problem.

Calling close() is not really practical, as that would require changing several scripts that use a module that uses pycurl.  I worked around by avoiding mentioned loop, which fixed the problem for me.  I don't believe this is the right place to request python garbage collector change, so I'm closing this.


Note You need to log in before you can comment on or make changes to this bug.