Bug 998714 - python: pycurl not deallocated properly during interpreter exit
python: pycurl not deallocated properly during interpreter exit
Status: CLOSED WORKSFORME
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Bohuslav "Slavek" Kabrda
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-19 16:31 EDT by Tomas Hoger
Modified: 2013-09-18 15:49 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-18 15:49:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
xmlrpclib pycurl transport (2.82 KB, text/plain)
2013-08-19 16:33 EDT, Tomas Hoger
no flags Details
test that does not trigger the problem (1.18 KB, text/plain)
2013-08-19 16:43 EDT, Tomas Hoger
no flags Details
better small tet (1.25 KB, text/plain)
2013-08-19 17:38 EDT, Tomas Hoger
no flags Details

  None (edit)
Description Tomas Hoger 2013-08-19 16:31:10 EDT
Description of problem:
pycurl is not always deallocated correctly when program using pycurl exits.  That causes curl_easy_cleanup() not to get called, which results in cookies not getting written to a specified COOKIEJAR file.

Attaching a script which reproduces the problem.  It implements xmlrpclib transport using pycurl.  It talks to RH bugzilla to log in, but the authentication cookie is not saved to the COOKIEJAR file.  I verified with gdb that do_curl_dealloc() does not get called.

I noticed that disabling HEADERFUNCTION callback fixes the problem.  I do not believe the callback function in the script is incorrect.  Using the same in a different / more simple script does not break module deallocation.

Version-Release number of selected component (if applicable):
python-pycurl-7.19.0-15.1.fc19.x86_64
libcurl-7.29.0-7.fc19.x86_64
python-2.7.5-3.fc19.x86_64
Comment 1 Tomas Hoger 2013-08-19 16:33:13 EDT
Created attachment 788197 [details]
xmlrpclib pycurl transport

This script demonstrates the issue.  Pass your BZ user name as argument.  Check cookies.txt file content after it exits (and empty it before running the test).
Comment 2 Tomas Hoger 2013-08-19 16:43:31 EDT
Created attachment 788198 [details]
test that does not trigger the problem
Comment 3 Tomas Hoger 2013-08-19 17:38:28 EDT
Created attachment 788208 [details]
better small tet

I realized that my minimal test case did not match xmlrpclib one very well.  This one should be better, and it does trigger the problem.

I suspect the problem here actually comes from the script.  It seems to create a loop in python object references (CURLHTTPRequest refers to pycurl.Curl instance and that instance refers back to the CURLHTTPRequest instance via reference to its method), which probably blocks garbage collection and proper module cleanup.

Yeah, I won't be surprised if this is closed:notabug.
Comment 4 Kamil Dudka 2013-08-27 11:02:15 EDT
(In reply to Tomas Hoger from comment #3)
> I suspect the problem here actually comes from the script.  It seems to
> create a loop in python object references (CURLHTTPRequest refers to
> pycurl.Curl instance and that instance refers back to the CURLHTTPRequest
> instance via reference to its method), which probably blocks garbage
> collection and proper module cleanup.

This is an issue with the python interpreter.  Please consider calling the close() method on the curl object explicitly to work around the problem.
Comment 5 Tomas Hoger 2013-09-18 15:49:54 EDT
(In reply to Kamil Dudka from comment #4)
> This is an issue with the python interpreter.  Please consider calling the
> close() method on the curl object explicitly to work around the problem.

Calling close() is not really practical, as that would require changing several scripts that use a module that uses pycurl.  I worked around by avoiding mentioned loop, which fixed the problem for me.  I don't believe this is the right place to request python garbage collector change, so I'm closing this.

Note You need to log in before you can comment on or make changes to this bug.