Bug 580963

Summary: python-nss does not release the interpreter lock on blocking operations
Product: [Fedora] Fedora Reporter: Miloslav Trmač <mitr>
Component: python-nssAssignee: John Dennis <jdennis>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 12CC: dcantrell, fschwarz, jdennis
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: python-nss-0.9-8.fc12 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-09-08 02:27:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Release the Python GIL for operations that might block
none
A few more GIL changes none

Description Miloslav Trmač 2010-04-09 15:31:24 UTC
AFAICS python-nss-0.8 does not manipulate the Python global interpreter lock, making it impossible to perform more than one I/O operation through NSS using threads, or in fact to perform _any_ operation in a thread if another thread is blocked in a NSS operation.

See http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock .  This is affecting my attempts to speed up the Fedora signing server.

Comment 1 John Dennis 2010-04-12 14:35:53 UTC
Yes, you're correct, python-nss is not releasing the GIL during blocking IO operations. I'm in the process of adding a bunch of new features to python-nss which should be complete in about a week. I'll make sure to add the thread support in the next release.

Comment 2 Miloslav Trmač 2010-05-13 15:41:43 UTC
Created attachment 413799 [details]
Release the Python GIL for operations that might block

This patch against python-nss-0.8 seems to work fine for me.

Comment 3 John Dennis 2010-05-13 15:55:13 UTC
Thank you for the patch, I truly appreciate it, but you really should have checked with me first before doing this work. It's already been implemented so I probably won't take advantage of the patch, plus I doubt it would apply cleanly because there has been a lot of updates to the code. Once I have the fixes in for the cyclic garbage collector I'll give you a pointer to a test package.

Comment 4 Miloslav Trmač 2010-05-13 18:54:11 UTC
Created attachment 413861 [details]
A few more GIL changes

This is the result of merging the above patch against current CVS, releasing the lock in a few places, and - more importantly - acquiring the lock in callbacks from C to Python.

Comment 5 John Dennis 2010-05-13 20:01:39 UTC
Thank you for the patch Miloslav. I appreciate it. Looks good. Thank you for addressing the issue with enusring the GIL state in the callback, that was a critical component I had missed.

The patch has been applied. I'll ping you with a pointer to a test build early next week.

Comment 6 John Dennis 2010-05-28 20:32:31 UTC
I have prepared a new python-nss package which includes your suggested fixes as well as fixes for reference counting, thread local storage, cyclic garbage collection, and a host of new features. The release is 0.9. I have built it in rawhide (http://koji.fedoraproject.org/koji/buildinfo?buildID=175538).

I have extensively tested the new version with valgrind, with a special python interpreter build which adds reference counting assertions, and using a script I wrote to verify there are exact matching deallocs for each new object created. There are no objects uncollected, no excessive decref's, no memory accessed when it shouldn't be. Of course I may have missed something.

I would appreciate if you could both test the package and do a source code review and provide feedback. Thank you!

Comment 7 Fedora Update System 2010-07-16 17:14:55 UTC
python-nss-0.9-8.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/python-nss-0.9-8.fc13

Comment 8 Fedora Update System 2010-07-16 17:16:03 UTC
python-nss-0.9-8.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/python-nss-0.9-8.fc12

Comment 9 Fedora Update System 2010-09-08 02:26:54 UTC
python-nss-0.9-8.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 10 Fedora Update System 2010-11-01 20:55:45 UTC
python-nss-0.9-8.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.