Bug 472690

Summary: m2crypto blocks python thread
Product: Red Hat Enterprise Linux 5 Reporter: Dan Kenigsberg <danken>
Component: m2cryptoAssignee: Miloslav Trmač <mitr>
Status: CLOSED ERRATA QA Contact: BaseOS QE <qe-baseos-auto>
Severity: high Docs Contact:
Priority: medium    
Version: 5.3CC: apevec, mjc, ohudlick, sgrubb, zmraz
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 489296 (view as bug list) Environment:
Last Closed: 2009-09-02 11:19:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 493327    
Description Flags
Backport the 0.18 and 0.19 GILState / ALLOW_THREADS changes
Updated backported patch (very lightly tested) none

Description Dan Kenigsberg 2008-11-23 14:30:43 UTC
Description of problem:
version 0.16 has various bugs that where solved in later versions.
for example, there was a case where it took the global thread lock and forgot to release it (corrected in 0.18) and a case where it made python segfault (corrected in 0.19)

please see "Fixed threading regressions introduced in 0.16, by Aaron Reizes and Keith Jackson" and "EVP.PKey.get_rsa() now returns RSA_pub, which fixess egmentation fault when trying to encrypt using public key from X509 certificate, by Ben Timby" on http://svn.osafoundation.org/m2crypto/trunk/CHANGES for probable corrections.

Version-Release number of selected component (if applicable):

Comment 1 Dan Kenigsberg 2008-11-30 15:21:26 UTC
mitr, according to your suggestion I am breaking this into two specific bug reports.

the first allows very simple (yet nasty) DoS attack on python applications using m2crypto. More details and patch are available upstream

BTW, is is too late to fix it for RHEL5.3 ?

Comment 2 Miloslav Trmač 2008-12-04 17:05:33 UTC
Thanks for your report.

So, blocking calls in M2Crypto block all Python threads, not just the caller - making it impossible to accept an additional connection while a different thread is waiting for incoming data, for example.

I'm not sure whether this should be classified as a security bug (it is exploitable probably only if someone created a multi-threaded application and didn't ever test parallel connections), cc:ing Mark Cox.

(And yes, it is probably too late to fix in 5.3.)

Comment 3 Miloslav Trmač 2008-12-04 18:14:10 UTC
Created attachment 325715 [details]
Backport the 0.18 and 0.19 GILState / ALLOW_THREADS changes

Comment 5 Dan Kenigsberg 2008-12-09 21:22:52 UTC
vdsm (Qumranet's virtual desktop server manager) uses m2crypto to communicate with SolidICE central management. Without this patch, buggy (or badly-networked, or malicious) client can easily render vdsm dysfunctional. All it needs to do is to open a tcp connection to vdsm and fail to finish ssl handshake.

Qumranet qa has encountered this issue repeatedly. Without the patch VDS will frequently turn non-operational in clients' sites, which usually makes clients less happy.

If I'm missing the point of business justification for solving this bug, let me know.

Comment 6 Martin Kočí 2009-02-05 08:05:00 UTC
Could you provide us some more concrete information about testing before qa_ack, please ? Thank you.

Comment 7 Miloslav Trmač 2009-02-06 11:21:06 UTC
To test:
$ cd /usr/share/doc/m2crypto-*/demo/ssl
$ cp dh1024.pem ca.pem client.pem server.pem echod-thread.py echo.py echod_lib.py $test_dir
$ cd $test_dir
# Either replace the .pem files by newer certificates (perhaps from rawhide
# m2crypto), or disable certificate verification by doing the following:
# - in echod-thread.py, replace "SSL.verify_.....cert" by "SSL.verify_none"
# - in echo.py, uncomment the "ctx.set_verify(SSL.verify_none, 10)" line and
#   delete the other ctx.set_verify line below it
$ python echod-thread.py
# in another terminal, check how it works single-threaded:
$ python echo.py -h localhost
# connects and repeats any typed text
# with the echo.py running, open yet another terminal:
$ python echo.py -h localhost

Current results:
The second echo.py does not connect and blocks until the first echo.py exits

Expected results:
Two clients to echod-thread.py connected in parallel, the server responding to input from both.

Comment 9 Miloslav Trmač 2009-04-06 17:40:18 UTC
More blocking calls were identified in vdsm.  See https://bugzilla.osafoundation.org/show_bug.cgi?id=12775 for proposed upstream patch.

(I'm afraid I don't have a good test case for the BIO_free / SSL_shutdown hang observed.)

Comment 10 Miloslav Trmač 2009-04-06 19:06:49 UTC
Created attachment 338381 [details]
Updated backported patch (very lightly tested)

Comment 12 Dan Kenigsberg 2009-06-10 09:48:46 UTC
we've encountered this (or very similar) bug again with m2crypto-0.16-6.el5.3 and openssl-0.9.8e-7.el5. should I reopen it?

#0  0x00000030d6e0d2cb in read () from /lib64/libpthread.so.0
#1  0x00000030d9a782c2 in sock_read (b=<value optimized out>, out=<value optimized out>, outl=<value optimized out>)
    at /usr/include/bits/unistd.h:35
#2  0x00000030d9a767ff in BIO_read (b=<value optimized out>, out=<value optimized out>, outl=<value optimized out>) at bio_lib.c:212
#3  0x000000394ac2090d in ssl3_read_n (s=<value optimized out>, n=<value optimized out>, max=<value optimized out>, 
    extend=<value optimized out>) at s3_pkt.c:198
#4  0x000000394ac20dcd in ssl3_read_bytes (s=<value optimized out>, type=<value optimized out>, buf=<value optimized out>, 
    len=<value optimized out>, peek=<value optimized out>) at s3_pkt.c:266
#5  0x000000394ac1e1d4 in ssl3_shutdown (s=<value optimized out>) at s3_lib.c:2358
#6  0x000000394ac37902 in ssl_free (a=<value optimized out>) at bio_ssl.c:127
#7  0x00000030d9a76921 in BIO_free (a=<value optimized out>) at bio_lib.c:136
#8  0x00002ba9b545d79f in _wrap_bio_free (self=<value optimized out>, args=<value optimized out>) at SWIG/_m2crypto_wrap.c:6510
#9  0x00000030d769497a in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3563
#10 0x00000030d76958a5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, 
    args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, 
    defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736
#11 0x00000030d764c279 in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>)
    at Objects/funcobject.c:548
#12 0x00000030d7635fb0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>)
    at Objects/abstract.c:1795
#13 0x00000030d763c03f in instancemethod_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>)
    at Objects/classobject.c:2447
#14 0x00000030d7635fb0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>)
    at Objects/abstract.c:1795
#15 0x00000030d768f4fd in PyEval_CallObjectWithKeywords (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>)
    at Python/ceval.c:3430
#16 0x00000030d763ef3d in instance_dealloc (inst=<value optimized out>) at Objects/classobject.c:646
#17 0x00000030d766c34f in tupledealloc (op=<value optimized out>) at Objects/tupleobject.c:169
#18 0x00000030d765b9ab in dict_dealloc (mp=<value optimized out>) at Objects/dictobject.c:728
#19 0x00000030d7675038 in subtype_dealloc (self=<value optimized out>) at Objects/typeobject.c:691
#20 0x00000030d763c47d in instancemethod_dealloc (im=<value optimized out>) at Objects/classobject.c:2237
#21 0x00000030d76bb44d in t_bootstrap (boot_raw=<value optimized out>) at Modules/threadmodule.c:454
#22 0x00000030d6e06367 in start_thread () from /lib64/libpthread.so.0
#23 0x00000030d66d30ad in clone () from /lib64/libc.so.6

Comment 13 Miloslav Trmač 2009-06-10 12:34:17 UTC
This backtrace only shows a process that is blocking on a read() from a socket (while attempting to close the socket).

It does not contain anything that would indicate this is related to the Python global interpreter lock.  (Other threads might be blocked on the Python GIL, but a backtrace of all threads would be necessary to demonstrate it - and AFAICS BIO_free() is already called without the GIL in m2crypto-0.16-6.el5.4.)

And the fix for #472690 is present only in m2crypto-0.16-6.el5.4, not in ...el5.3.

Comment 16 errata-xmlrpc 2009-09-02 11:19:59 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.