Bug 645193 - python26 ctypes MemoryError under mod_wsgi
Summary: python26 ctypes MemoryError under mod_wsgi
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: python26
Version: el5
Hardware: x86_64
OS: Linux
low
medium
Target Milestone: ---
Assignee: Adrien Devresse
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-10-20 23:43 UTC by BJ Dierkes
Modified: 2017-04-06 10:25 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-06 10:25:41 UTC


Attachments (Terms of Use)

Description BJ Dierkes 2010-10-20 23:43:47 UTC
Attempting to import ctypes under python26, and python26-mod_wsgi results in a 
MemoryError.

TESTED WITH:

Test 1:

RHEL 5.5 64bit with SELinux *Disabled*
python26-2.6.5-3.el5
mod_python26-2.6-3.ius, and python26-mod_wsgi-3.3-1.el5 (epel)


Test 2 (did not fail):

Fedora 13 64bit
python-2.6.4-25.fc13, and python-2.6.5-17.fc14 (rebuilt on fc13)
mod_wsgi-3.1-1.fc13, mod_wsgi-3.2-2.fc14 (rebuilt), and mod_wsgi-3.3-1 (rebuilt with latest source)


Possibly related to:

https://bugzilla.redhat.com/show_bug.cgi?id=582009
http://bugs.python.org/issue5504


HOW TO REPRODUCE:

Install python26 (EPEL 5), and python26-mod_wsgi (soon to be added) which can be found at:

http://koji.fedoraproject.org/koji/taskinfo?taskID=2545585


/etc/httpd/python26-mod_wsgi.conf

# BOF

LoadModule wsgi_module modules/python26-mod_wsgi.so
WSGIScriptAlias /wsgi_test /var/www/wsgi/wsgi_test.py 

# EOF


/var/www/wsgi/wsgi_test.py 

# BOF

PROJECT_ROOT = '/var/www/wsgi'

def simple_app(environ, start_response):
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    exc = None
    import ctypes
    return ['Hello world!\n%s\n%s' % (exc, environ)]

application = simple_app

# EOF


Restart Apache, and attempt to view http://localhost/wsgi_test

/var/log/httpd/error_log

# BOF

[Sun Oct 17 02:20:03 2010] [notice] Digest: generating secret for digest authentication ...
[Sun Oct 17 02:20:03 2010] [notice] Digest: done
[Sun Oct 17 02:20:03 2010] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations
[Wed Oct 20 16:55:55 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2377): Exception occurred processing WSGI script '/var/www/wsgi/wsgi_test.py'.
[Wed Oct 20 16:55:55 2010] [error] [client 127.0.0.1] Traceback (most recent call last):
[Wed Oct 20 16:55:55 2010] [error] [client 127.0.0.1]   File "/var/www/wsgi/wsgi_test.py", line 20, in simple_app
[Wed Oct 20 16:55:55 2010] [error] [client 127.0.0.1]     import ctypes
[Wed Oct 20 16:55:55 2010] [error] [client 127.0.0.1]   File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>
[Wed Oct 20 16:55:55 2010] [error] [client 127.0.0.1]     CFUNCTYPE(c_int)(lambda: None)
[Wed Oct 20 16:55:55 2010] [error] [client 127.0.0.1] MemoryError

# EOF

Comment 1 Dave Malcolm 2010-10-21 15:04:02 UTC
Thanks for filing this bug.

This seems a lot like bug 582009 (albeit for a different build of python on a different OS).

Presumably the failure is seen with SELinux in Enforcing mode and there's an AVC denial occurring; please can you confirm, and please can you post the exact message ("sealert -b" will let you browse these; dwalsh: is there a better way to locate the error in question?).

(please can you also confirm that the files on disk are correctly labelled; in comment #0 you refer to SELinux being "Disabled" (as opposed to "Permissive"), which can lead to files and directories having no label, as I understand it)

If you launch /usr/bin/python from a shell, are you able to "import ctypes" from the interactive prompt?  I wonder if the context in which that runs has more permissions than the one in which httpd runs, or which the wsgi_test.py runs.

Thanks!

Comment 2 BJ Dierkes 2010-10-21 19:01:38 UTC
[root@208934-el5-64 ~]# sealert -b
[root@208934-el5-64 ~]# echo $?
0

[root@208934-el5-64 ~]# getenforce 
Disabled


Presumably, you meant '/usr/bin/python2.6' (but /usr/bin/python works as well):

[root@208934-el5-64 ~]# python2.6
Python 2.6.5 (r265:79063, Oct  4 2010, 18:53:06) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> 


The files (/etc/httpd/conf.d/python26-mod_wsgi, and /var/www/wsgi/wsgi_test.py) did *not* have any labels.  To test the theory I set SELinux to Permissive, rebooted, and checked again... which they now *do* have labels:

[root@208934-el5-64 ~]# ls -Z /etc/httpd/conf.d/python26-mod_wsgi.conf
-rw-r--r--  root root system_u:object_r:file_t         /etc/httpd/conf.d/python26-mod_wsgi.conf

[root@208934-el5-64 ~]# ls -Z /var/www/wsgi/wsgi_test.py 
-rw-r--r--  root root system_u:object_r:file_t         /var/www/wsgi/wsgi_test.py


I then re-attempted the tests:

[root@208934-el5-64 ~]# getenforce 
Permissive

[root@208934-el5-64 ~]# sealert -b
[root@208934-el5-64 ~]# echo $?
0

[root@208934-el5-64 ~]# /usr/bin/python2.6
Python 2.6.5 (r265:79063, Oct  4 2010, 18:53:06) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> 

[root@208934-el5-64 ~]# curl http://localhost/wsgi_test
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.3 (Red Hat) Server at localhost Port 80</address>
</body></html>


[root@208934-el5-64 ~]# cat /var/log/httpd/error_log
... snip ...
[Thu Oct 21 13:58:01 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2008): Exception occurred processing WSGI script '/var/www/wsgi/wsgi_test.py'.
[Thu Oct 21 13:58:01 2010] [error] [client 127.0.0.1] Traceback (most recent call last):
[Thu Oct 21 13:58:01 2010] [error] [client 127.0.0.1]   File "/var/www/wsgi/wsgi_test.py", line 20, in simple_app
[Thu Oct 21 13:58:01 2010] [error] [client 127.0.0.1]     import ctypes
[Thu Oct 21 13:58:01 2010] [error] [client 127.0.0.1]   File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>
[Thu Oct 21 13:58:01 2010] [error] [client 127.0.0.1]     CFUNCTYPE(c_int)(lambda: None)
[Thu Oct 21 13:58:01 2010] [error] [client 127.0.0.1] MemoryError

Comment 3 BJ Dierkes 2010-11-29 19:16:59 UTC
Similar issues:

http://bugs.gentoo.org/329499


Also, doing the same from an apache bash shell produces a segfault like so (with backtrace):

# need shell to run gdb
[root@208933-el5-i386 ~]# usermod -s /bin/bash apache

[root@208933-el5-i386 ~]# sudo su - apache
-bash-3.2$ 

-bash-3.2$ gdb --args python26 -c "import ctypes"
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python26...Reading symbols from /usr/lib/debug/usr/bin/python2.6.debug...
warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/python2.6.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/python2.6.debug
done.
done.



(gdb) run
Starting program: /usr/bin/python26 -c import\ ctypes
warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/python2.6.debug
[Thread debugging using libthread_db enabled]
warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/python2.6.debug
warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/python2.6.debug

Program received signal SIGSEGV, Segmentation fault.
0x00964b21 in CThunkObject_dealloc (_self=0xb7fb0084)
    at /usr/src/debug/Python-2.6.5/Modules/_ctypes/callbacks.c:23
23              Py_XDECREF(self->restype);




(gdb) bt
#0  0x00964b21 in CThunkObject_dealloc (_self=0xb7fb0084)
    at /usr/src/debug/Python-2.6.5/Modules/_ctypes/callbacks.c:23
#1  0x00964e43 in AllocFunctionCallback (callable=0xb7d3a144, converters=0xb7fb802c, 
    restype=0x80be2e4, flags=257) at /usr/src/debug/Python-2.6.5/Modules/_ctypes/callbacks.c:470
#2  0x0095facf in CFuncPtr_new (type=0x80c8a6c, args=0xb7d393ac, kwds=0x0)
    at /usr/src/debug/Python-2.6.5/Modules/_ctypes/_ctypes.c:3429
#3  0x006ba76e in type_call (type=0x80c8a6c, args=0xb7d393ac, kwds=0x0) at Objects/typeobject.c:726
#4  0x0066184c in PyObject_Call (func=0x80c8a6c, arg=0xb7d393ac, kw=0x0) at Objects/abstract.c:2492
#5  0x006f7d09 in call_function (f=0x80b3a6c, throwflag=0) at Python/ceval.c:3968
#6  PyEval_EvalFrameEx (f=0x80b3a6c, throwflag=0) at Python/ceval.c:2412
#7  0x006fd461 in PyEval_EvalCodeEx (co=0xb7fb10f8, globals=0xb7fa9604, locals=0xb7fa9604, 
    args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3000
#8  0x006fd523 in PyEval_EvalCode (co=0xb7fb10f8, globals=0xb7fa9604, locals=0xb7fa9604)
    at Python/ceval.c:541
#9  0x0070c71d in PyImport_ExecCodeModuleEx (name=0xbfffd5e7 "ctypes", co=0xb7fb10f8, 
    pathname=0xbfffa4b7 "/usr/lib/python2.6/ctypes/__init__.pyc") at Python/import.c:677
#10 0x0070cd5c in load_source_module (name=0xbfffd5e7 "ctypes", 
    pathname=0xbfffa4b7 "/usr/lib/python2.6/ctypes/__init__.pyc", fp=<value optimized out>)
    at Python/import.c:1017
#11 0x0070e7fa in load_package (name=0xbfffd5e7 "ctypes", pathname=<value optimized out>)
    at Python/import.c:1073
#12 0x0070dfd3 in import_submodule (mod=0x793800, subname=0xbfffd5e7 "ctypes", 
    fullname=0xbfffd5e7 "ctypes") at Python/import.c:2591
#13 0x0070e4ac in load_next (mod=0x793800, altmod=0x793800, p_name=<value optimized out>, 
    buf=0xbfffd5e7 "ctypes", p_buflen=0xbfffd5dc) at Python/import.c:2411
#14 0x0070f1ab in import_module_level (name=0xb7fa8eb4 "ctypes", globals=0xb7fea24c, 
    locals=0xb7fea24c, fromlist=0x793800, level=-1) at Python/import.c:2133
#15 PyImport_ImportModuleLevel (name=0xb7fa8eb4 "ctypes", globals=0xb7fea24c, locals=0xb7fea24c, 
    fromlist=0x793800, level=-1) at Python/import.c:2184
#16 0x006f58d4 in builtin___import__ (self=0x0, args=0xb7ff1284, kwds=0x0)
    at Python/bltinmodule.c:48
#17 0x006a087d in PyCFunction_Call (func=0xb7fd52ac, arg=0xb7ff1284, kw=0x7465725f)
    at Objects/methodobject.c:85
#18 0x0066184c in PyObject_Call (func=0xb7fd52ac, arg=0xb7ff1284, kw=0x0)
    at Objects/abstract.c:2492
#19 0x006f5fb4 in PyEval_CallObjectWithKeywords (func=0xb7fd52ac, arg=0xb7ff1284, kw=0x0)
    at Python/ceval.c:3619
#20 0x006f7f06 in PyEval_EvalFrameEx (f=0x80ae274, throwflag=0) at Python/ceval.c:2159
#21 0x006fd461 in PyEval_EvalCodeEx (co=0xb7fee608, globals=0xb7fea24c, locals=0xb7fea24c, 
    args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3000
#22 0x006fd523 in PyEval_EvalCode (co=0xb7fee608, globals=0xb7fea24c, locals=0xb7fea24c)
    at Python/ceval.c:541
#23 0x00717ad6 in run_mod (mod=<value optimized out>, filename=<value optimized out>, 
    globals=0xb7fea24c, locals=0xb7fea24c, flags=0xbfffeae8, arena=0x8059f58)
    at Python/pythonrun.c:1339
#24 0x00717d31 in PyRun_StringFlags (str=0x804a008 "import ctypes\n", start=257, 
    globals=0xb7fea24c, locals=0xb7fea24c, flags=0xbfffeae8) at Python/pythonrun.c:1302
#25 0x00718fe3 in PyRun_SimpleStringFlags (command=0x804a008 "import ctypes\n", flags=0xbfffeae8)
---Type <return> to continue, or q <return> to quit---

Comment 4 BJ Dierkes 2010-11-29 20:08:31 UTC
The further the test from above, it seems to only affect the apache user.  Other non-root users do not experience the same issue.

Comment 5 richard.lobb 2011-07-20 23:45:06 UTC
I've run into this bug too. In my case it occurs when I exec a Python script from within a PHP script running under Apache on a 64-bit Red Hat Enterprise Linux Server release 6.1 (Santiago) with SELinux enabled. [The Python code being run is the front-end to a pypy sandbox.] The same bit of code works fine on the same server when the PHP script is executed directly from the command line. My "fix" has been simply to comment out the last line in ctypes/__init__.py, viz "CFUNCTYPE(c_int)(lambda: None)". The comment immediately preceding that line shows that it was added as a hack to solve some other non-understood problem on a 64-bit Windows system.

Comment 6 Nicolas 2011-11-17 13:52:52 UTC
richard: the error occurs (at least) every time you call CFUNCTYPE. Your workaround bought me some time, but as soon as you call [code that calls] CFUNCTYPE, the error is back. Only way for me to get this to work was to turn SELinux completely off, ie /usr/sbin/setenforce 0.

On a note more geared towards solving the problem, I just wanted to add that the problem's still around in Python2.7

Comment 7 Daniel Walsh 2011-11-17 14:40:37 UTC
What AVC did you see?

David lets talk about this issue.  Might be time to revert the SELinux hack and see if we can approach this from a different angle.

Comment 8 Nicolas 2011-11-17 14:52:30 UTC
I hope this is it, I have no idea what it is saying...

type=AVC msg=audit(1321541391.967:57971): avc:  denied  { execute } for  pid=2332 comm="httpd" path=2F6465762F73686D2F6666695564536B6F50202864656C6574656429 dev=tmpfs ino=9342974 scontext=user_u:system_r:httpd_t:s0 tcontext=user_u:object_r:httpd_tmpfs_t:s0 tclass=file

Comment 9 Daniel Walsh 2011-11-17 18:57:15 UTC
It is saying that httpd_t (httpd daemon) wants to execute something it wrote to the tmpfs file system.  The file was "/dev/shm/ffiUdSkoP (deleted)".

Usually we don't like to allow a domain to write to a type that it can execute.

If you want to allow this, you can just make a local module.

# grep httpd_t /var/log/audit/audit.log | audit2allow -M myhttp
# semodule -i myhttp.pp

Comment 10 richard.lobb 2011-11-18 00:03:23 UTC
Ah, the light starts to dawn, albeit faintly. Good to see progress being made on this.

A bit of googling now reveals several other discussions on this issue, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=582009 and https://bugzilla.redhat.com/show_bug.cgi?id=522731  I don't understand the discussion fully as I don't myself use SELinux -- I just hit the problem when I ported my Moodle plug-in to an SELinux server. But it seems clear that the fundamental problem is with the execute permissions granted to httpd.

The first of the above postings implies that the problem can be bypassed by telling SELinux to allow httpd to execute code from the tmp directory, for which there's apparently now a special flag. I haven't tried myself, but it looks like the command

setsebool httpd_tmp_exec on

should allow httpd to execute files from temporary directories. See also http://docs.fedoraproject.org/en-US/Fedora/13/html/Managing_Confined_Services/sect-Managing_Confined_Services-The_Apache_HTTP_Server-Booleans.html


I'm not quite sure where this command goes, but hopefully an SELinux administrator can tell us.

Comment 11 Nicolas 2011-11-18 11:10:37 UTC
FYI

> sudo /usr/sbin/setsebool httpd_tmp_exec on
libsemanage.dbase_llist_set: record not found in the database
libsemanage.dbase_llist_set: could not set record value
Could not change boolean httpd_tmp_exec
Could not change policy booleans

Comment 12 Daniel Walsh 2011-11-18 15:07:31 UTC
The boolean does not exist on the selinux system you are trying this on.

Comment 13 Daniel Walsh 2011-11-18 15:08:49 UTC
Do the audit2allow -M myhttp described above.

THe boolean exists in RHEL6 not RHEL5.

Comment 14 richard.lobb 2011-11-21 08:02:54 UTC
Thanks Daniel. Can I just check I have this right, before I pass it on the SELinux administrator, please? My understanding is:

If you're using RHEL6, just type the command

#setsebool -P httpd_tmp_exec on

If not using RHEL6, then you need to be running the audit daemon, and need to have at least one AVC message relating to the above problem in the audit log since the last reboot. Verify that there is such a message with the command

#grep httpd_t /var/log/audit/audit.log

If there is such a message then subsequent execution denials can be prevented by executing the two commands

# grep httpd_t /var/log/audit/audit.log | audit2allow -M myhttp
# semodule -i myhttp.pp

Sorry if that all seems a bit tediously detailed to you, but I'm not an SELinux administrator myself.

Comment 15 Daniel Walsh 2011-11-23 16:34:18 UTC
Yes.

Comment 16 Fedora End Of Life 2017-04-06 10:25:41 UTC
Fedora EPEL 5 changed to end-of-life (EOL) status on 2017-03-31. Fedora EPEL 5
is no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora
or Fedora EPEL, please feel free to reopen this bug against that version. If
you are unable to reopen this bug, please file a new report against the current
release. If you experience problems, please add a comment to this bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.