Description of problem:
Running loggerhead under mod_wsgi, I get the following traceback in the httpd/error_log if python-imaging is installed:
[Tue Apr 13 11:15:40 2010] [error] [client ::1] Traceback (most recent call last):
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/bin/loggerhead.wsgi", line 9, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] from loggerhead.apps.transport import BranchesFromTransportRoot
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib/python2.6/site-packages/loggerhead/apps/transport.p
y", line 30, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] from loggerhead.apps.branch import BranchWSGIApp
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib/python2.6/site-packages/loggerhead/apps/branch.py",
line 30, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] from loggerhead.controllers.annotate_ui import AnnotateUI
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib/python2.6/site-packages/loggerhead/controllers/anno
tate_ui.py", line 32, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] from loggerhead.highlight import highlight
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib/python2.6/site-packages/loggerhead/highlight.py", l
ine 21, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] from pygments.formatters import HtmlFormatter
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib/python2.6/site-packages/pygments/formatters/__init_
_.py", line 14, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] from pygments.formatters._mapping import FORMATTERS
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib/python2.6/site-packages/pygments/formatters/_mappin
g.py", line 21, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] from pygments.formatters.img import BmpImageFormatter
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib/python2.6/site-packages/pygments/formatters/img.py"
, line 21, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] import Image, ImageDraw, ImageFont
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib64/python2.6/site-packages/PIL/Image.py", line 44, i
[Tue Apr 13 11:15:40 2010] [error] [client ::1] __import__("FixTk")
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib64/python2.6/lib-tk/FixTk.py", line 15, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] import ctypes
[Tue Apr 13 11:15:40 2010] [error] [client ::1] File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>
[Tue Apr 13 11:15:40 2010] [error] [client ::1] CFUNCTYPE(c_int)(lambda: None)
[Tue Apr 13 11:15:40 2010] [error] [client ::1] MemoryError
A much simpler reproducer is to use a wsgi script that does:
The script should exit with MemoryError when it imports ctypes.
Version-Release number of selected component (if applicable):
Also affects python on rawhide
SELinux must be in enforcing mode and httpd_tmp_exec must be off:
setsebool httpd_tmp_exec off
Pulling up the URL that's configured to invoke the sample wsgi script will cause the traceback. This should also affect mod_python.
This bug pretty much details what's happening:
A little background for the bug is here:
What this means is that out of the box, mod_wsgi and mod_python scripts that end up calling ctypes will fail. Even worse, the user won't associate python throwing a MemoryError in ctypes with the httpd_tmp_exec selinux boolean. How can we fix this? dwalsh says that for random scripts being executed, there's nothing to be done as you'd essentially have to set up a directory that has /tmp's permissions. However, this only affects programs running inside the python process. So perhaps we can set up a directory in /var/run that apache is able to write to for this purpose? Then the libffi in python could be configured to try this directory if the other directories have failed.
For reference, you mentioned this bug on IRC: https://dev.openwrt.org/ticket/6192
who worked around it by removing the
line from ctypes/__init__.py
A simpler reproducer may be a wsgi script that does:
(i.e. do you need the second line to reproduce it? I don't think you do).
My initial thought was to remove the line in our ctypes, given that it's only there as a workaround for a Win64 issue, but presumably the tk fix actually uses ctypes for something.
It's unclear to me how to tell libffi which directory to use.
The Python-level syntax is:
not something like
Potentially we could add an API hook to tell it which directory, but somehow mod_wsgi needs to call it to tell it to work one way, in a way that's different to how /usr/bin/python does it.
Another approach might be to patch the MemoryError so that it provides a hint as to the SELinux booleans to tweak.
> The script should exit with MemoryError when it imports ctypes
Untested, so I included both lines.
Also correct that merely patching ctypes to not do this:
is just a temporary workaround.... Some app could legitimately use ctypes (or require a module that uses ctypes) that causes the problem.
I was thinking of the directories to use be a hardcoded list in python's libffi/ctypes. So no need to pass in a value. OTOH, passing in a value could work. We'd probably do it something like this:
* Patch ctypes to not call CFUNCTYPE(c_int)(lambda: None) as that's a hack that's causing us issues.
* Add an API call to tell which directory or a list of directories to try. The code then looks like:
* mod_wsgi and mod_python could be configured to call set_ffi_dir() but another possibility is for the application code to call set_ffi_dir(). As long as this occurs before any python code runs a problematic ctypes function it's not too bad. Although it still leaves us with a documentation issue.
* Simply patching the MemoryError doesn't seem like the best solution (as tweaking the boolean means apache can read anything in /tmp while these fixes limit it to a directory especially for this case) but it does address getting the documentation of what's going on out there. If it's simple, we might want to do it as a temporary measure.
Ricky found that this affects the Fedora Account System as well. That path lead through the tgcaptcha plugin. Since PIL is pretty much the only python package that manipulates images, it's a good guess that all python-based captcha software will hit this bug.
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.
More information and reason for this action is here:
Not yet fixed
I'm seeing this in F16. I get this from httpd:
open("/dev/shm/ffifu9bni", O_RDWR|O_CREAT|O_EXCL, 0600) = 24
unlink("/dev/shm/ffifu9bni") = 0
ftruncate(24, 4096) = 0
mmap(NULL, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 24, 0) = -1 EACCES (Permission
close(24) = 0
which seems to be exactly the same problem.
See further notes in bug 814391 (which is filed against RHEL 6)
Downstream removal of the:
line committed to python "master" for Fedora 18.
Building python-2.7.3-4.fc18 for dist-rawhide
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=4009760
I checked Python 3.2.3, and it exhibits the same behavior:
python3 -m test.test_uuid
only allocates a single thunk, and only because of that Win64 workaround.
I've committed a similar downstream removal of that line for "python3" in Fedora 18:
Building python3-3.2.3-4.fc18 for dist-rawhide
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=4009802
Also pushed to python on f17 and building:
Building python-2.7.3-4.fc17 for f17-candidate
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=4009876
I can confirm that patching ctypes/__init__.py as above also eliminates the thunk allocation for doing:
"from PIL import Image"
in the initial report (which again comes from "import ctypes", via "FixTk").
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.
(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)
More information and reason for this action is here:
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Dave can't we close this as won't fix, and if you run apache with this python module you have to turn on the httpd_execmem boolean?
We could but it strikes me as a bit insecure. Also, when reported, it was httpd_tmp_exec but maybe libffi detects on the fly that it can't execmem and fallsback to a temporary file? I haven't tested.
This message is a notice that Fedora 19 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 19. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 19 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
Thank you for reporting this bug and we are sorry it could not be fixed.