Bug 1001122

Summary: malloc memory corruption
Product: Red Hat Enterprise Linux 7 Reporter: Roman Rakus <rrakus>
Component: python-pillowAssignee: Michal Minar <miminar>
Status: CLOSED CURRENTRELEASE QA Contact: Adam Kolář <akolar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0CC: bkabrda, david.m.highley, dmalcolm, ivazqueznet, jamatos, jcapik, jonathansteffan, lilyfan, manisandro, tomspur, tsmetana
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: python-pillow-2.0.0-14.gitd1c6db8.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 995783 Environment:
Last Closed: 2014-06-13 10:29:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 995783    
Bug Blocks: 982412    
Attachments:
Description Flags
pysol-valgrind.txt
none
Valgrind output with debuginfo none

Description Roman Rakus 2013-08-26 14:17:07 UTC
+++ This bug was initially created as a clone of Bug #995783 +++

Description of problem:
Hangs on start up, running in command window see the malloc errors below.

Version-Release number of selected component (if applicable):
PySolFC-2.0-8.fc19.noarch

How reproducible:
Everytime

Steps to Reproduce:
1. type pysol
2.
3.

Actual results:
pysol
*** Error in `/usr/bin/python': malloc(): memory corruption: 0x0000000004a8f720 ***
*** Error in `/usr/bin/python': malloc(): memory corruption: 0x0000000004a8f720 ***


Expected results:


Additional info:

--- Additional comment from David Highley on 2013-08-12 23:43:09 EDT ---

Found a posted work around; pysol --tk

- Temporary fix was to; mv /bin/pysol pysol.orig
- Create a shell wrapper scripts named pysol
#!/bin/sh

exec pysol.orig --tk


- Set execution chmod u+x,g+x,o+x

--- Additional comment from Jaromír Cápík on 2013-08-13 08:41:42 EDT ---

Hello David.

I can reproduce it here. Anyway, this looks like a python bug. I'm changing the component to python2 and we'll see.

Regards,
Jaromir.

--- Additional comment from Jean-Charles Malahieude on 2013-08-23 13:40:29 EDT ---

Hi,

Reproduced on a 32 bit architecture, with another memory address:

---
jcharles@localhost ~$ python --version
Python 2.7.5
jcharles@localhost ~$ uname -r
3.10.9-200.fc19.i686
jcharles@localhost ~$ pysol
*** Error in `/usr/bin/python': malloc(): memory corruption: 0x0b883af0 ***
*** Error in `/usr/bin/python': malloc(): memory corruption: 0x0b883af0 ***

jcharles@localhost ~$ ps -ef | grep pysol
jcharles  3584  3386  0 19:34 pts/0    00:00:00 /bin/sh /usr/bin/pysol
jcharles  3585  3584  1 19:34 pts/0    00:00:03 /usr/bin/python /usr/share/PySolFC/pysol.py --sound-mod=pygame
jcharles  3600  3386  0 19:38 pts/0    00:00:00 grep --color=auto pysol
jcharles@localhost ~$
---

Regards,
Jean-Charles

--- Additional comment from Jaromír Cápík on 2013-08-26 09:31:29 EDT ---

It seems python-imaging combines libc malloc calls with python object manipulations and that's forbidden:

http://docs.python.org/2/c-api/memory.html

The libc calls should be replaced with the python alternatives.

--- Additional comment from Roman Rakus on 2013-08-26 09:42:00 EDT ---

For reproducer;  must have set MALLOC_CHECK_ at least 1.

--- Additional comment from Jaromír Cápík on 2013-08-26 09:43:15 EDT ---

According to the following stacktrace, it seems the malicious call lies in the _imaging.so / Storage.c / ImagingNewPrologueSubtype(). As the libc calloc is used for creating a structure that contains a python object, it might be the reason of the memory corruption.

Thread 1 (Thread 0x7ffff7fc6740 (LWP 6047)):
#0  pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:94
#1  0x0000003aeb50909c in __GI___backtrace (array=array@entry=0x7fffffffa730, size=size@entry=64) at ../sysdeps/x86_64/backtrace.c:103
#2  0x0000003aeb475d64 in __libc_message (do_abort=2, fmt=fmt@entry=0x3aeb57db88 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:176
#3  0x0000003aeb47e24c in malloc_printerr (ptr=0x2e54580, str=0x3aeb57b2d7 "malloc(): memory corruption", action=<optimized out>) at malloc.c:4916
#4  _int_malloc (av=0x3aeb7b9780 <main_arena>, bytes=56) at malloc.c:3390
#5  0x0000003aeb48005c in __GI___libc_malloc (bytes=56) at malloc.c:2863
#6  0x0000003aeac0d379 in _dl_map_object_deps (map=map@entry=0x1b96b00, preloads=preloads@entry=0x0, npreloads=npreloads@entry=0, trace_mode=trace_mode@entry=0, open_mode=open_mode@entry=-2147483648) at dl-deps.c:515
#7  0x0000003aeac138ec in dl_open_worker (a=a@entry=0x7fffffffb2c8) at dl-open.c:265
#8  0x0000003aeac0f304 in _dl_catch_error (objname=objname@entry=0x7fffffffb2b8, errstring=errstring@entry=0x7fffffffb2c0, mallocedp=mallocedp@entry=0x7fffffffb2b0, operate=operate@entry=0x3aeac13770 <dl_open_worker>, args=args@entry=0x7fffffffb2c8) at dl-error.c:177
#9  0x0000003aeac1321b in _dl_open (file=0x3aeb57a0be "libgcc_s.so.1", mode=-2147483647, caller_dlopen=0x3aeb508f85 <init+21>, nsid=-2, argc=2, argv=<optimized out>, env=0x8ef770) at dl-open.c:656
#10 0x0000003aeb52fbc2 in do_dlopen (ptr=ptr@entry=0x7fffffffb4d0) at dl-libc.c:87
#11 0x0000003aeac0f304 in _dl_catch_error (objname=0x7fffffffb4b0, errstring=0x7fffffffb4c0, mallocedp=0x7fffffffb4a0, operate=0x3aeb52fb80 <do_dlopen>, args=0x7fffffffb4d0) at dl-error.c:177
#12 0x0000003aeb52fc82 in dlerror_run (args=0x7fffffffb4d0, operate=0x3aeb52fb80 <do_dlopen>) at dl-libc.c:46
#13 __GI___libc_dlopen_mode (name=name@entry=0x3aeb57a0be "libgcc_s.so.1", mode=mode@entry=-2147483647) at dl-libc.c:163
#14 0x0000003aeb508f85 in init () at ../sysdeps/x86_64/backtrace.c:52
#15 0x0000003aebc0ca50 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#16 0x0000003aeb50909c in __GI___backtrace (array=array@entry=0x7fffffffb790, size=size@entry=64) at ../sysdeps/x86_64/backtrace.c:103
#17 0x0000003aeb475d64 in __libc_message (do_abort=2, fmt=fmt@entry=0x3aeb57db88 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:176
#18 0x0000003aeb47e24c in malloc_printerr (ptr=0x2e54580, str=0x3aeb57b2d7 "malloc(): memory corruption", action=<optimized out>) at malloc.c:4916
#19 _int_malloc (av=av@entry=0x3aeb7b9780 <main_arena>, bytes=bytes@entry=88) at malloc.c:3390
#20 0x0000003aeb480a1a in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3173
#21 0x00007fffef123d40 in ImagingNewPrologueSubtype () from /usr/lib64/python2.7/site-packages/_imaging.so
#22 0x00007fffef12427b in ImagingNewBlock () from /usr/lib64/python2.7/site-packages/_imaging.so
#23 0x00007fffef12433d in ImagingNew () from /usr/lib64/python2.7/site-packages/_imaging.so
#24 0x00007fffef10f2d8 in convert () from /usr/lib64/python2.7/site-packages/_imaging.so
#25 0x00007fffef107287 in _convert () from /usr/lib64/python2.7/site-packages/_imaging.so
#26 0x0000003b01addcee in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#27 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#28 0x0000003b01add769 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#29 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#30 0x0000003b01add769 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#31 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#32 0x0000003b01add769 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#33 0x0000003b01add80c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#34 0x0000003b01add80c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#35 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#36 0x0000003b01add769 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#37 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#38 0x0000003b01a6dd7d in ?? () from /lib64/libpython2.7.so.1.0
#39 0x0000003b01a49dd3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#40 0x0000003b01a58555 in ?? () from /lib64/libpython2.7.so.1.0
#41 0x0000003b01a49dd3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#42 0x0000003b01ad9f1d in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#43 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#44 0x0000003b01a6dd7d in ?? () from /lib64/libpython2.7.so.1.0
#45 0x0000003b01a49dd3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#46 0x0000003b01a58555 in ?? () from /lib64/libpython2.7.so.1.0
#47 0x0000003b01a49dd3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#48 0x0000003b01ad8af7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#49 0x0000003b01a591dc in PyInstance_New () from /lib64/libpython2.7.so.1.0
#50 0x0000003b01a49dd3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#51 0x0000003b01adb6dc in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#52 0x0000003b01add80c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#53 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#54 0x0000003b01add769 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#55 0x0000003b01adec7d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#56 0x0000003b01aded82 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#57 0x0000003b01af78af in ?? () from /lib64/libpython2.7.so.1.0
#58 0x0000003b01af89ce in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#59 0x0000003b01af9b39 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#60 0x0000003b01b0a66f in Py_Main () from /lib64/libpython2.7.so.1.0
#61 0x0000003aeb421b75 in __libc_start_main (main=0x4006f0 <main>, argc=2, ubp_av=0x7fffffffdf78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdf68) at libc-start.c:258
#62 0x0000000000400721 in _start ()

Comment 2 Sandro Mani 2013-08-28 12:24:09 UTC
Roman, are you looking at this, or should I have a look?

Comment 3 Roman Rakus 2013-08-28 12:25:41 UTC
(In reply to Sandro Mani from comment #2)
> Roman, are you looking at this, or should I have a look?

Not right now. You can look at this, or I will next week.

Comment 4 Sandro Mani 2013-08-28 12:28:04 UTC
Ok, I will give it a shot today or tomorrow.

Comment 5 Jaromír Cápík 2013-08-28 13:16:06 UTC
Hi Sandro.

If you're going to look at that, please, consider looking at the patch attached to the Bug 995783. It doesn't resolve the issue, but it replaces all libc memory manipulations with the PyMem ones (recommended by the python upstream). I tried to do some analysis yesterday and it seems the allocation called from ImagingNewPrologueSubtype does not cause the corruption. The memory is very probably corrupted elsewhere and consequently the ImagingNewPrologueSubtype allocation fails.

Regards,
Jaromir.

Comment 6 Sandro Mani 2013-08-28 13:50:13 UTC
Hi Jaromir, did you check whether valgrind memcheck gives any useful info?

Comment 7 Jaromír Cápík 2013-08-28 14:56:56 UTC
Hi Sandro. Yes, I did, but it was not much helpful without a deep analysis.

Comment 8 Jaromír Cápík 2013-08-28 15:09:54 UTC
Created attachment 791430 [details]
pysol-valgrind.txt

Comment 9 Jaromír Cápík 2013-08-28 17:22:13 UTC
The missing piece of puzzle is called _putpalette. Debugging now ...

Comment 10 Sandro Mani 2013-08-28 17:35:47 UTC
Created attachment 791495 [details]
Valgrind output with debuginfo

I'm at it too. So basically the palettesize value passed to putpalette is wrong (i.e. 3510). The palette is passed at Image.py:601, which is called from Image.py:699, and at Image.py:697 the palette is constructed. Now we would need to find out who is responsible for the wrong palettesize value.

(attached is a valgrind output with debuginfos installed).

Comment 11 Jaromír Cápík 2013-08-28 17:52:02 UTC
That's exactly what I've found out. But it has no sense to duplicate the effort. If you're looking at the issue right now, it has no sense for me to continue. Hopefully you'll find the root cause soon. Thanks and good luck :]

Cheers,
Jaromir.

Comment 12 Sandro Mani 2013-08-28 17:52:59 UTC
Right, well basically the incorrect statement is at Image.py:696

bytePalette = bytes([i//3 for i in range(768)])

yields an array of size 3510 instead of 768 as intended. I guess the problem is that the i//3 entries should be uint8_t?

Comment 13 Jaromír Cápík 2013-08-28 18:17:29 UTC
The internal expression returns 768. It's the bytes() conversion that makes the evil.

Btw ... maybe there should be an assert in the ImagingUnpackRGB, checking if the palettesize is < 256. That would help to resolve future issues like that ...

Comment 14 Jaromír Cápík 2013-08-28 18:22:09 UTC
The author probably wanted to use bytearray() instead of bytes(). That needs to be analysed first.

Comment 15 Sandro Mani 2013-08-28 18:23:14 UTC
Right, reported upstream with suggested fix: https://github.com/python-imaging/Pillow/pull/325

Assertion: I'll add it to the pull request.

Comment 16 Sandro Mani 2013-08-28 18:24:40 UTC
*** Bug 995783 has been marked as a duplicate of this bug. ***

Comment 17 Sandro Mani 2013-08-28 20:18:55 UTC
I've pushed fixed builds to f19, f20 and rawhide, but I guess this bug should be closed when the update hits rhel7?

Comment 18 Jaromír Cápík 2013-08-29 10:52:14 UTC
Of course.

Comment 19 Sandro Mani 2013-08-29 11:01:21 UTC
Right, reassigning. Thanks Jaromir for your effort.

Comment 20 Jaromír Cápík 2013-09-03 15:46:30 UTC
I thank you for taking care of it so quickly.

Comment 27 Ludek Smid 2014-06-13 10:29:36 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.