Bug 188327 - outrageous memory usage by some regular expressions
outrageous memory usage by some regular expressions
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
5
All Linux
medium Severity medium
: ---
: ---
Assigned To: James Antill
Brock Organ
:
: 188977 (view as bug list)
Depends On: 188649
Blocks:
  Show dependency treegraph
 
Reported: 2006-04-07 18:25 EDT by Shahms E. King
Modified: 2008-03-11 12:55 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-03-11 12:55:17 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Python pmap before and after import (1.60 KB, text/plain)
2006-04-07 18:25 EDT, Shahms E. King
no flags Details
Test case demonstrating gcc optimization bug (1.01 KB, text/plain)
2006-04-11 19:59 EDT, Shahms E. King
no flags Details

  None (edit)
Description Shahms E. King 2006-04-07 18:25:16 EDT
Created attachment 127486 [details]
Python pmap before and after import
Comment 1 Shahms E. King 2006-04-07 18:25:16 EDT
Description of problem:
Importing the cElementTree module causes the python process to allocate an extra
32MB in a rw anonymous memory mapping for no discernable reason.

Version-Release number of selected component (if applicable):
python-2.4.2-3.2.1
python-elementtree-1.2.6-4.2.1

How reproducible:
Always

Steps to Reproduce:
1. Start a python interpreter
2. Run pmap on its pid
3. type 'import cElementTree' in the interpreter (or from elementtree import
ElementTree)
4. Run pmap again
  
Actual results:
The unsharable memory used by the python process increases by ~32MB.


Expected results:
The unsharable memory used by python increases somewhat, but certainly not by 32MB.


Additional info:

This happens on x86_64 (although the increase is slightly more pronounced) and
i386. But only under FC5, neither FC4 nor FC3 exhibit this behavior.  I was also
unable to find any other python modules that exhibited this behavior.  Reading
through the ElementTree code I don't see any reason this might happen at import,
either.  I'm baffled by the problem, hence this bug report.
Comment 2 Shahms E. King 2006-04-07 19:10:39 EDT
The problem is not with ElementTree, but can be traced to a specific regular
expression:

_escape = re.compile(u'[&<>"\x80-\uffff]+')

The memory usage doesn't go down after:

>>> del _escape
>>> re.purge()
>>> gc.collect()

either.  I suspect it's a bug/misplaced optimization in the re module, but
haven't investigated any further than that.  The specific problem with the
regular expression in the \x80-\uffff range.  Changing that to '\ufffa-\uffff'
gets rid of the astronomical memory usage.  I'm going to keep looking into this
as it's a particularly vexing bug ;-P
Comment 3 Jeremy Katz 2006-04-07 22:42:33 EDT
Can you try with the re module from the FC4 python?  That might help narrow down
if it's a change which has been made to python since then (which I don't see any
real candidates from a quick look at the changelog but it could be something in
python 2.4.2) or if it's something more along the lines of glibc or the compiler. 
Comment 4 Shahms E. King 2006-04-08 02:43:42 EDT
I installed the python 2.4.1 source into another prefix and tested using that
binary and re module, as well as using the 2.4.1 re module with the system
python and the problem persists in both cases.
Comment 5 Jeremy Katz 2006-04-10 14:56:03 EDT
Starting to sound like it might be something in the glibc regex code
Comment 6 Shahms E. King 2006-04-10 18:46:59 EDT
Poking around a little more, I discovered that the problem goes away in
non-optimized builds of python (either --with-pydebug or explicitly setting OPT
so as to exclude -O3).

Running the test case under valgrind --tool=massif shows that most or all of the
extra memory is consumed from calls to PyInt_FromLong.  In the non-optimized
python, most of the memory is allocated by new_arena.  I have the postscript
graphs and HTML massif output if you think they'd be useful.

The fact that the profile changes so dramatically simply by changing the
compiler optimizations leads me to think it may be a GCC bug.  I will try
compiling with an older gcc and see if the problem persists.
Comment 7 Shahms E. King 2006-04-11 13:17:57 EDT
A little more testing reveals the memory is allocated from a call to "range()"
within sre_compile.  Specifically, for the example regular expression given
above, the call is range(128, 65536).  A simplified test case is just the line:

range(128, 65536)

After still more testing, the problem is not present with compat-gcc-32-3.2.3-55
at any optimization level, nor is it present on gcc-4.1.0-3 using all of the
explicitly-named optimizations enabled by -O1, but does exist using -O1 and above.

This leads me to believe the problem is triggered by some unnamed optimization
performed by -O1.  At this point, I'm pretty sure this should be a gcc bug, but
I don't know enough about GCC or the "anonymous" optimizations performed by -O
to proceed much further.
Comment 8 Shahms E. King 2006-04-11 19:59:26 EDT
Created attachment 127641 [details]
Test case demonstrating gcc optimization bug
Comment 9 Jeremy Katz 2006-04-12 13:16:23 EDT
Okay, that's awesome -- thanks for the deep diving investigation here.  I've
gone ahead and rebuilt python for rawhide with the new gcc.   I'll watch the
other bug and we can push a rebuilt python if gcc is pushed as an update (misa
and I were discussing doing a 2.4.3 update for fc5 anyway)
Comment 10 Shahms E. King 2006-04-12 13:49:15 EDT
Not a problem, bug hunting can be fun :-).  I'm surprised more people haven't
been bitten by this particular bug, as it seems potentially nasty.

I pinged Jakub about pushing a new gcc.  Considering the Coverity defects fixed
in Python 2.4.3 it seems like a good idea to update it.  If gcc isn't updated
for fc5, recompiling the python package and explicitly enabling the named
options from -O3 should work as a temporary (and ugly) workaround.  But
considering the number of -f options that implies, it's definitely non-optimal.

Unfortunately, this bug makes using mod_python and ElementTree nearly impossible
as every httpd process consumes ~50MB :-(
Comment 11 Juha Sahakangas 2006-04-25 18:32:52 EDT
I was bitten by this, (PIL returns image data as huge lists of tuples of python
integers, which resulted in swap hell) but didn't have the time to file a bug at
the time.

Also managed to trace the problem to PyInt_FromLong, and noticed that other
python datatypes suffer from the same problem, for example float
(PyFloat_FromDouble), but some others, like long aren't affected.

The difference seems to be that ints and floats are allocated with PyMem_Malloc,
which is a pretty straightforward wrapper for malloc (#define PyMem_MALLOC(n)  
      malloc((n) ? (n) : 1)), whereas longs use python's much more complex
object allocator.
Comment 12 Juha Sahakangas 2006-04-25 18:35:33 EDT
As for others being bitten, bug 188977 is almost certainly a dupe of this. His
python takes long time to write the data simply because the two-million element
integer list takes a gigabyte of memory.
Comment 13 Michael Schwendt 2006-04-25 20:24:47 EDT
With regard to comment 10, is there any test update I could try
out to see whether it fixes the excessive heap space usage I'm
experiencing with FC5 Python?
Comment 14 Shahms E. King 2006-04-26 10:16:32 EDT
The python in rawhide has been recompiled against the fixed GCC (also in
rawhide) and "works" (i.e. it hasn't eaten any babies on my system lately).  It
might be worth giving that a try.  But note, I'm not recommending rawhide, just
saying the package has been fixed there and works for me ;-)
Comment 15 Jeremy Katz 2006-04-26 13:00:10 EDT
Michael -- the python in rawhide is safe to try for now.  We're trying to wait
for a fixed gcc in fc5-updates before pushing python
Comment 16 Jeremy Katz 2006-04-26 13:01:07 EDT
*** Bug 188977 has been marked as a duplicate of this bug. ***
Comment 17 Michael Schwendt 2006-04-26 14:28:13 EDT
Thanks. python-2.4.3-2 from rawhide running on fc5 is the cure here, too.

Previously, at least:

09572000-2d27d000 rw-p 09572000 00:00 0          [heap]
Size:            586796 kB
Rss:             540100 kB

Now at most:

08d50000-0d7ec000 rw-p 08d50000 00:00 0          [heap]
Size:             76400 kB
Rss:              76400 kB
Comment 18 Dawid Gajownik 2006-06-04 14:38:23 EDT
GCC 4.1.1 landed in updates repo few days ago. Would you be willing to push new
python package, too?

Thanks.
Comment 19 Jeremy Katz 2006-06-04 14:44:49 EDT
Yep -- the plan is to push python 2.4.3 to updates-testing first.  Mihai and I
were just both at the Red Hat Summit last week and so didn't get a chance to get
it done while there.
Comment 20 Mihai Ibanescu 2006-06-08 09:40:57 EDT
2.4.3-4.FC5 should fix this. Pushed to FC5 Testing.

Note You need to log in before you can comment on or make changes to this bug.