Description of problem:
Importing the cElementTree module causes the python process to allocate an extra
32MB in a rw anonymous memory mapping for no discernable reason.

Version-Release number of selected component (if applicable):
python-2.4.2-3.2.1
python-elementtree-1.2.6-4.2.1

How reproducible:
Always

Steps to Reproduce:
1. Start a python interpreter
2. Run pmap on its pid
3. type 'import cElementTree' in the interpreter (or from elementtree import
ElementTree)
4. Run pmap again
  
Actual results:
The unsharable memory used by the python process increases by ~32MB.


Expected results:
The unsharable memory used by python increases somewhat, but certainly not by 32MB.


Additional info:

This happens on x86_64 (although the increase is slightly more pronounced) and
i386. But only under FC5, neither FC4 nor FC3 exhibit this behavior.  I was also
unable to find any other python modules that exhibited this behavior.  Reading
through the ElementTree code I don't see any reason this might happen at import,
either.  I'm baffled by the problem, hence this bug report.

The problem is not with ElementTree, but can be traced to a specific regular
expression:

_escape = re.compile(u'[&<>"\x80-\uffff]+')

The memory usage doesn't go down after:

>>> del _escape
>>> re.purge()
>>> gc.collect()

either.  I suspect it's a bug/misplaced optimization in the re module, but
haven't investigated any further than that.  The specific problem with the
regular expression in the \x80-\uffff range.  Changing that to '\ufffa-\uffff'
gets rid of the astronomical memory usage.  I'm going to keep looking into this
as it's a particularly vexing bug ;-P

Can you try with the re module from the FC4 python?  That might help narrow down
if it's a change which has been made to python since then (which I don't see any
real candidates from a quick look at the changelog but it could be something in
python 2.4.2) or if it's something more along the lines of glibc or the compiler. 

I installed the python 2.4.1 source into another prefix and tested using that
binary and re module, as well as using the 2.4.1 re module with the system
python and the problem persists in both cases.

Starting to sound like it might be something in the glibc regex code

Poking around a little more, I discovered that the problem goes away in
non-optimized builds of python (either --with-pydebug or explicitly setting OPT
so as to exclude -O3).

Running the test case under valgrind --tool=massif shows that most or all of the
extra memory is consumed from calls to PyInt_FromLong.  In the non-optimized
python, most of the memory is allocated by new_arena.  I have the postscript
graphs and HTML massif output if you think they'd be useful.

The fact that the profile changes so dramatically simply by changing the
compiler optimizations leads me to think it may be a GCC bug.  I will try
compiling with an older gcc and see if the problem persists.

A little more testing reveals the memory is allocated from a call to "range()"
within sre_compile.  Specifically, for the example regular expression given
above, the call is range(128, 65536).  A simplified test case is just the line:

range(128, 65536)

After still more testing, the problem is not present with compat-gcc-32-3.2.3-55
at any optimization level, nor is it present on gcc-4.1.0-3 using all of the
explicitly-named optimizations enabled by -O1, but does exist using -O1 and above.

This leads me to believe the problem is triggered by some unnamed optimization
performed by -O1.  At this point, I'm pretty sure this should be a gcc bug, but
I don't know enough about GCC or the "anonymous" optimizations performed by -O
to proceed much further.

Created attachment 127641 [details]
Test case demonstrating gcc optimization bug

Okay, that's awesome -- thanks for the deep diving investigation here.  I've
gone ahead and rebuilt python for rawhide with the new gcc.   I'll watch the
other bug and we can push a rebuilt python if gcc is pushed as an update (misa
and I were discussing doing a 2.4.3 update for fc5 anyway)

Not a problem, bug hunting can be fun :-).  I'm surprised more people haven't
been bitten by this particular bug, as it seems potentially nasty.

I pinged Jakub about pushing a new gcc.  Considering the Coverity defects fixed
in Python 2.4.3 it seems like a good idea to update it.  If gcc isn't updated
for fc5, recompiling the python package and explicitly enabling the named
options from -O3 should work as a temporary (and ugly) workaround.  But
considering the number of -f options that implies, it's definitely non-optimal.

Unfortunately, this bug makes using mod_python and ElementTree nearly impossible
as every httpd process consumes ~50MB :-(

I was bitten by this, (PIL returns image data as huge lists of tuples of python
integers, which resulted in swap hell) but didn't have the time to file a bug at
the time.

Also managed to trace the problem to PyInt_FromLong, and noticed that other
python datatypes suffer from the same problem, for example float
(PyFloat_FromDouble), but some others, like long aren't affected.

The difference seems to be that ints and floats are allocated with PyMem_Malloc,
which is a pretty straightforward wrapper for malloc (#define PyMem_MALLOC(n)  
      malloc((n) ? (n) : 1)), whereas longs use python's much more complex
object allocator.

As for others being bitten, bug 188977 is almost certainly a dupe of this. His
python takes long time to write the data simply because the two-million element
integer list takes a gigabyte of memory.

With regard to comment 10, is there any test update I could try
out to see whether it fixes the excessive heap space usage I'm
experiencing with FC5 Python?

The python in rawhide has been recompiled against the fixed GCC (also in
rawhide) and "works" (i.e. it hasn't eaten any babies on my system lately).  It
might be worth giving that a try.  But note, I'm not recommending rawhide, just
saying the package has been fixed there and works for me ;-)

Michael -- the python in rawhide is safe to try for now.  We're trying to wait
for a fixed gcc in fc5-updates before pushing python

*** Bug 188977 has been marked as a duplicate of this bug. ***

Thanks. python-2.4.3-2 from rawhide running on fc5 is the cure here, too.

Previously, at least:

09572000-2d27d000 rw-p 09572000 00:00 0          [heap]
Size:            586796 kB
Rss:             540100 kB

Now at most:

08d50000-0d7ec000 rw-p 08d50000 00:00 0          [heap]
Size:             76400 kB
Rss:              76400 kB

GCC 4.1.1 landed in updates repo few days ago. Would you be willing to push new
python package, too?

Thanks.

Yep -- the plan is to push python 2.4.3 to updates-testing first.  Mihai and I
were just both at the Red Hat Summit last week and so didn't get a chance to get
it done while there.

2.4.3-4.FC5 should fix this. Pushed to FC5 Testing.