Bug 548249 - Perl threads leak memory when detached
Perl threads leak memory when detached
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: perl (Show other bugs)
All Linux
urgent Severity high
: rc
: ---
Assigned To: perl-maint-list
Petr Šplíchal
: Patch, ZStream
Depends On: 641320
Blocks: 502912 590060 701632
  Show dependency treegraph
Reported: 2009-12-16 20:36 EST by Bryan Mason
Modified: 2016-05-31 21:38 EDT (History)
11 users (show)

See Also:
Fixed In Version: perl-5.8.8-29.el5
Doc Type: Bug Fix
Doc Text:
Cause Perl appears to leak memory when threads are created and then detached. Consequence The program consumed memory, seemingly without bounds. Fix Perl internal is freed. Result No huge memory leaks after detaching of threads.
Story Points: ---
Clone Of:
Last Closed: 2012-02-21 00:39:41 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Script measuring leaked memory per thread (2.44 KB, application/octet-stream)
2010-09-01 11:02 EDT, Petr Pisar
no flags Details
Script measuring leaked memory per thread (2.71 KB, text/plain)
2010-09-02 08:19 EDT, Petr Pisar
no flags Details
Proposed fix (61.79 KB, patch)
2010-09-06 09:50 EDT, Petr Pisar
no flags Details | Diff
Reduced fix (350 bytes, patch)
2010-12-06 09:56 EST, Siddhesh Poyarekar
no flags Details | Diff

  None (edit)
Description Bryan Mason 2009-12-16 20:36:51 EST
Description of problem:

    Perl appears to leak memory when threads are created and 
    then detached.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
    1.  Create and run simple test program like:


        use threads;

        my $subref = sub { 42; };

        my $t;

        while(1) {

Actual results:
    The test program will consume memory, seemingly without bounds.

    Running "valgrind --leak-check=full ..." on the test program
    reports bytes that are "definitely lost."

Expected results:
    No memory lost.  If the thread is detached, it should release its
    memory after exiting.

Additional info:

I think the root cause of the problem may be somewhere in Perl_ithread_destruct () in ext/threads/thread.xs.  Specifically, lines 137-140:

    126 /*
    127  *  free an ithread structure and any attached data if its count == 0
    128  */
    129 void
    130 Perl_ithread_destruct (pTHX_ ithread* thread, const char *why)
    131 {
    132         MUTEX_LOCK(&thread->mutex);
    133         if (!thread->next) {
    134             MUTEX_UNLOCK(&thread->mutex);
    135             Perl_croak(aTHX_ "panic: destruct destroyed thread %p (%s)",
    135 thread, why);
    136         }
    137         if (thread->count != 0) {
    138                 MUTEX_UNLOCK(&thread->mutex);
    139                 return;
    140         }
    141         MUTEX_LOCK(&create_destruct_mutex);
    142         /* Remove from circular list of threads */
    143         if (thread->next == thread) {
    144             /* last one should never get here ? */
    145             threads = NULL;
    146         }
    147         else {
    148             thread->next->prev = thread->prev;
    149             thread->prev->next = thread->next;
    150             if (threads == thread) {
    151                 threads = thread->next;
    152             }
    153             thread->next = NULL;
    154             thread->prev = NULL;
    155         }
    156         known_threads--;
    157         assert( known_threads >= 0 );
    158 #if 0
    159         Perl_warn(aTHX_ "destruct %d @ %p by %p now %d",
    160                   thread->tid,thread->interp,aTHX, known_threads);
    161 #endif
    162         MUTEX_UNLOCK(&create_destruct_mutex);
    163         /* Thread is now disowned */
    165         S_ithread_clear(aTHX_ thread);
    166         MUTEX_UNLOCK(&thread->mutex);
    167         MUTEX_DESTROY(&thread->mutex);
    168 #ifdef WIN32
    169         if (thread->handle)
    170             CloseHandle(thread->handle);
    171         thread->handle = 0;
    172 #endif
    173         PerlMemShared_free(thread);
    174 }

While running the perl test program under gdb, the following was discovered:

    Starting program: /usr/bin/perl ./test2.pl
    [Thread debugging using libthread_db enabled]
    [New Thread 0x2b8a73ba0250 (LWP 1680)]
    [New Thread 0x41b64940 (LWP 1683)]
    [Switching to Thread 0x41b64940 (LWP 1683)]
    Breakpoint 1, Perl_ithread_destruct (my_perl=0xa0d0590, thread=0xa0ca900,
        why=0x2b8a7719f0d3 "detached finish") at threads.xs:131
    131     {
    (gdb) display thread->count
    1: thread->count = 2

    Because thread->count == 2 when the thread is destroyed,
    Perl_ithread_destruct() exits immediately and S_ithread_clear()
    (which actually cleans up the thread memory) is never called.

    I'm not sure if having thread->count == 2 when
    Perl_ithread_destruct() is called is correct or not, but it does
    seem to be preventing cleanup of the Perl thread's memory.

As a quick hack, I tried adding the following patch:

    --- perl-5.8.8.old/ext/threads/threads.xs 2009-12-15 12:02:49.940004000 -0800
    +++ perl-5.8.8.new/ext/threads/threads.xs 2009-12-15 13:06:30.493553000 -0800
    @@ -201,6 +201,11 @@
    +    /* Make sure the thread is cleaned up after it's done */
    +    if ( thread->count > 0 )
    +       thread->count--;
        if ((thread->state & PERL_ITHR_FINISHED) &&
            (thread->state & PERL_ITHR_DETACHED)) {

but that resulted in the following error during the build:

    ext/threads/t/basic.......................*** glibc detected *** ./perl:
    munmap_chunk(): invalid pointer: 0x00000000097a11f0 ***
    ======= Backtrace: =========

There are reports of problems like this upstream:


These seem to have been resolved in the latest version of perl, but the upstream threading code seems to be significantly different, so I'm not sure exactly what changed to fix this.

This can be worked around by explicitly undef-ing the thread variable in the loop, but another problem, as described in Bug 537777, is preventing this.
Comment 7 RHEL Product and Program Management 2010-08-09 14:55:26 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 8 Petr Pisar 2010-08-27 10:01:42 EDT
The test case is wrong as it can create threads faster than they terminate.

Better test case:

use threads;

sub ThreadRoutine {}

while (1) {
    $thread = threads->new(\&ThreadRoutine);
Comment 9 Petr Pisar 2010-09-01 11:02:24 EDT
Created attachment 442431 [details]
Script measuring leaked memory per thread

This code can be used to test how much memory leaks per thread.

The memory must be ballooned in memory() function to prevent other leaks/fragmentation of different type. Also calling the memory() as &memory is important part of ballooning (yet another bug).
Comment 10 Petr Pisar 2010-09-01 11:08:42 EDT
This bug presents in 5.8 branch only (5.9.0 and higher are not affected).

5.8.0 is ok, 5.8.8 tip is broken. I identified 
5f2f4dbeb153fb32f97b9eec41efcf5c3e2f2a81 commit (some Win32 related things) as first one introducing this bug.

However simple revert does not fix 5.8.8. There are other bugs introduced later after the commit probably. I will try to found them (319 interleaving commits).
Comment 11 Petr Pisar 2010-09-02 08:19:53 EDT
Created attachment 442606 [details]
Script measuring leaked memory per thread

Former test contained a race (measure while running worker thread). This one is better. However it does behaves differently depending on ballooning.

Bisects show two different places where bug has been introduced:

3848b9623cf7b0bd4a28a02073a51953b168b7a6 Bad (perl-5.8.0-749-g3848b96)
26ab6a78adda605ad55978d97dd5c8d4b282375a Bad (perl-5.8.7-76-g26ab6a7)

I'm still trying to find the latest one.
Comment 12 Petr Pisar 2010-09-06 09:50:43 EDT
Created attachment 443303 [details]
Proposed fix

This patch reverts 26ab6a78adda605ad55978d97dd5c8d4b282375a (1200 lines) that introduced the memory leaks in perl-5.8.8.

The reverted patch changed a way ho to manipulate with perl variables to get simpler (and maybe faster) code.

With the revert, all tests bundled within perl-5.8.8 pass (except those that failed before either), the measurement script shows 0 leaks, the test case from this bug does not grow perl memory.

Everything seems O.k. so far, but I'm not able to tell whether this patch introduce some other (performance) bugs. It's quite long patch.
Comment 13 Petr Pisar 2010-09-07 09:18:21 EDT
5.8.9 (last maintenance version of 5.8 released by upstream) and HEAD of maint-5.8 branch (VCS branch for 5.8 development) is affected too. Thus my revert is currently only way how to fix this issue without rebasing to 5.10.
Comment 26 Siddhesh Poyarekar 2010-12-06 09:56:10 EST
Created attachment 464991 [details]
Reduced fix

I went through the patch today and reduced it to the one-liner attached. This is working the same as the 12k+ chunk in terms of memory allocation and does not seem to have any regressions.
Comment 44 Marcela Mašláňová 2011-09-26 03:26:18 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    New Contents:
    Perl appears to leak memory when threads are created and then detached.
    The program consumed memory, seemingly without bounds.
    Perl internal is freed.
    No huge memory leaks after detaching of threads.
Comment 46 errata-xmlrpc 2012-02-21 00:39:41 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.