Bug 675863 - memory corruption with perl threads
memory corruption with perl threads
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: perl (Show other bugs)
5.6
Unspecified Linux
high Severity high
: rc
: ---
Assigned To: Marcela Mašláňová
Petr Šplíchal
:
Depends On: 641320
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-07 19:57 EST by Jeff Bastian
Modified: 2012-02-21 00:40 EST (History)
4 users (show)

See Also:
Fixed In Version: perl-5.8.8-33.el5
Doc Type: Bug Fix
Doc Text:
Cause Rapidly creating and detaching threads on a multi-cpu/core system with Perl. Consequence It can eventually result in a variety of crashes. Fix Update of threads module to 1.79. Result Rapidly created and detached threads do not lead to crashes.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-02-21 00:40:17 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Jeff Bastian 2011-02-07 19:57:38 EST
Description of problem:
Rapidly creating and detaching threads on a multi-cpu/core system with Perl will eventually result in a variety of crashes:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[root@sun-x4500-1 ~]# cat test-threads.pl 
#!/usr/bin/perl               
use threads; 
while (1) { 
        threads->new( sub { 42; } )->detach(); 
}

[root@sun-x4500-1 ~]# ./test-threads.pl 
*** glibc detected *** /usr/bin/perl: free(): corrupted unsorted chunks: 0x00002aaab03532a0 ***
======= Backtrace: =========
(snip)

[root@sun-x4500-1 ~]# ./test-threads.pl
panic: MUTEX_UNLOCK (22) [threads.xs:166] during global destruction.

[root@sun-x4500-1 ~]# ./test-threads.pl
*** glibc detected *** /usr/bin/perl: malloc(): memory corruption: 0x00002aaab198b130 ***
======= Backtrace: =========
(snip)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Version-Release number of selected component (if applicable):
perl-5.8.8-32.el5_5.2

How reproducible:
always on a multi-CPU/core system

Steps to Reproduce:
1. create test-threads.pl script as shown above
2. ./test-threads.pl
  
Actual results:
perl crashes

Expected results:
no crash

Additional info:
It seems to crash a little quicker if you generate some other system activity, e.g., run 'rpm -Va'.
Comment 1 Jeff Bastian 2011-02-07 20:07:59 EST
Siddhesh Poyarekar has done some research and discovered this bug is fixed between 5.8.8 and 5.8.9 with commit e4ca037:
http://perl5.git.perl.org/perl.git/commit/e4ca0379346a88607fedb72c8e4e73f68cb800f2

This commit is an aggregate of 65 commits from blead and it's a rather large patch to apply to RHEL-5.

I'm not sure if we can narrow it down to a smaller patch, or if we would even want to.  The various commits fix a number of race conditions, so it's likely that all the commits are necessary.
Comment 2 Jeff Bastian 2011-02-07 23:49:10 EST
Note, this is a follow-up to bug 548249 which dealt with memory leaking.  This bug is about memory corruption.
Comment 3 Marcela Mašláňová 2011-02-08 03:36:16 EST
(In reply to comment #1)
> Siddhesh Poyarekar has done some research and discovered this bug is fixed
> between 5.8.8 and 5.8.9 with commit e4ca037:
> http://perl5.git.perl.org/perl.git/commit/e4ca0379346a88607fedb72c8e4e73f68cb800f2
> 
> This commit is an aggregate of 65 commits from blead and it's a rather large
> patch to apply to RHEL-5.
> 
> I'm not sure if we can narrow it down to a smaller patch, or if we would even
> want to.  The various commits fix a number of race conditions, so it's likely
> that all the commits are necessary.

According to changed files in patch, it looks like as update of threads module to version 1.59. Users are often installing core modules from CPAN, so it could be working well. It will definitely need lot of testing. We can start with updating on RHEL-5 machine to CPAN threads v1.59 and check the test-case.
Comment 4 Petr Pisar 2011-02-08 10:45:28 EST
Can somebody reproduce it? I gave a try in Qemu with 2 CPUs and I could not provoke the crash. What architecture do you have? I believe upgrade to perl 5.8.9 is worth to do it. Just, I'd like to have a reproducible reproducer.
Comment 5 Jeff Bastian 2011-02-08 10:58:58 EST
I also had trouble reproducing it with a KVM/QEMU virtual machine with 2 CPUs (on a host with a quad-core Core 2 CPU).  But when I tried it on real hardware, a Sun x4500 with two dual-core AMD Opterons, it crashed pretty quickly.
Comment 6 Petr Pisar 2011-02-08 11:22:11 EST
Yeap, you are right. I've just run test on real silicon (Intel(R) Xeon(R) CPU E5630 @ 2.53GHz) and glibc aborts perl now:

*** glibc detected *** /usr/bin/perl: malloc(): memory corruption: 0x00002aaabc9791e0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2afa997420fe]
/lib64/libc.so.6(__libc_malloc+0x6e)[0x2afa99743e2e]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(perl_clone+0x20)[0x2afa9851ba80]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/threads/threads.so(Perl_ithread_create+0x174)[0x2afa9d01e9e4]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/threads/threads.so(XS_threads_new+0x13f)[0x2afa9d01eecf]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x3f6)[0x2afa98506a96]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_standard+0xe)[0x2afa9850033e]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(perl_run+0x30a)[0x2afa984ae08a]
/usr/bin/perl(main+0xfc)[0x4017bc]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2afa996ec994]
/usr/bin/perl[0x401609]

It just takes different time to the crash.
Comment 7 Jeff Bastian 2011-02-08 12:34:35 EST
I tested Perl 5.8.9 from upstream and the test program ran for 36 minutes with no issues (I killed with ^C).  It usually crashes within 2 or 3 minutes with perl-5.8.8-32.el5_5.2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@sun-x4500-1 ~]$ date ; /usr/bin/perl ./test-threads.pl ; date
Tue Feb  8 11:53:16 EST 2011
panic: MUTEX_UNLOCK (22) [threads.xs:134] during global destruction.
Segmentation fault
Tue Feb  8 11:53:44 EST 2011

[jbastian@sun-x4500-1 ~]$ /home/jbastian/local/bin/perl --version

This is perl, v5.8.9 built for x86_64-linux-thread-multi
...

[jbastian@sun-x4500-1 ~]$ date ; /home/jbastian/local/bin/perl ./test-threads.pl ; date
Tue Feb  8 11:54:09 EST 2011
^C
Tue Feb  8 12:30:15 EST 2011
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


For the record, here's how I built Perl 5.8.9:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tar xf perl-5.8.9.tar.bz2
cd perl-5.8.9
sh Configure -des \
        -Dprefix=/home/jbastian/local \
        -Dmyhostname=localhost \
        -Dperladmin=root@localhost \
        -Duseshrplib \
        -Dusethreads \
        -Duseithreads \
        -Duselargefiles \
        -Uuselargefiles \
        -Ud_dosuid \
        -Dd_semctl_semun \
        -Di_db \
        -Ui_ndbm \
        -Di_gdbm \
        -Di_shadow \
        -Di_syslog \
        -Dman3ext=3pm \
        -Duseperlio \
        -Dinstallusrbinperl=n \
        -Ubincompat5005 \
        -Uversiononly \
        -Dpager='/usr/bin/less -isr' \
        -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto \
        -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto \
        -Ud_endservent_r_proto -Ud_setservent_r_proto && \
make -j4 && \
make test && \
make install
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Comment 8 RHEL Product and Program Management 2011-05-31 10:08:45 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 17 Marcela Mašláňová 2011-09-26 03:42:53 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
    Rapidly creating and detaching threads on a multi-cpu/core system with Perl.
Consequence
    It can eventually result in a variety of crashes.
Fix
    Update of threads module to 1.79.
Result
    Rapidly created and detached threads do not lead to crashes.
Comment 26 errata-xmlrpc 2012-02-21 00:40:17 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0199.html

Note You need to log in before you can comment on or make changes to this bug.