Bug 675863
| Summary: | memory corruption with perl threads | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Jeff Bastian <jbastian> |
| Component: | perl | Assignee: | Marcela Mašláňová <mmaslano> |
| Status: | CLOSED ERRATA | QA Contact: | Petr Šplíchal <psplicha> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 5.6 | CC: | ohudlick, ppisar, psabata, psplicha, spoyarek |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | perl-5.8.8-33.el5 | Doc Type: | Bug Fix |
| Doc Text: |
Cause
Rapidly creating and detaching threads on a multi-cpu/core system with Perl.
Consequence
It can eventually result in a variety of crashes.
Fix
Update of threads module to 1.79.
Result
Rapidly created and detached threads do not lead to crashes.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-02-21 05:40:17 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 641320 | ||
| Bug Blocks: | |||
Siddhesh Poyarekar has done some research and discovered this bug is fixed between 5.8.8 and 5.8.9 with commit e4ca037: http://perl5.git.perl.org/perl.git/commit/e4ca0379346a88607fedb72c8e4e73f68cb800f2 This commit is an aggregate of 65 commits from blead and it's a rather large patch to apply to RHEL-5. I'm not sure if we can narrow it down to a smaller patch, or if we would even want to. The various commits fix a number of race conditions, so it's likely that all the commits are necessary. Note, this is a follow-up to bug 548249 which dealt with memory leaking. This bug is about memory corruption. (In reply to comment #1) > Siddhesh Poyarekar has done some research and discovered this bug is fixed > between 5.8.8 and 5.8.9 with commit e4ca037: > http://perl5.git.perl.org/perl.git/commit/e4ca0379346a88607fedb72c8e4e73f68cb800f2 > > This commit is an aggregate of 65 commits from blead and it's a rather large > patch to apply to RHEL-5. > > I'm not sure if we can narrow it down to a smaller patch, or if we would even > want to. The various commits fix a number of race conditions, so it's likely > that all the commits are necessary. According to changed files in patch, it looks like as update of threads module to version 1.59. Users are often installing core modules from CPAN, so it could be working well. It will definitely need lot of testing. We can start with updating on RHEL-5 machine to CPAN threads v1.59 and check the test-case. Can somebody reproduce it? I gave a try in Qemu with 2 CPUs and I could not provoke the crash. What architecture do you have? I believe upgrade to perl 5.8.9 is worth to do it. Just, I'd like to have a reproducible reproducer. I also had trouble reproducing it with a KVM/QEMU virtual machine with 2 CPUs (on a host with a quad-core Core 2 CPU). But when I tried it on real hardware, a Sun x4500 with two dual-core AMD Opterons, it crashed pretty quickly. Yeap, you are right. I've just run test on real silicon (Intel(R) Xeon(R) CPU E5630 @ 2.53GHz) and glibc aborts perl now: *** glibc detected *** /usr/bin/perl: malloc(): memory corruption: 0x00002aaabc9791e0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x2afa997420fe] /lib64/libc.so.6(__libc_malloc+0x6e)[0x2afa99743e2e] /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(perl_clone+0x20)[0x2afa9851ba80] /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/threads/threads.so(Perl_ithread_create+0x174)[0x2afa9d01e9e4] /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/threads/threads.so(XS_threads_new+0x13f)[0x2afa9d01eecf] /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x3f6)[0x2afa98506a96] /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_standard+0xe)[0x2afa9850033e] /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(perl_run+0x30a)[0x2afa984ae08a] /usr/bin/perl(main+0xfc)[0x4017bc] /lib64/libc.so.6(__libc_start_main+0xf4)[0x2afa996ec994] /usr/bin/perl[0x401609] It just takes different time to the crash. I tested Perl 5.8.9 from upstream and the test program ran for 36 minutes with no issues (I killed with ^C). It usually crashes within 2 or 3 minutes with perl-5.8.8-32.el5_5.2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@sun-x4500-1 ~]$ date ; /usr/bin/perl ./test-threads.pl ; date Tue Feb 8 11:53:16 EST 2011 panic: MUTEX_UNLOCK (22) [threads.xs:134] during global destruction. Segmentation fault Tue Feb 8 11:53:44 EST 2011 [jbastian@sun-x4500-1 ~]$ /home/jbastian/local/bin/perl --version This is perl, v5.8.9 built for x86_64-linux-thread-multi ... [jbastian@sun-x4500-1 ~]$ date ; /home/jbastian/local/bin/perl ./test-threads.pl ; date Tue Feb 8 11:54:09 EST 2011 ^C Tue Feb 8 12:30:15 EST 2011 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For the record, here's how I built Perl 5.8.9: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tar xf perl-5.8.9.tar.bz2 cd perl-5.8.9 sh Configure -des \ -Dprefix=/home/jbastian/local \ -Dmyhostname=localhost \ -Dperladmin=root@localhost \ -Duseshrplib \ -Dusethreads \ -Duseithreads \ -Duselargefiles \ -Uuselargefiles \ -Ud_dosuid \ -Dd_semctl_semun \ -Di_db \ -Ui_ndbm \ -Di_gdbm \ -Di_shadow \ -Di_syslog \ -Dman3ext=3pm \ -Duseperlio \ -Dinstallusrbinperl=n \ -Ubincompat5005 \ -Uversiononly \ -Dpager='/usr/bin/less -isr' \ -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto \ -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto \ -Ud_endservent_r_proto -Ud_setservent_r_proto && \ make -j4 && \ make test && \ make install ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Cause
Rapidly creating and detaching threads on a multi-cpu/core system with Perl.
Consequence
It can eventually result in a variety of crashes.
Fix
Update of threads module to 1.79.
Result
Rapidly created and detached threads do not lead to crashes.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2012-0199.html |
Description of problem: Rapidly creating and detaching threads on a multi-cpu/core system with Perl will eventually result in a variety of crashes: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [root@sun-x4500-1 ~]# cat test-threads.pl #!/usr/bin/perl use threads; while (1) { threads->new( sub { 42; } )->detach(); } [root@sun-x4500-1 ~]# ./test-threads.pl *** glibc detected *** /usr/bin/perl: free(): corrupted unsorted chunks: 0x00002aaab03532a0 *** ======= Backtrace: ========= (snip) [root@sun-x4500-1 ~]# ./test-threads.pl panic: MUTEX_UNLOCK (22) [threads.xs:166] during global destruction. [root@sun-x4500-1 ~]# ./test-threads.pl *** glibc detected *** /usr/bin/perl: malloc(): memory corruption: 0x00002aaab198b130 *** ======= Backtrace: ========= (snip) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Version-Release number of selected component (if applicable): perl-5.8.8-32.el5_5.2 How reproducible: always on a multi-CPU/core system Steps to Reproduce: 1. create test-threads.pl script as shown above 2. ./test-threads.pl Actual results: perl crashes Expected results: no crash Additional info: It seems to crash a little quicker if you generate some other system activity, e.g., run 'rpm -Va'.