Bug 1104827

Summary: Perl core-dumps if a hash is tied to SDBM_File before spawning a thread
Product: Red Hat Enterprise Linux 6 Reporter: Terry Bowling <tbowling>
Component: perlAssignee: Petr Pisar <ppisar>
Status: CLOSED ERRATA QA Contact: Martin Kyral <mkyral>
Severity: medium Docs Contact:
Priority: urgent    
Version: 6.5CC: fkrska, jkurik, jorton, mkyral, ppisar, psabata, rvokal
Target Milestone: rcKeywords: Patch, ZStream
Target Release: 6.7   
Hardware: All   
OS: Linux   
URL: https://rt.perl.org/Ticket/Display.html?id=61912
Whiteboard:
Fixed In Version: perl-5.10.1-137.el6 Doc Type: Bug Fix
Doc Text:
Due to creating threads after tying a variable to an SDBM database using the SDBM_File Perl module, the Perl interpreter terminated unexpectedly when terminating Perl threads. With this update, the DB_File, GDBM_File, NDBM_File, ODBM_File, and SDBM_File Perl modules have been modified to destroy their objects only from the thread context which created the objects. As a result, the destructors of the aforementioned file objects are now thread-safe. Note, however, that other operations on the objects cannot be called from other threads. In general, the DB_File, GDBM_File, NDBM_File, ODBM_File, and SDBM_File Perl modules remain thread-unsafe.
Story Points: ---
Clone Of:
: 1107542 1161103 (view as bug list) Environment:
Last Closed: 2015-07-22 05:59:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1075802, 1159820, 1161103    
Attachments:
Description Flags
Minimal reproducer
none
Three-thread reproducer
none
Fix proposed to upstream
none
Automated test case for GDBM_File, NDBM_File, ODBM_File, SDBM_File
none
DB_File fix proposed to upstream none

Description Terry Bowling 2014-06-04 18:33:59 UTC
Description of problem:

Perl script using SDBM_File module is core dumping. Seems to match this upstream bug:  https://rt.perl.org/Public/Bug/Display.html?id=61912#txn-515026


Version-Release number of selected component (if applicable):

rhel6.5
perl-5.10.1-136.el6.x86_64


How reproducible:

The reproducer script from the above upstream bug report reproduces the issue perfectly on RHEL 6.5 perl-5.10.1-136.el6.x86_64.


Steps to Reproduce:

Create reproducer test script sdbm_test.pl containing the following lines, as described in the upstream bug report:

	#!/usr/bin/perl
	
	use strict;
	use Fcntl;
	use SDBM_File;
	use threads;
	use threads::shared;
	
	my %dbtest;
	tie(%dbtest, 'SDBM_File', "test.db", O_RDWR|O_CREAT, 0666);
	
	for (1 .. 2)
	{
	        my $thr = threads->new(\&testThread, $_);
	        $thr->detach();
	}
	sleep 4;
	
	sub testThread
	{
	        my $n = shift;
	        print "thread #" . $n . " started\n";
	}

Make script executable and run which produces the following output:

	[root@util6vm ~]# chmod u+x sdbm_test.pl 
	[root@util6vm ~]# ./sdbm_test.pl 

Expected results:

No errors.

Actual results:

	thread #1 started
	thread #2 started
	*** glibc detected *** /usr/bin/perl: double free or corruption (out): 0x0000000000e2c2c0 ***
	======= Backtrace: =========
	/lib64/libc.so.6[0x3d2ca76166]
	/lib64/libc.so.6[0x3d2ca78c93]
	/usr/lib64/perl5/auto/SDBM_File/SDBM_File.so(XS_SDBM_File_DESTROY+0xc0)[0x7f9d58fb06f0]
	/usr/lib64/perl5/CORE/libperl.so(Perl_pp_entersub+0x5a5)[0x31030a6815]
	/usr/lib64/perl5/CORE/libperl.so(Perl_call_sv+0x696)[0x310304c7a6]
	/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0xb6)[0x31030b8dd6]
	/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x31030b95d2]
	/usr/lib64/perl5/CORE/libperl.so[0x31030ae5c1]
	/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clean_objs+0x21)[0x31030ae621]
	/usr/lib64/perl5/CORE/libperl.so(perl_destruct+0x11d1)[0x310304e901]
	/usr/lib64/perl5/auto/threads/threads.so(+0x5516)[0x7f9d58da5516]
	/usr/lib64/perl5/auto/threads/threads.so(+0x5702)[0x7f9d58da5702]
	/usr/lib64/perl5/auto/threads/threads.so(+0x68b9)[0x7f9d58da68b9]
	/lib64/libpthread.so.0[0x3d2ce079d1]
	/lib64/libc.so.6(clone+0x6d)[0x3d2cae8b6d]
	======= Memory map: ========
	00400000-00402000 r-xp 00000000 fd:00 22083                              /usr/bin/perl
	00601000-00603000 rw-p 00001000 fd:00 22083                              /usr/bin/perl
	00c71000-00f26000 rw-p 00000000 00:00 0                                  [heap]
	3103000000-3103162000 r-xp 00000000 fd:00 32638                          /usr/lib64/perl5/CORE/libperl.so
	3103162000-3103362000 ---p 00162000 fd:00 32638                          /usr/lib64/perl5/CORE/libperl.so
	3103362000-310336b000 rw-p 00162000 fd:00 32638                          /usr/lib64/perl5/CORE/libperl.so
	3104000000-3104071000 r-xp 00000000 fd:00 9198                           /lib64/libfreebl3.so
	3104071000-3104270000 ---p 00071000 fd:00 9198                           /lib64/libfreebl3.so
	3104270000-3104272000 r--p 00070000 fd:00 9198                           /lib64/libfreebl3.so
	3104272000-3104273000 rw-p 00072000 fd:00 9198                           /lib64/libfreebl3.so
	3104273000-3104277000 rw-p 00000000 00:00 0 
	3105000000-3105007000 r-xp 00000000 fd:00 9439                           /lib64/libcrypt-2.12.so
	3105007000-3105207000 ---p 00007000 fd:00 9439                           /lib64/libcrypt-2.12.so
	3105207000-3105208000 r--p 00007000 fd:00 9439                           /lib64/libcrypt-2.12.so
	3105208000-3105209000 rw-p 00008000 fd:00 9439                           /lib64/libcrypt-2.12.so
	3105209000-3105237000 rw-p 00000000 00:00 0 
	3d2c200000-3d2c220000 r-xp 00000000 fd:00 1198                           /lib64/ld-2.12.so
	3d2c41f000-3d2c420000 r--p 0001f000 fd:00 1198                           /lib64/ld-2.12.so
	3d2c420000-3d2c421000 rw-p 00020000 fd:00 1198                           /lib64/ld-2.12.so
	3d2c421000-3d2c422000 rw-p 00000000 00:00 0 
	3d2c600000-3d2c602000 r-xp 00000000 fd:00 1296                           /lib64/libdl-2.12.so
	3d2c602000-3d2c802000 ---p 00002000 fd:00 1296                           /lib64/libdl-2.12.so
	3d2c802000-3d2c803000 r--p 00002000 fd:00 1296                           /lib64/libdl-2.12.so
	3d2c803000-3d2c804000 rw-p 00003000 fd:00 1296                           /lib64/libdl-2.12.so
	3d2ca00000-3d2cb8b000 r-xp 00000000 fd:00 1199                           /lib64/libc-2.12.so
	3d2cb8b000-3d2cd8a000 ---p 0018b000 fd:00 1199                           /lib64/libc-2.12.so
	3d2cd8a000-3d2cd8e000 r--p 0018a000 fd:00 1199                           /lib64/libc-2.12.so
	3d2cd8e000-3d2cd8f000 rw-p 0018e000 fd:00 1199                           /lib64/libc-2.12.so
	3d2cd8f000-3d2cd94000 rw-p 00000000 00:00 0 
	3d2ce00000-3d2ce17000 r-xp 00000000 fd:00 1295                           /lib64/libpthread-2.12.so
	3d2ce17000-3d2d017000 ---p 00017000 fd:00 1295                           /lib64/libpthread-2.12.so
	3d2d017000-3d2d018000 r--p 00017000 fd:00 1295                           /lib64/libpthread-2.12.so
	3d2d018000-3d2d019000 rw-p 00018000 fd:00 1295                           /lib64/libpthread-2.12.so
	3d2d019000-3d2d01d000 rw-p 00000000 00:00 0 
	3d2d600000-3d2d683000 r-xp 00000000 fd:00 1297                           /lib64/libm-2.12.so
	3d2d683000-3d2d882000 ---p 00083000 fd:00 1297                           /lib64/libm-2.12.so
	3d2d882000-3d2d883000 r--p 00082000 fd:00 1297                           /lib64/libm-2.12.so
	3d2d883000-3d2d884000 rw-p 00083000 fd:00 1297                           /lib64/libm-2.12.so
	3d2e200000-3d2e216000 r-xp 00000000 fd:00 12310                          /lib64/libnsl-2.12.so
	3d2e216000-3d2e415000 ---p 00016000 fd:00 12310                          /lib64/libnsl-2.12.so
	3d2e415000-3d2e416000 r--p 00015000 fd:00 12310                          /lib64/libnsl-2.12.so
	3d2e416000-3d2e417000 rw-p 00016000 fd:00 12310                          /lib64/libnsl-2.12.so
	3d2e417000-3d2e419000 rw-p 00000000 00:00 0 
	3d2e600000-3d2e616000 r-xp 00000000 fd:00 13855                          /lib64/libresolv-2.12.so
	3d2e616000-3d2e816000 ---p 00016000 fd:00 13855                          /lib64/libresolv-2.12.so
	3d2e816000-3d2e817000 r--p 00016000 fd:00 13855                          /lib64/libresolv-2.12.so
	3d2e817000-3d2e818000 rw-p 00017000 fd:00 13855                          /lib64/libresolv-2.12.so
	3d2e818000-3d2e81a000 rw-p 00000000 00:00 0 
	3d2ee00000-3d2ee16000 r-xp 00000000 fd:00 1393                           /lib64/libgcc_s-4.4.7-20120601.so.1
	3d2ee16000-3d2f015000 ---p 00016000 fd:00 1393                           /lib64/libgcc_s-4.4.7-20120601.so.1
	3d2f015000-3d2f016000 rw-p 00015000 fd:00 1393                           /lib64/libgcc_s-4.4.7-20120601.so.1
	3d35200000-3d35202000 r-xp 00000000 fd:00 12311                          /lib64/libutil-2.12.so
	3d35202000-3d35401000 ---p 00002000 fd:00 12311                          /lib64/libutil-2.12.so
	3d35401000-3d35402000 r--p 00001000 fd:00 12311                          /lib64/libutil-2.12.so
	3d35402000-3d35403000 rw-p 00002000 fd:00 12311                          /lib64/libutil-2.12.so
	7f9d48000000-7f9d48021000 rw-p 00000000 00:00 0 
	7f9d48021000-7f9d4c000000 ---p 00000000 00:00 0 
	7f9d50000000-7f9d50021000 rw-p 00000000 00:00 0 
	7f9d50021000-7f9d54000000 ---p 00000000 00:00 0 
	7f9d5758a000-7f9d5758b000 ---p 00000000 00:00 0 
	7f9d5758b000-7f9d57f8b000 rw-p 00000000 00:00 0 
	7f9d57f8b000-7f9d57f8c000 ---p 00000000 00:00 0 
	7f9d57f8c000-7f9d5898c000 rw-p 00000000 00:00 0 
	7f9d5898c000-7f9d58996000 r-xp 00000000 fd:00 132958                     /usr/lib64/perl5/auto/threads/shared/shared.so
	7f9d58996000-7f9d58b96000 ---p 0000a000 fd:00 132958                     /usr/lib64/perl5/auto/threads/shared/shared.so
	7f9d58b96000-7f9d58b97000 rw-p 0000a000 fd:00 132958                     /usr/lib64/perl5/auto/threads/shared/shared.soAborted (core dumped)

Comment 1 Terry Bowling 2014-06-04 18:38:59 UTC
Request to keep this BZ public for customer visibility.

Comment 3 Petr Pisar 2014-06-05 05:56:58 UTC
This happens with any perl, even the development version.

Comment 4 Petr Pisar 2014-06-05 06:00:06 UTC
Created attachment 902406 [details]
Minimal reproducer

Comment 5 Petr Pisar 2014-06-05 13:06:53 UTC
Created attachment 902531 [details]
Three-thread reproducer

Unfortunately the 2-thread "Minimal reproducer" does not fail always. This test uses a thread more which is more reliable.

Comment 6 Petr Pisar 2014-06-05 13:30:04 UTC
While I understand the surprise from the the crash, I believe this is intended behavior.

You create an object (the tie() call) which allocates a global resource (opens a database by sdbm_open()), then create a bunch of threads which duplicates the object making each replica private to each respective thread. Then when each thread terminates, each replica is destroyed independently (and possibly concurrently) which involves deallocating a global resource (close a database by sdbm_close()). Because sdbm library is not thread-safe, the database handle gets closed multiple times leading to the crash.

Obvious solution is to untie the variable before spawning a new thread.

While it could be possible to improve the SDBM_File module to globally track each tied database so that each database got closed only once when destroying the tied hash, it would not still not make the module completely thread-safe. E.g. any store operation would corrupted data visible from other threads.

Additionally I'd like to quote "Thread-Safe Modules" paragraph from perlthrtut manual:

    Not all modules that you might use are thread-safe, and you should always
    assume a module is unsafe unless the documentation says otherwise. This
    includes modules that are distributed as part of the core.

SDBM_File documentation does not declare the thread-safety.

Comment 7 Petr Pisar 2014-06-10 07:19:46 UTC
Created attachment 907058 [details]
Fix proposed to upstream

Comment 8 Petr Pisar 2014-06-10 11:51:20 UTC
Created attachment 907181 [details]
Automated test case for GDBM_File, NDBM_File, ODBM_File, SDBM_File

This tests GDBM_File, NDBM_File, ODBM_File, SDBM_File.

Separate patch (and maybe a test) will be provided for DB_File as it has different upstream.

Comment 9 Petr Pisar 2014-06-10 13:17:12 UTC
Created attachment 907248 [details]
DB_File fix proposed to upstream

Comment 12 Branislav NĂ¡ter 2014-11-06 11:28:46 UTC
Test available => qa_ack for rhel-6.6.z

Comment 15 Petr Pisar 2014-12-02 12:55:19 UTC
*** Bug 1169705 has been marked as a duplicate of this bug. ***

Comment 19 errata-xmlrpc 2015-07-22 05:59:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1266.html