__nptl_deallocate_tsd is calling destructors registered with pthread_key_create.
The crash is apparently when it tries to call one of such handlers.
My guess would be that some shared library is calling pthread_key_create and registering some destructor in that shared library, but forgotting to pthread_key_delete in library destructors.  Therefore, if you dlopen the library, get a key created, then dlclose the library and then some thread is about to exit, it will try to call a function in an already dlclosed library.

This is regression in libselinux-2.0.94-3.el6. It does not segfault with libselinux-2.0.94-2.el6.

It needed to come with the following 

* Fri Feb 04 2011 Miroslav Grepl <mgrepl> - 2.0.94-3
- Thread local storage fixes from Eamon Walsh
Resolves: #658571

which resolves the #658571 bug.

Created attachment 481728 [details]
Proposed patch 1

First cut of fix.

Mirek,
could you test pkgs which I added on

http://scratch.englab.brq.redhat.com/mgrepl/RHEL6/

Looks like it does not work.

Are these other test packages than I tried yesterday? I don't see the segfault happening anymore :)

(In reply to comment #8)
> Are these other test packages than I tried yesterday? I don't see the segfault
> happening anymore :)

So it works.

Pushed to libselinux 2.0.100.

However I would note that pthread_key_delete does not call the key destructors to clean up the data.  This job is left to the caller, and I have no idea how to find all of the threads and clean up their data from the library destructor.  Some advice on how to go about doing this would be beneficial.

The patch added to libselinux-2.0.94-4.el6

Bug report changed from ON_QA to ASSIGNED status by the Errata System: 
Advisory RHBA-2011:10756-01: 
http://errata.devel.redhat.com/errata/stateview/10756

# rpm -qa | grep -e pam- -e libselinux | sort
libselinux-2.0.94-4.el6.x86_64
libselinux-debuginfo-2.0.94-4.el6.x86_64
libselinux-devel-2.0.94-4.el6.x86_64
libselinux-python-2.0.94-4.el6.x86_64
libselinux-ruby-2.0.94-4.el6.x86_64
libselinux-static-2.0.94-4.el6.x86_64
libselinux-utils-2.0.94-4.el6.x86_64
pam-1.1.1-8.el6.x86_64
pam-devel-1.1.1-8.el6.x86_64
# useradd pamtest
# g++ -pthread -lpam MTPamTest.cpp -o MTPamTest
# ./MTPamTest pamtest test

Changing password from main thread
pam_start SUCCESSFUL
pam_chauthtok SUCCESSFUL
pam_end SUCCESSFUL

Changing password from child thread
pthread_create SUCCESSFUL
pam_start SUCCESSFUL
pam_chauthtok SUMTPamTest[3073]: segfault at 19 ip 00007fdafd191266 sp 00007fdafc281de0 error 4CCESSFUL
 in libdl-2.12.so[7fdafd190000+2000]
Segmentation fault (core dumped)
# echo $?
139

(gdb) thread apply all bt

Thread 2 (Thread 0x7ffff5b4a700 (LWP 3213)):
#0  0x00007ffff6a59266 in _dlerror_run () from /lib64/libdl.so.2
#1  0x00007ffff6a5900f in dlclose () from /lib64/libdl.so.2
#2  0x00007ffff7bd6797 in _pam_free_handlers (pamh=0x7ffff00008c0)
    at pam_handlers.c:949
#3  0x00007ffff7bd5165 in pam_end (pamh=0x7ffff00008c0, pam_status=0)
    at pam_end.c:35
#4  0x0000000000400f3a in (anonymous namespace)::changePassword ()
    at MTPamTest.cpp:102
#5  0x00007ffff721c7e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff6f58d2d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fe7720 (LWP 3206)):
#0  0x00007ffff721d03d in pthread_join () from /lib64/libpthread.so.0
#1  0x000000000040114a in main (argc=3, argv=0x7fffffffe9f8)
    at MTPamTest.cpp:138
(gdb)

(gdb) run
Starting program: /mnt/testarea/tests/pam/Regression/bz469857-pam-chauthtok-works-in-main-but-not-child-threads/MTPamTest pamtest test
[Thread debugging using libthread_db enabled]

Changing password from main thread
pam_start SUCCESSFUL
Detaching after fork from child process 3209.
Detaching after fork from child process 3210.
Detaching after fork from child process 3212.
pam_chauthtok SUCCESSFUL
pam_end SUCCESSFUL

Changing password from child thread
[New Thread 0x7ffff5b4a700 (LWP 3213)]
pthread_create SUCCESSFUL
pam_start SUCCESSFUL
Detaching after fork from child process 3214.
Detaching after fork from child process 3215.
Detaching after fork from child process 3216.
pam_chauthtok SUCCESSFUL

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5b4a700 (LWP 3213)]
0x00007ffff6a59266 in _dlerror_run () from /lib64/libdl.so.2

Mirek,
do you have the same issue?

Any idea with this?

Hm, I can see the segfault also with libselinux-2.0.94-4.el6 :( Looks like the testing I made in comment #8 was invalid :(

# ./MTPamTest pamtest test

Changing password from main thread
pam_start SUCCESSFUL
pam_chauthtok SUCCESSFUL
pam_end SUCCESSFUL

Changing password from child thread
pthread_create SUCCESSFUL
pam_start SUCCESSFUL
pam_chauthtok SUCCESSFUL
Segmentation fault (core dumped)

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
An update to libselinux caused a segmentation fault to appear in the multi-threaded pam_chauthtok() test program. If a shared library attempted to call pthread_key_create, the associated destructors were registered with that library. The segmentation fault occurred when pthread_key_delete() was called, if that library was dereferenced with dlclose() before the destructors were removed with pthread_key_delete(). This issue has now been corrected. Note that this issue was discovered and corrected during development, and was not seen in production systems in the field.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0751.html