Bug 43742

I'm setting this to "blocker" since this should be a BIG issue at Red Hat.
As 
you can see from the report the whole server can come unglued. But only in
this 
context which I (the reporter) does not know if others code like this. If
this 
is a rare case then "critical" would better describe the Severity
 
PROBLEM SUMMARY
===============

When a program has created some threads, and those threads create more
threads, the pthreads library appears to become confused to the point that
threads are created with the wrong arguments.

ORIGINAL SYMPTOMS OF PROBLEM
============================

I first saw this problem in a program where the thread argument was a
pointer
to allocated memory.  Each thread created was to be provided with the
address
of allocated memory that was for its exclusive use.  Once the thread was
done
with the allocated memory, it would free the memory.  But I saw that every
now
and then multiple threads were being given the same address, causing the
same
memory to be freed twice.  In order to convince myself that the problem was
not with how my program was managing its memory or serializing itself, I
devised a test program to illustrate the problem.

TEST PROGRAM TO ILLUSTRATE THE PROBLEM
======================================

The threads of the test program fall into 3 categories.  First, there's the
initial or primary thread.  Next, there are the secondary threads.  All the
secondary threads are created by the primary thread, and then the primary
thread waits for the secondary threads to end.  The number of secondary
threads created is specified by the SEC_THREADS constant (see the source
code
below).  The secondary threads create the tertiary threads.  The number of
tertiary threads created by a secondary thread at once is specified by the
SEC_THREADS_PER_REP constant.  Each secondary thread creates tertiary
threads,
and then waits for those tertiary threads to end.  This is done in a loop.
The number of times the loop is executed is specified by the
SEC_THREADS_REP
constant.  The total number of tertiary threads created during the life of
the
program is specified by the constant TER_THREADS.

As each secondary thread is created, the thread id returned by
pthread_create() is stored in an element of the static array cr_sec_tids. 
As
each tertiary thread is created, the thread id returned by pthread_create()
is
stored in an element of the static array cr_ter_tids.  When a secondary
thread
starts executing its routine, sec_thread_rtn(), it will record its thread
ID,
as determined by pthread_self(), in the element of the ex_sec_tids array
specified by the thread's argument.  But, first, it will make sure no other
thread has put a thread ID in the element.  When a tertiary thread starts
executing its routine, ter_thread_rtn(), it will record its thread ID, as
determined by pthread_self(), in the element of the ex_ter_tids array
specified by the thread's argument.  First, it will make sure no other
thread
has put a thread ID in the element.  When everything is working properly,
no
thread should ever detect that another thread has recorded a thread ID in
the
element specified for it.  The ex_sec_tids array has one element for every
secondary thread to be created during the run of the program, and the
ex_ter_tids array has one element for every tertiary thread to be created
during the run of the program.

However, on a SMP system with 2 CPUs, and the level of glibc indicated
below,
tertiary threads detect that some other thread has been given the address
of
the same array element.

Here's some information about the system.  The system has Redhat 7.1
installed.

# rpm -qa | grep glibc
glibc-devel-2.2.2-10
compat-glibc-6.2-2.1.3.2
glibc-common-2.2.2-10
glibc-2.2.2-10
glibc-profile-2.2.2-10

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 5
model name      : Pentium II (Deschutes)
stepping        : 0
cpu MHz         : 333.059
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov 
pat pse36 mmx fxsr
bogomips        : 665.19

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 5
model name      : Pentium II (Deschutes)
stepping        : 0
cpu MHz         : 333.059
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov 
pat pse36 mmx fxsr
bogomips        : 665.19


Here's the source code of the test program, threadarg.c:


======================= source code for threadarg.c
===========================

#include <pthread.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>


#define SEC_THREADS             10
#define SEC_THREADS_REP         20000
#define TER_THREADS_PER_REP     2

#define TER_THREADS         SEC_THREADS * SEC_THREADS_REP *
TER_THREADS_PER_REP

static pthread_t cr_sec_tids[SEC_THREADS];
static pthread_t ex_sec_tids[SEC_THREADS];
static pthread_t cr_ter_tids[TER_THREADS];
static pthread_t ex_ter_tids[TER_THREADS];


void *
ter_thread_rtn(void *arg)
{
    pthread_t       me;
    pthread_t      *ex_ter_tid_p;
    int             ter_tid_ndx;


    me = pthread_self();
    ex_ter_tid_p = (pthread_t *)arg;
    ter_tid_ndx = ex_ter_tid_p - ex_ter_tids;

    if (ter_tid_ndx < 0 || ter_tid_ndx >= TER_THREADS) {
        fprintf(stderr, "%u: Got invalid tertiary thread index: %d\n",
                me, ter_tid_ndx);
        return NULL;
    }

    if (*ex_ter_tid_p != 0) {
        fprintf(stderr, "%u: ex_ter_tid[%u] already set to %u!\n",
                me, ter_tid_ndx, *ex_ter_tid_p);
        return NULL;
    }

    *ex_ter_tid_p = me;

    return NULL;
}


void *
sec_thread_rtn(void *arg)
{
    pthread_t       me;
    int             i, r, rc;
    pthread_t      *ex_sec_tid_p;
    int             sec_tid_ndx, ter_tid_ndx0, ter_tid_ndx;


    me = pthread_self();
    ex_sec_tid_p = (pthread_t *)arg;
    sec_tid_ndx = ex_sec_tid_p - ex_sec_tids;

    if (sec_tid_ndx < 0 || sec_tid_ndx >= SEC_THREADS) {
        fprintf(stderr, "%u: Got invalid secondary thread index: %d\n",
                me, sec_tid_ndx);
        return NULL;
    }

    if (*ex_sec_tid_p != 0) {
        fprintf(stderr, "%u: ex_sec_tids[%u] already set to %u!\n",
                me, sec_tid_ndx, *ex_sec_tid_p);
        return NULL;
    }

    *ex_sec_tid_p = me;

    ter_tid_ndx0 = (sec_tid_ndx * SEC_THREADS_REP * TER_THREADS_PER_REP);

    for (r = 0; r < SEC_THREADS_REP; r++) {

        ter_tid_ndx = ter_tid_ndx0 + (r * TER_THREADS_PER_REP);

        for (i = 0; i < TER_THREADS_PER_REP; i++, ter_tid_ndx++) {

            rc = pthread_create(&cr_ter_tids[ter_tid_ndx], NULL,
                                ter_thread_rtn, &ex_ter_tids[ter_tid_ndx]);
            if (rc != 0) {
                fprintf(stderr, "Error creating tertiary thread with "
                        "index %d: (%d) %s\n", ter_tid_ndx, rc,
strerror(rc));
                exit(1);
            }
        }

        ter_tid_ndx = ter_tid_ndx0 + (r * TER_THREADS_PER_REP);

        for (i = 0; i < TER_THREADS_PER_REP; i++, ter_tid_ndx++) {
            rc = pthread_join(cr_ter_tids[ter_tid_ndx], NULL);

            if (rc != 0) {
                fprintf(stderr, "Error joining tertiary thread with "
                        "index %d, thread ID %u: (%d) %s\n",
                        ter_tid_ndx, cr_ter_tids[ter_tid_ndx],
                        rc, strerror(rc));
                /* keep going */
            }
        }

    }

    return NULL;
}


int
main(int argc, char **argv)
{
    int i, rc;


    memset(cr_sec_tids, '\0', sizeof cr_sec_tids);
    memset(ex_sec_tids, '\0', sizeof ex_sec_tids);
    memset(cr_ter_tids, '\0', sizeof cr_ter_tids);
    memset(ex_ter_tids, '\0', sizeof ex_ter_tids);

    for (i = 0; i < SEC_THREADS; i++) {
        rc = pthread_create(&cr_sec_tids[i], NULL,
                            sec_thread_rtn, &ex_sec_tids[i]);
        if (rc != 0) {
            fprintf(stderr,
                    "Error creating secondary thread with index %d: (%d)
%s\n",
                    i, rc, strerror(rc));
            exit(1);
        }
    }

    for (i = 0; i < SEC_THREADS; i++) {
        rc = pthread_join(cr_sec_tids[i], NULL);

        if (rc != 0) {
            fprintf(stderr, "Error joining secondary thread with "
                    "index %d, thread ID %u: (%d) %s\n",
                    i, cr_sec_tids[i], rc, strerror(rc));
            /* keep going */
        }
}

    for (i = 0; i < SEC_THREADS; i++) {
        if (ex_sec_tids[i] != cr_sec_tids[i]) {
            fprintf(stderr,
                    "For index %d, secondary thread id mismatch: %u %u.\n",
                    i, cr_sec_tids[i], ex_sec_tids[i]);
            /* keep going */
        }
    }

    for (i = 0; i < TER_THREADS; i++) {
        if (ex_ter_tids[i] != cr_ter_tids[i]) {
            fprintf(stderr,
                    "For index %d, tertiary thread id mismatch: %u %u.\n",
                    i, cr_ter_tids[i], ex_ter_tids[i]);
            /* keep going */
        }
    }

    return 0;
}

======================= source code for threadarg.c
===========================


RUNNING THE TEST PROGRAM
========================

Here's how I compiled threadarg.c:

#  cc threadarg.c -o threadarg -lpthread -lc


Here's the output of a run of the test program:

2782236: ex_ter_tid[120271] already set to 2781211!
5587993: ex_ter_tid[120769] already set to 5586965!
9749534: ex_ter_tid[40861] already set to 9748509!
11813914: ex_ter_tid[41171] already set to 11812886!
13165601: ex_ter_tid[122531] already set to 13164576!
Error joining tertiary thread with index 122630, thread ID 13540382: (22) 
Invalid argument
Error joining tertiary thread with index 122680, thread ID 13733922: (22) 
Invalid argument
14037027: ex_ter_tid[122752] already set to 14036002!
Error joining tertiary thread with index 122817, thread ID 14271535: (22) 
Invalid argument
Error joining tertiary thread with index 122819, thread ID 14275636: (22) 
Invalid argument
Error joining tertiary thread with index 122863, thread ID 14445620: (22) 
Invalid argument
Error joining tertiary thread with index 41985, thread ID 16016508: (22)
Invalid 
argument
Error joining tertiary thread with index 123310, thread ID 16020616: (22) 
Invalid argument
Error joining tertiary thread with index 123311, thread ID 16020616: (22) 
Invalid argument
Error joining tertiary thread with index 123983, thread ID 18274487: (22) 
Invalid argument
Error joining tertiary thread with index 42411, thread ID 18327752: (22)
Invalid 
argument
Error creating tertiary thread with index 201571: (11) Resource temporarily 
unavailable


Some of the tertiary threads detect the element of the ex_ter_tids array
they've been given already has some thread ID recorded in the element. 
This
should never happen, since no two pthread_create() invocations specify the
same element.  This indicates to me the pthreads library is confused and
creating some threads with identical arguments by mistake.  Once this has
happened there is another problem: some of the calls to pthread_join() fail
--
perhaps because some data structure is corrupted in the pthreads library.
(During other runs of the test program the pthread_join errno value was 3,
instead of 22).  Then eventually new threads cannot be created because of
the
unavailability of resources.  That should not happen in this program.  The
program certainly creates a lot of threads -- over 400,000 if the program
runs
to completion, but only about 30 threads can be active at any one time.


ENVIRONMENTS WHERE THE TEST PROGRAM SHOWED NO PROBLEMS
======================================================

I ran the test program on a uniprocessor system running Redhat 6.2.  The
test
program ran successfully to completion.  I don't know if this was because
the
system only has one processor or because it has a different version of the
pthreads library installed.  Here's information about that system.

# rpm -q -a | grep libc
glibc-2.1.3-15
compat-glibc-5.2-2.0.7.2
glibc-devel-2.1.3-15
glibc-profile-2.1.3-15
libc-5.3.12-31

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 5
model name      : Pentium II (Deschutes)
stepping        : 1
cpu MHz         : 299.613059
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
sep_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov
pat 
pse36 mmx fxsr
bogomips        : 299.01


I also ran the test program on a non-Intel SMP running another flavor of
Unix.  This system has a different implementation of pthreads.  The test
program
ran successfully on that system.

http://www.redhat.com/bugzilla/show_bug.cgi?id=43741

-