I have a problem with pthread_create. The problem is that I get a core dump
when a pthread_create call fails to set a real time policy.

Testcode:

#include <pthread.h>
#include <errno.h>
#include <stdlib.h>
#include <stdio.h>

static void *dummy_thread (void *arg) { printf ("started\n"); return arg; }

static int start (int policy, int priority)
{
  pthread_attr_t attr;
  struct sched_param param;
  pthread_t thread_id;
  int r;

  pthread_attr_init(&attr);
  pthread_attr_setschedpolicy (&attr, policy);
  param.sched_priority = priority;
  pthread_attr_setschedparam (&attr, &param);
  pthread_attr_setinheritsched (&attr, PTHREAD_EXPLICIT_SCHED);
  r = pthread_create(&thread_id, &attr, dummy_thread, NULL);
  pthread_attr_destroy(&attr);
  if (r == 0) { pthread_join(thread_id, NULL); }
  else { errno = r; perror ("pthread_create"); }
  return r;
}

int main(int argc, char **argv)
{
  if (argc > 1) {
    switch (atoi (argv[1])) {
    case 0: start (SCHED_OTHER, 0); break;
    case 1: start (SCHED_OTHER, 10); break;
    case 2: start (SCHED_FIFO, 0); break;
    case 3: start (SCHED_FIFO, 10); break;
    case 4: if (start (SCHED_FIFO, 10) != 0) start (SCHED_OTHER, 0); break;
    }
  }
  return 0;
}

When I run this test with argument set to 3 or 4 I get a core dump.
(I also see a problem with testcase 1. This should fail because SCHED_OTHER
 has no real time priorities but that is not my main problem.)

The stack trace is from case 4 is:
(gdb) where
#0  _dl_map_object_from_fd (name=0x3d0e8109b7 "libgcc_s.so.1", fd=-1, 
    fbp=0x7f9ed2ebd6a8, realname=0x7f9ecc0008c0 "/lib64/libgcc_s.so.1", 
    loader=<optimized out>, l_type=<optimized out>, mode=-1879048191, 
    stack_endp=0x7f9ed2ebda08, nsid=0) at dl-load.c:1566
#1  0x0000003d0dc07e77 in _dl_map_object (loader=0x0, 
    name=0x3d0e8109b7 "libgcc_s.so.1", type=2, trace_mode=0, mode=-1879048191, 
    nsid=<optimized out>) at dl-load.c:2338
#2  0x0000003d0dc11fb8 in dl_open_worker (a=0x7f9ed2ebdc18) at dl-open.c:226
#3  0x0000003d0dc0e146 in _dl_catch_error (objname=0x7f9ed2ebdc60, 
    errstring=0x7f9ed2ebdc68, mallocedp=0x7f9ed2ebdc77, 
    operate=0x3d0dc11ea0 <dl_open_worker>, args=0x7f9ed2ebdc18)
    at dl-error.c:178
#4  0x0000003d0dc1273a in _dl_open (file=0x3d0e8109b7 "libgcc_s.so.1", 
    mode=-2147483647, caller_dlopen=0x0, nsid=-2, argc=2, 
    argv=<optimized out>, env=0x7fffcb31f4c0) at dl-open.c:569
#5  0x0000003d0e11b270 in do_dlopen (ptr=0x7f9ed2ebde18) at dl-libc.c:86
#6  0x0000003d0dc0e146 in _dl_catch_error (objname=0x7f9ed2ebde30, 
    errstring=0x7f9ed2ebde38, mallocedp=0x7f9ed2ebde47, 
    operate=0x3d0e11b230 <do_dlopen>, args=0x7f9ed2ebde18) at dl-error.c:178
#7  0x0000003d0e11b32a in dlerror_run (args=0x7f9ed2ebde18, 
    operate=0x3d0e11b230 <do_dlopen>) at dl-libc.c:47
#8  __GI___libc_dlopen_mode (name=<optimized out>, mode=<optimized out>)
    at dl-libc.c:160
#9  0x0000003d0e80f5dc in pthread_cancel_init ()
    at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:53
#10 0x0000003d0e80f79c in _Unwind_ForcedUnwind (exc=<optimized out>, 
    stop=<optimized out>, stop_argument=<optimized out>)
    at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:126
#11 0x0000003d0e80db00 in __GI___pthread_unwind (buf=<optimized out>)
    at unwind.c:130
#12 0x0000003d0e80dc41 in __pthread_enable_asynccancel ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/cancellation.S:79
#13 0x0000003d0e807be5 in start_thread (arg=0x7f9ed2ebe700)
    at pthread_create.c:291
#14 0x0000003d0e0dfb7d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

So it seems that the cancelation of the the thread fails.

I use method 4 a lot during host testing. This allows me to test things
on host (I do not have root access on this my company host for obvious
reasons). After testing the same code is run on target with root access
or with /etc/security/limits.conf modifications.

The platform information is (fedora 15):

kernel:
Linux htbrug2 2.6.40-4.fc15.x86_64 #1 SMP Fri Jul 29 18:46:53 UTC 2011 x86_64 6_64 x86_64 GNU/Linux

libc:
GNU C Library stable release version 2.14, by Roland McGrath et al.
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.6.0 20110530 (Red Hat 4.6.0-9).
Compiled on a Linux 2.6.38 system on 2011-06-28.
Available extensions:
        Support for some architectures added on, not maintained in glibc core.
        The C stubs add-on version 2.1.2.
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
        RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

When I run the same code on ubuntu it works correctly.
The platform information for ubuntu is:

kernel:
Linux ubuntu 2.6.38-10-generic #46-Ubuntu SMP Tue Jun 28 15:07:17 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

libc:
GNU C Library (Ubuntu EGLIBC 2.13-0ubuntu13) stable release version 2.13, by Roland McGrath et al.
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.5.2.
Compiled on a Linux 2.6.38 system on 2011-04-11.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.