The attached test case blocks forever on kernel-2.6.31-0.139.rc5.git3.fc12.x86_64/glibc-2.10.90-12.x86_64. It works fine on Program works as intended on kernel-2.6.31-0.38.rc1.git7.fc12.i586/glibc-2.10.1-2.i686.
Disabling PTHREAD_PRIO_INHERIT for the mutex in question makes the test case work properly.
Not sure if this is a kernel or glibc problem.
Created attachment 356836 [details]
peterz thinks it's a kernel problem. Reassigning.
No, it isn't.
Took a look at the trace, the kernel is doing what it should. If you put the thread to sleep with FUTEX_WAIT_REQUEUE_PI_PRIVATE (0x8b) you have to wake it up with FUTEX_CMP_REQUEUE_PI_PRIVATE (0x8c), but the trace shows a call to FUTEX_WAKE_PRIVATE instead. This causes the kernel routine the thread is blocked in to wake, detect invalid usage, and return EINVAL. This is a glibc bug.
Fixed in 2.10.90-14.