Bug 1591087

This bug will cause VM start failure. Please fix it ASAP.

Thanks, and sorry.  I can reproduce it, and even after fixing strncmp in rtld, so it's another issue.  Will investigate further.

Note for debugging: This reproduces only when building with --enable-kernel=2.6.32, which triggers the definition of __ASSUME_FUTEX_CLOCK_REALTIME, among other things.

git bisect blames this on this backport:

    2738: glibc-rh1401665-1.patch
    
    Includes generic lll_futex_timed_wait_bitset for x86_64, to use
    standard code paths from upstream.
    
    commit 35df5a77f3ad2a35761631928440d2994a9e4bc5
    Author: Carlos O'Donell <carlos>
    Date:   Tue Jun 12 16:17:05 2018 -0400
    
        Fix fallback path in __pthread_mutex_timedlock ().
    
        Fix the typo in the fallback path in __pthread_mutex_timedlock ()
        whic hcalls lll_futex_timed_wait ().  This is only useful for cases
        where the patch is being backported to older distributions where
        only lll_futex_timed_wait () is available.
    
    commit 65810f0ef05e8c9e333f17a44e77808b163ca298
    Author: Torvald Riegel <triegel>
    Date:   Thu Dec 22 10:20:43 2016 +0100
    
        robust mutexes: Fix broken x86 assembly by removing it
    
        lll_robust_unlock on i386 and x86_64 first sets the futex word to
        FUTEX_WAITERS|0 before calling __lll_unlock_wake, which will set the
        futex word to 0.  If the thread is killed between these steps, then the
        futex word will be FUTEX_WAITERS|0, and the kernel (at least current
        upstream) will not set it to FUTEX_OWNER_DIED|FUTEX_WAITERS because 0 is
        not equal to the TID of the crashed thread.
    
        The lll_robust_lock assembly code on i386 and x86_64 is not prepared to
        deal with this case because the fastpath tries to only CAS 0 to TID and
        not FUTEX_WAITERS|0 to TID; the slowpath simply waits until it can CAS 0
        to TID or the futex_word has the FUTEX_OWNER_DIED bit set.
    
        This issue is fixed by removing the custom x86 assembly code and using
        the generic C code instead.  However, instead of adding more duplicate
        code to the custom x86 lowlevellock.h, the code of the lll_robust* functions
        is inlined into the single call sites that exist for each of these functions
        in the pthread_mutex_* functions.  The robust mutex paths in the latter
        have been slightly reorganized to make them simpler.
    
        This patch is meant to be easy to backport, so C11-style atomics are not
        used.

(In reply to Han Han from comment #3)
> This bug will cause VM start failure. Please fix it ASAP.

The glibc -257 build went out with 2 defects in it which were not caught in early testing, the -257 build had been untagged, and the working -254  build should be used again (back on the errata).

Sorry for the problems the build caused.

(In reply to Florian Weimer from comment #6)
> git bisect blames this on this backport:
> 
>     2738: glibc-rh1401665-1.patch

That's what I expected.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1401665#c36

I'm going to drop this patch because it does cause two regressions, already seen in tst-sem5, and tst-sem10 in the glibc testsuite.

lll_futex_timed_wait_bitset and its callers do not agree whether error return codes are negative.  This appears to fix the issue:

diff --git a/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h b/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h
index 5fd4e07..99c2fe9 100644
--- a/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h
+++ b/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h
@@ -233,7 +233,8 @@ LLL_STUB_UNWIND_INFO_END
                              __lll_private_flag (__op, private),             \
                              (val), (timespec), NULL /* Unused.  */,         \
                              FUTEX_BITSET_MATCH_ANY);                        \
-    INTERNAL_SYSCALL_ERROR_P (__ret, __err) ? -__ret : __ret;                \
+    (__glibc_unlikely (INTERNAL_SYSCALL_ERROR_P (__ret, __err))               \
+     ? -INTERNAL_SYSCALL_ERRNO (__ret, __err) : 0);                           \
   })

The broken code happens to work on POWER because INTERNAL_SYSCALL_ERROR_P looks at a different register there, and the kernel returns a *positive* error number, so negating that gives you a negative error code, as expected by the callers.

On x86-64, however, the kernel already gives us a negative error code, and we negate that again, giving us a positive return value for lll_futex_timed_wait_bitset on error.

This is fixed in glibc-2.17-258.

*** This bug has been marked as a duplicate of bug 1401665 ***