Hide Forgot
Fixes of bugs in process-shared robust mutex Previously, bugs in the implementation of POSIX thread robust mutexes could cause such mutexes to fail to be recoverable. Consequently, users of the interface could see a process terminate but not return *EOWNERDEAD* to another process that was trying to recover the mutex. This update fixes the threading library and all known and fixable bugs in robust mutexes.
Created attachment 1228146 [details] Test program Description of problem: Process Shared Robust Mutexes can fail to wake up waiting mutex_lock callers when a process dies under SIGKILL Version-Release number of selected component (if applicable): How reproducible: A few seconds of running a test program in a while : ; do ; done loop shows the problem Steps to Reproduce: 1. Compile the attached program test_mutex_raw.c 2. Run it in a tight while true ; do ./test_mutex_raw; done 3. Wait a few iterations, watch it hang Actual results: At some point the program hangs with a few children sitting in a kernel mutex lock call with nobody holding the lock Expected results: Reliably run the program forever Additional info: https://lists.samba.org/archive/samba-technical/2016-December/117466.html together with thread-folluw-ups has some background. Under FreeBSD 11 with robust mutexes it runs fine. The program was found to hang under RHEL7, RHEL6, (unknown subversions, test was not done by me), Debian Jessie X86_64 as a VMWare guest, and also on a Raspi2 with Debian Testing as of December 1.
An additional comment: I've assigned it to glibc first, because that is our (Samba/tdb)'s point of contact. I know that robust mutexes are closely related to both glibc and kernel, so it might well be that this is in fact a kernel issue.
(In reply to VL from comment #0) > 1. Compile the attached program test_mutex_raw.c > 2. Run it in a tight while true ; do ./test_mutex_raw; done Could you please double-check if you have attached the right reproducer? The C program expects a few command line arguments.
I am soo sorry to waste your time. In the additional info there was the correct line: while true ; do ./bin/test_mutex_raw /tmp/foobar 10 0 ; done Please tell me about your exact paths where you are compiling the program and where your account on your test machine is allowed to create files, such that I can get you better information on how to reproduce the issue.
I can reproduce this. The test case looks like it should not hang.
This may be related to statements about design issues in robust mutexes: https://sourceware.org/ml/libc-alpha/2015-04/msg00151.html I haven't had time to review these statements in detail yet, so I can't yet comment in detail.
It seems to me that this glibc bug is related if not actually the same phenomenon: https://sourceware.org/bugzilla/show_bug.cgi?id=19402 @Torvald, it seems that the class of problems that the post you cited mentions is more about data corruption than about hanging, but it may have related causes. (e.g. https://sourceware.org/bugzilla/show_bug.cgi?id=14485) Is there any libc expert who can look more deeply into these issues? We do have two reproducers, and this is really critical for us (Samba). Thanks!
(In reply to Michael Adam from comment #7) > Is there any libc expert who can look more deeply into these issues? > We do have two reproducers, and this is really critical for us (Samba). I'm currently reviewing the synchronization in robust mutexes (and I'm glibc's concurrency maintainer). I believe I have found the bug that the test case attached to this BZ triggers. This one can be fixed rather easily. However, I've found other bugs in the synchronization that need to be fixed too. One is a problem in the kernel, there's another potential problem in glibc (although that may happen to just work right now), and the biggest one is that the mutex destruction requirements are not guaranteed currently (which will need both glibc and kernel changes to be fixed). So, stay tuned :)
is this one related? https://bugzilla.redhat.com/show_bug.cgi?id=1302389
I created an upstream bug for this bug here: https://sourceware.org/bugzilla/show_bug.cgi?id=20973 (In reply to Michael Adam from comment #9) > is this one related? > > https://bugzilla.redhat.com/show_bug.cgi?id=1302389 No, this seems to be a separate issue. I'm looking into that one as well.
We understand this issue is critical for Samba and we are continuing to look at the robust shared mutex support code in glibc and the linux kernel. When we have a concrete result we'll update this issue.
swbz 19402 is a separate bug, at least regarding the reproducer posted in this bug here. The reproducer from this bug worked flawlessly for me for an hour after applying this patch to upstream glibc: https://sourceware.org/ml/libc-alpha/2016-12/msg00587.html
The test case has its own race condition: while (sig_child_count < num_children) { if (sig_received) { int status; while (1) { pid = waitpid(-1, &status, WNOHANG); if (pid <= 0) { break; } sig_child_count += 1; } sig_received = false; } usleep(100*1000); } The assignment to sig_received could hide the arrival of a new SIGCHLD signal and the update of the same variable in the signal handler.
(In reply to Florian Weimer from comment #14) > The assignment to sig_received could hide the arrival of a new SIGCHLD > signal and the update of the same variable in the signal handler. This is true (and I've observed this to happen while testing). I've been testing with a loop like this one: while (sig_child_count < num_children) { int status; pid = wait (&status); if (pid < 0) { if (errno == EINTR) continue; fprintf (stderr, "waitpid failed\n"); exit (1); } sig_child_count++; }
Created attachment 1232820 [details] Test Code v2
(In reply to Torvald Riegel from comment #15) > (In reply to Florian Weimer from comment #14) > > The assignment to sig_received could hide the arrival of a new SIGCHLD > > signal and the update of the same variable in the signal handler. > > This is true (and I've observed this to happen while testing). I've been > testing with a loop like this one: I have attached slightly modified version of the test code which does not use signal handler. This one should be free from the race condition.
We have a build of a test fix for this issue on top of what will be released as the final rhel-6.9 glibc. For now this fix is not for use in production environments and should only be used to validate in RHEL 6 that the issue is fixed. You can download the new RPMs by adding a new repo here: cat >> /etc/yum.repos.d/rhbz1401665.repo <<EOF [rhbz1401665] name=RHEL 6.9 hotfix for bug 1401665 baseurl=http://people.redhat.com/codonell/rhel-6.9-rhbz1401665 enabled=1 gpgcheck=0 protect=1 EOF Then you can upgrade glibc e.g. yum upgrade glibc, to install the new packages. The fixed packages include Torvald's upstream fix from comment #13 and have gone through some minimal engineering sanity testing along with validation that the fixes allow the reproducer in comment #16 to run without issue.
We have a build of a test fix for this issue on top rhel-7.3 glibc. For now this fix is not for use in production environments and should only be used to validate in RHEL 7 that the issue is fixed. You can download the new RPMs by adding a new repo here: cat >> /etc/yum.repos.d/rhbz1401665.repo <<EOF [rhbz1401665] name=RHEL 7.3 hotfix for bug 1401665 baseurl=http://people.redhat.com/codonell/rhel-7.3-rhbz1401665 enabled=1 gpgcheck=0 protect=1 EOF Then you can upgrade glibc e.g. yum upgrade glibc, to install the new packages. The fixed packages include Torvald's upstream fix from comment #13 and have gone through some minimal engineering sanity testing along with validation that the fixes allow the reproducer in comment #16 to run without issue.
I have verified with RHEL 6.9 hotfix RPMs that the bug is fixed. What versions of RHEL6 and RHEL7 will have this fix backported?
(In reply to Amitay Isaacs from comment #22) > I have verified with RHEL 6.9 hotfix RPMs that the bug is fixed. > > What versions of RHEL6 and RHEL7 will have this fix backported? We will review this bug for inclusion in RHEL 7.5 and RHEL 6.10.
This issue is included in our Red Hat Enterprise Linux 7.6 review, and the robust mutex fixes will be considered for backporting here after we evaluate the risk and depth of the backport.
The following upstream bug fixes were backported: Bug 20973 - robust mutexes: Lost wake-ups https://sourceware.org/bugzilla/show_bug.cgi?id=20973 - Commit 353683a22ed8a493a6bd1d78d63e144bc3e85d2f Bug 20985 - robust mutexes: lowlevelrobustlock assembly on x86 blocks on wrong condition https://sourceware.org/bugzilla/show_bug.cgi?id=20985 - Commit 65810f0ef05e8c9e333f17a44e77808b163ca298, and 35df5a77f3ad2a35761631928440d2994a9e4bc5 (typo fix). - Yes, RHEL 7.6 is using the lll_futex_timed_wait fallback, which should be fine. Bug 19402 - Deadlock with robust shared mutex and asynchronous termination https://sourceware.org/bugzilla/show_bug.cgi?id=19402 - Commit 8e31cafb268938729a1314806a924d73fb1991c5 "Add compiler barriers around modifications of the robust mutex list." - Commit 8f9450a0b7a9e78267e8ae1ab1000ebca08e473e Bug 21778 - mutex: Fix robust mutex lock acquire https://sourceware.org/bugzilla/show_bug.cgi?id=21778 - Commit 5920a4a624b1f4db310d1c44997b640e2a4653e5 - Fixes bug in the latest fix. For testing we add: commit faf0e9c84119742dd9ebb79060faa22c52ae80a1 Author: Florian Weimer <fweimer@redhat.com> Date: Fri Jan 27 06:53:19 2017 +0100 nptl: Add tst-robust-fork Which tests the fork issues. Otherwise we rely on the existing tests to verify robust mutext support. The 'test code v2' attached to this issue will still hang, but always with a thread as owner of the robust mutex.
Created attachment 1451011 [details] Test Code v3 I rewrote the test code to use a single shared memory segment, and to avoid some of the undefined behaviour, like initializing the same mutex twice which happens when the low and high priority processes start (both do the initialization and that's dangerous.) Instead now we start an "init" process to start the shared memory segment, and then everything else just uses it without truncation or unlinking (same mutex).
I am going to drop these two fixes for rhel-7.6 since they have caused some regression in the semaphore uses which rely on the existing assembly futex operations. Bug 20985 - robust mutexes: lowlevelrobustlock assembly on x86 blocks on wrong condition https://sourceware.org/bugzilla/show_bug.cgi?id=20985 - Commit 65810f0ef05e8c9e333f17a44e77808b163ca298, and 35df5a77f3ad2a35761631928440d2994a9e4bc5 (typo fix). - Yes, RHEL 7.6 is using the lll_futex_timed_wait fallback, which should be fine. "Add compiler barriers around modifications of the robust mutex list." - Commit 8f9450a0b7a9e78267e8ae1ab1000ebca08e473e We will file another issue to go back to the fix for bug 20985 in rhel-7.7. At the very least in rhel-7.6 we will fix the lost wakeup and the fork synchronization issues.
(In reply to Carlos O'Donell from comment #36) > I am going to drop these two fixes for rhel-7.6 since they have caused some > regression in the semaphore uses which rely on the existing assembly futex > operations. > > Bug 20985 - robust mutexes: lowlevelrobustlock assembly on x86 blocks on > wrong condition > https://sourceware.org/bugzilla/show_bug.cgi?id=20985 > - Commit 65810f0ef05e8c9e333f17a44e77808b163ca298, and > 35df5a77f3ad2a35761631928440d2994a9e4bc5 (typo fix). > - Yes, RHEL 7.6 is using the lll_futex_timed_wait fallback, which should be > fine. > > "Add compiler barriers around modifications of the robust mutex list." > - Commit 8f9450a0b7a9e78267e8ae1ab1000ebca08e473e > > We will file another issue to go back to the fix for bug 20985 in rhel-7.7. > > At the very least in rhel-7.6 we will fix the lost wakeup and the fork > synchronization issues. Thanks to Florian Weimer we found the source of the bug that was blocking these issues. I've verified with v3 reproducer that it now passes and runs for hours without any failure on a >8 core boxes. Looks like we have stable robust mutexes in RHEL 7.6 for the set of problems we set out to solve this release.
All fixed in -258 now.
*** Bug 1591087 has been marked as a duplicate of this bug. ***
Created attachment 1479264 [details] Robust mutex test (v4)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3092