Description of problem: A simple LD_AUDIT library, just la_version, causes a SIGSEGV on upstream rust binaries as of 1.46.0. Version-Release number of selected component (if applicable): glibc-2.31-4.fc32.x86_64 How reproducible: 100% Steps to Reproduce: 1. Download and extract rustc 1.46.0: $ wget https://static.rust-lang.org/dist/2020-08-27/rustc-1.46.0-unknown-linux-gnu.tar.xz $ tar xf rustc-1.46.0-unknown-linux-gnu.tar.xz 2. Build a trivial audit library: $ echo 'unsigned la_version(unsigned version) { return version; }' >audit.c $ gcc -fPIC -shared audit.c -o libaudit.so 3. Try to audit rustc: $ LD_AUDIT=./libaudit.so ./rustc-1.46.0-x86_64-unknown-linux-gnu/rustc/bin/rustc -V Segmentation fault (core dumped) Additional info: GDB shows all "??" in its backtrace, but "coredumpctl info" managed: Stack trace of thread 8652: #0 0x00007ffbdb675fcd get_nprocs (libc.so.6 + 0xfefcd) #1 0x00007ffbdb647291 __sysconf (libc.so.6 + 0xd0291) #2 0x0000559f8209550d n/a (/tmp/rustc-1.46.0-x86_64-unknown-linux-gnu/rustc/bin/rustc + 0x850d) Several past versions of upstream rustc that I sampled were similarly affected. However, 1.47.0-beta.3 and 1.48.0-nightly (7402a3944 2020-09-13) run fine! The Fedora build of rustc 1.46.0 is not affected.
(In reply to Josh Stone from comment #0) > 1. Download and extract rustc 1.46.0: > $ wget > https://static.rust-lang.org/dist/2020-08-27/rustc-1.46.0-unknown-linux-gnu. > tar.xz This URL is 404 for me. Do you have a replacement?
https://static.rust-lang.org/dist/rust-1.46.0-x86_64-unknown-linux-gnu.tar.gz worked and reproduces the issue. This appears to be related to jemalloc.
On Fedora 33 with the system jemalloc, a simple C program just deadlocks: (gdb) bt #0 __lll_lock_wait (futex=0x7ffff76032a8, private=0) at lowlevellock.c:52 #1 0x00007ffff7967763 in __GI___pthread_mutex_lock (mutex=0x7ffff76032a8) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007ffff7ba0475 in je_malloc_mutex_lock_slow () from /lib64/libjemalloc.so.2 #3 0x00007ffff7bbeef8 in extent_recycle.isra () from /lib64/libjemalloc.so.2 #4 0x00007ffff7b6c4d7 in arena_bin_malloc_hard.lto_priv () from /lib64/libjemalloc.so.2 #5 0x00007ffff7bc65ed in je_arena_tcache_fill_small.constprop () from /lib64/libjemalloc.so.2 #6 0x00007ffff7b5d558 in je_malloc_default () from /lib64/libjemalloc.so.2 #7 0x00007ffff79fa8e4 in __GI__IO_file_doallocate ( fp=0x7ffff7b4b520 <_IO_2_1_stdout_>) at filedoalloc.c:101 #8 0x00007ffff7a092a0 in __GI__IO_doallocbuf ( fp=0x7ffff7b4b520 <_IO_2_1_stdout_>) at libioP.h:948 #9 __GI__IO_doallocbuf (fp=0x7ffff7b4b520 <_IO_2_1_stdout_>) at genops.c:342 #10 0x00007ffff7a08438 in _IO_new_file_overflow ( f=0x7ffff7b4b520 <_IO_2_1_stdout_>, ch=-1) at fileops.c:745 #11 0x00007ffff7a074e6 in _IO_new_file_xsputn (n=4, data=<optimized out>, f=<optimized out>) at libioP.h:948 #12 _IO_new_file_xsputn (f=0x7ffff7b4b520 <_IO_2_1_stdout_>, data=<optimized out>, n=4) at fileops.c:1197 #13 0x00007ffff79f2219 in outstring_func (done=0, length=<optimized out>, string=<optimized out>, s=0x7ffff7b4b520 <_IO_2_1_stdout_>) at ../libio/libioP.h:948 #14 __vfprintf_internal (s=0x7ffff7b4b520 <_IO_2_1_stdout_>, format=0x402010 "%ld\n", ap=0x7fffffffdcc0, mode_flags=0) at vfprintf-internal.c:1646 #15 0x00007ffff79de4af in __printf (format=<optimized out>) at printf.c:33 #16 0x0000000000401156 in main () C sources: #include <unistd.h> #include <stdio.h> int main (void) { printf ("%ld\n", sysconf (_SC_PAGESIZE)); } Debugging this is difficult because of the lack of audit namespace support in GDB.
Sorry, forgot to mention that jemalloc is linked with -ljemalloc (no LD_PRELOAD).
(In reply to Florian Weimer from comment #1) > This URL is 404 for me. Do you have a replacement? Sorry about that -- somehow I omitted "x86_64" from the triple in the filename. https://static.rust-lang.org/dist/2020-08-27/rustc-1.46.0-x86_64-unknown-linux-gnu.tar.xz The URL you picked from the dist root is also fine, should be the same thing.
This seems to be a build issue with rust; the only builds that I could get to fail like this were the pre-built 1.46.0 and older. I tried the following and all of them worked just fine: 1. rustc built off the 1.46.0, 1.47.0 and 1.48.0 tags as well as the master branch without jemalloc 2. rustc built off the 1.46.0, 1.47.0 and 1.48.0 tags as well as the master branch with jemalloc 3. rustc 1.47.0 prebuilt Something seems to have changed in the upstream build environment to cause that change.
(In reply to Siddhesh Poyarekar from comment #7) > This seems to be a build issue with rust; the only builds that I could get > to fail like this were the pre-built 1.46.0 and older. Yes, it's related to linking with jemalloc, see comment 3 and comment 4.
The full command with upstream glibc to reproduce the deadlock in comment 3: env LD_AUDIT=./libaudit.so \ GLIBC_TUNABLES=glibc.rtld.optional_static_tls=5120 \ $builddir/elf/ld.so \ --library-path $builddir:$builddir/elf:$builddir/nptl \ ./jemalloc
I spent some time debugging this today and the root cause is that jemalloc, when linked in directly, comes into use before pthreads are initialized. libc.so has symbols for pthread_mutex_lock and pthread_mutex_unlock to take care of that, wherein those operations become nops until the pthreads subsystem is initialized. The twist in the plot is that jemalloc uses *pthread_mutex_trylock*, which does not have a forwarder in libc.so and actually sets the lock primitive. Its paired pthread_mutex_unlock is still a nop because it's still too early, thus setting the stage for the deadlock we see. It's straightforward to fix this by adding a forwarder for pthread_mutex_trylock in libc.so, but on discussion with Florian, we agreed to move all of the mutex functions into libc.so instead, since that's something we want to do anyway.
The original issue had SIGSEGV, not a deadlock -- are you confident that it's the same root cause?
(In reply to Josh Stone from comment #12) > The original issue had SIGSEGV, not a deadlock -- are you confident that > it's the same root cause? Sorry I forgot about the original report :/ I'm pretty sure that it's not the same root cause - your report looks build environment induced - but I'll take a closer look to be 100% sure after which I'll file a separate bug for the deadlock.
Confirmed it is a distinct issue from the deadlock, although the underlying cause is the same - jemalloc getting initialized before libc.so is. This time it is isspace() that causes problems, trying to read into TLS (__libc_tsd_CTYPE_B) when it has not yet been set up. I've taken this bug back on me and have cloned a separate bug for the deadlock that Florian is working on fixing by moving pthread functions to libc.so. Now to figure out a way to make a minimal reproducer out of this because I still can't get a built rust to behave the same way even with jemalloc enabled.
So this is what the jemalloc code says: --- static unsigned malloc_ncpus(void) { long result; #ifdef _WIN32 SYSTEM_INFO si; GetSystemInfo(&si); result = si.dwNumberOfProcessors; #elif defined(JEMALLOC_GLIBC_MALLOC_HOOK) && defined(CPU_COUNT) /* * glibc >= 2.6 has the CPU_COUNT macro. * * glibc's sysconf() uses isspace(). glibc allocates for the first time * *before* setting up the isspace tables. Therefore we need a * different method to get the number of CPUs. */ { cpu_set_t set; pthread_getaffinity_np(pthread_self(), sizeof(set), &set); result = CPU_COUNT(&set); } #else result = sysconf(_SC_NPROCESSORS_ONLN); #endif return ((result == -1) ? 1 : (unsigned)result); } --- so the call to sysconf technically should not get used when building for Linux with glibc. However it's evident from the binary that sysconf is called, so the build environment somehow makes this happen. Josh, would you be able to track down how 1.46.0 got configured and built? Given that 1.47 and later don't have this problem, it seems like whoever did the 1.46.0 build, did so in an environment that either did not have CPU_COUNT or ended up with JEMALLOC_GLIBC_MALLOC_HOOK undefined.
(In reply to Siddhesh Poyarekar from comment #15) > So this is what the jemalloc code says: ... > * glibc >= 2.6 has the CPU_COUNT macro. ... > so the call to sysconf technically should not get used when building for > Linux with glibc. However it's evident from the binary that sysconf is > called, so the build environment somehow makes this happen. Josh, would you > be able to track down how 1.46.0 got configured and built? Given that 1.47 > and later don't have this problem, it seems like whoever did the 1.46.0 > build, did so in an environment that either did not have CPU_COUNT or ended > up with JEMALLOC_GLIBC_MALLOC_HOOK undefined. Aha! Up to 1.46, Rust upstream was using a CentOS 5 environment for its minimum compatibility, which had glibc 2.5. Rust 1.47 updated that baseline to glibc 2.11: https://github.com/rust-lang/rust/pull/74163/
Perfect, that mystery is solved then :) We'd still like to explore options to make the auditor more robust, so I'll keep this open until we file an upstream bug.
This message is a reminder that Fedora 32 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '32'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 32 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
This appears to have been fixed in rawhide too, probably with the same fix as bug 1909920.