This test is fragile at the best of times, and now I've seen it fail on s390x. 31 /* There is an obscure bug in the kernel due to which RLIMIT_STACK is sometimes 32 returned as unlimited when it is not, which may cause this test to fail. 33 There is also the other case where RLIMIT_STACK is intentionally set as 34 unlimited or very high, which may result in a vma that is too large and again 35 results in a test case failure. To avoid these problems, we cap the stack 36 size to one less than 8M. See the following mailing list threads for more 37 information about this problem: 38 <https://sourceware.org/ml/libc-alpha/2012-06/msg00599.html> 39 <https://sourceware.org/ml/libc-alpha/2012-06/msg00713.html>. */ 40 #define MAX_STACK_SIZE (8192 * 1024 - 1) =====FAIL: nptl/tst-pthread-getattr.out===== Verifying that stack top is accessible current rlimit_stack is 8388608 Adjusting RLIMIT_STACK to 6733825 Adjusted rlimit: stacksize=6729728, stackaddr=0x3ffc1314000 Didn't expect signal from child: got `Segmentation fault' + uname -a Linux buildvm-s390x-08.s390.fedoraproject.org 4.18.7-200.fc28.s390x #1 SMP Mon Sep 10 15:21:35 UTC 2018 s390x s390x s390x GNU/Linux + LD_SHOW_AUXV=1 + /bin/true AT_SYSINFO_EHDR: 0x3ffbc3fe000 AT_HWCAP: esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx AT_PAGESZ: 4096 AT_CLKTCK: 100 AT_PHDR: 0x2aa0a000040 AT_PHENT: 56 AT_PHNUM: 9 AT_BASE: 0x3ffbc380000 AT_FLAGS: 0x0 AT_ENTRY: 0x2aa0a0018f8 AT_UID: 1000 AT_EUID: 1000 AT_GID: 425 AT_EGID: 425 AT_SECURE: 0 AT_RANDOM: 0x3fff177fb0c AT_EXECFN: /bin/true AT_PLATFORM: z13 + cat /proc/cpuinfo vendor_id : IBM/S390 # processors : 2 bogomips per cpu: 3033.00 max thread id : 0 features : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx sie facilities : 0 1 2 3 4 6 7 8 9 10 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 30 31 32 33 34 35 36 37 40 41 42 43 44 45 46 47 48 49 50 51 52 53 55 57 69 70 71 72 73 74 75 76 77 80 81 82 128 129 cache0 : level=1 type=Data scope=Private size=128K line_size=256 associativity=8 cache1 : level=1 type=Instruction scope=Private size=96K line_size=256 associativity=6 cache2 : level=2 type=Data scope=Private size=2048K line_size=256 associativity=8 cache3 : level=2 type=Instruction scope=Private size=2048K line_size=256 associativity=8 cache4 : level=3 type=Unified scope=Shared size=65536K line_size=256 associativity=16 cache5 : level=4 type=Unified scope=Shared size=491520K line_size=256 associativity=30 processor 0: version = FF, identification = 3533E8, machine = 2964 processor 1: version = FF, identification = 3533E8, machine = 2964 cpu number : 0 cpu MHz dynamic : 5000 cpu MHz static : 5000 cpu number : 1 cpu MHz dynamic : 5000 cpu MHz static : 5000 + cat /proc/sysinfo Manufacturer: IBM Type: 2964 Model: 400 N63 Sequence Code: 00000000000033E8 Plant: 02 Model Capacity: 400 00000000 Capacity Adj. Ind.: 100 Capacity Ch. Reason: 0 Capacity Transient: 0 Type 1 Percentage: 0 Type 2 Percentage: 0 Type 3 Percentage: 0 Type 4 Percentage: 0 Type 5 Percentage: 0 CPUs Total: 63 CPUs Configured: 0 CPUs Standby: 0 CPUs Reserved: 63 CPUs G-MTID: 0 CPUs S-MTID: 1 Capability: 3296 Nominal Capability: 3296 Secondary Capability: 492 ... LPAR Number: 53 LPAR Characteristics: Shared LPAR Name: L35 LPAR Adjustment: 253 LPAR CPUs Total: 32 LPAR CPUs Configured: 16 LPAR CPUs Standby: 16 LPAR CPUs Reserved: 0 LPAR CPUs Dedicated: 0 LPAR CPUs Shared: 16 LPAR CPUs G-MTID: 0 LPAR CPUs S-MTID: 1 LPAR CPUs PS-MTID: 0 VM00 Name: FBLDR8 VM00 Control Program: z/VM 6.4.0 VM00 Adjustment: 125 VM00 CPUs Total: 2 VM00 CPUs Configured: 2 VM00 CPUs Standby: 0 VM00 CPUs Reserved: 0 + cat /proc/meminfo MemTotal: 8238368 kB MemFree: 2976264 kB MemAvailable: 7054664 kB Buffers: 1616 kB Cached: 3845612 kB SwapCached: 18332 kB Active: 1343812 kB Inactive: 2720528 kB Active(anon): 117492 kB Inactive(anon): 100316 kB Active(file): 1226320 kB Inactive(file): 2620212 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 6929220 kB SwapFree: 6744896 kB Dirty: 574076 kB Writeback: 0 kB AnonPages: 215484 kB Mapped: 66668 kB Shmem: 696 kB Slab: 957316 kB SReclaimable: 321868 kB SUnreclaim: 635448 kB KernelStack: 3248 kB PageTables: 8948 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 11048404 kB Committed_AS: 924648 kB VmallocTotal: 130023424 kB VmallocUsed: 0 kB VmallocChunk: 0 kB CmaTotal: 4096 kB CmaFree: 4056 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1024 kB Hugetlb: 0 kB DirectMap4k: 4096 kB DirectMap1M: 8384512 kB DirectMap2G: 0 kB + df Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 4119184 0 4119184 0% /dev/shm tmpfs 4119184 4 4119180 1% /proc/filesystems
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle. Changing version to '31'.
Reproduces on RHEL 7.7 (kernel 3.10); just keep looping until failure and the failing case looks like this (map dump added just before line that coredumps). Note that the failing case is the only one that lists [heap] before [stack]; all other cases put the heap after the stack, so yeah, it sounds like the same kind of kernel ASLR bug we've seen before. Does not reproduce on F32 (kernel 5.3) where the [heap] is in a completely different segment of the memory map. Verifying that stack top is accessible current rlimit_stack is 8388608 Adjusting RLIMIT_STACK to 7684097 Adjusted rlimit: stacksize=7680000, stackaddr=0x3ffff406000 01000000-01004000 r-xp 00000000 00:28 109078265 /home/nfs/dj/glibc.build/nptl/tst-pthread-getattr 01004000-01005000 r--p 00003000 00:28 109078265 /home/nfs/dj/glibc.build/nptl/tst-pthread-getattr 01005000-01006000 rw-p 00004000 00:28 109078265 /home/nfs/dj/glibc.build/nptl/tst-pthread-getattr 3fffd533000-3fffd535000 rw-p 00000000 00:00 0 3fffd535000-3fffd6c6000 r-xp 00000000 00:28 109067969 /home/nfs/dj/glibc.build/libc.so 3fffd6c6000-3fffd6c7000 ---p 00191000 00:28 109067969 /home/nfs/dj/glibc.build/libc.so 3fffd6c7000-3fffd6ca000 r--p 00191000 00:28 109067969 /home/nfs/dj/glibc.build/libc.so 3fffd6ca000-3fffd6cd000 rw-p 00194000 00:28 109067969 /home/nfs/dj/glibc.build/libc.so 3fffd6cd000-3fffd6d1000 rw-p 00000000 00:00 0 3fffd6d1000-3fffd6ee000 r-xp 00000000 00:28 109071924 /home/nfs/dj/glibc.build/nptl/libpthread.so 3fffd6ee000-3fffd6ef000 r--p 0001c000 00:28 109071924 /home/nfs/dj/glibc.build/nptl/libpthread.so 3fffd6ef000-3fffd6f0000 rw-p 0001d000 00:28 109071924 /home/nfs/dj/glibc.build/nptl/libpthread.so 3fffd6f0000-3fffd6f6000 rw-p 00000000 00:00 0 3fffd6f6000-3fffd71a000 r-xp 00000000 00:28 109067968 /home/nfs/dj/glibc.build/elf/ld.so 3fffd71b000-3fffd71c000 r--p 00024000 00:28 109067968 /home/nfs/dj/glibc.build/elf/ld.so 3fffd71c000-3fffd71d000 rw-p 00025000 00:28 109067968 /home/nfs/dj/glibc.build/elf/ld.so 3fffd71d000-3fffd71e000 rw-p 00000000 00:00 0 3ffff3e3000-3ffff404000 rw-p 00000000 00:00 0 [heap] 3ffffb39000-3ffffb5a000 rw-p 00000000 00:00 0 [stack] Segmentation fault (core dumped) $ $ while env GCONV_PATH=/home/nfs/dj/glibc.build/iconvdata \ LOCPATH=/home/nfs/dj/glibc.build/localedata LC_ALL=C \ /home/nfs/dj/glibc.build/elf/ld64.so.1 --library-path \ /home/nfs/dj/glibc.build:/home/nfs/dj/glibc.build/math:/home/nfs/dj/glibc.build/elf:/home/nfs/dj/glibc.build/dlfcn:/home/nfs/dj/glibc.build/nss:/home/nfs/dj/glibc.build/nis:/home/nfs/dj/glibc.build/rt:/home/nfs/dj/glibc.build/resolv:/home/nfs/dj/glibc.build/mathvec:/home/nfs/dj/glibc.build/support:/home/nfs/dj/glibc.build/crypt:/home/nfs/dj/glibc.build/nptl /home/nfs/dj/glibc.build/nptl/tst-pthread-getattr --direct ; do echo;echo;echo; done
(In reply to DJ Delorie from comment #3) > Reproduces on RHEL 7.7 (kernel 3.10); just keep looping until failure > and the failing case looks like this (map dump added just before line > that coredumps). Note that the failing case is the only one that lists > [heap] before [stack]; all other cases put the heap after the stack, so > yeah, it sounds like the same kind of kernel ASLR bug we've seen before. > Does not > reproduce on F32 (kernel 5.3) where the [heap] is in a completely > different segment of the memory map. Thank you for digging into this. Yes, it looks like our suspicions were correct. Can you please file a kernel bug for this then? It should be a clean bug with a clean reproducer for the kernel team. Then we can close this as a duplicate of the clean kernel bug (links the two bugs). So if we run into this again we should be able to find it given the description here.
I'm not 100% sure the root cause is the kernel. With more research I've come up with a standalone version of the test with additional debug info: $ while /lib64/ld-2.17.so ./1653942.x ; do true; done You eventually get this: current rlimit_stack is 8388608 Adjusting RLIMIT_STACK to 7806977 Adjusted rlimit: stacksize=7802880, stackaddr=0x3ffff5d4000 80000000-80002000 r-xp 00000000 00:28 107611085 /home/nfs/dj/1653942.x 80002000-80003000 r--p 00001000 00:28 107611085 /home/nfs/dj/1653942.x 80003000-80004000 rw-p 00002000 00:28 107611085 /home/nfs/dj/1653942.x 3fffcf50000-3fffcf52000 rw-p 00000000 00:00 0 3fffcf52000-3fffd101000 r-xp 00000000 fd:01 67235395 /usr/lib64/libc-2.17.so 3fffd101000-3fffd105000 r--p 001ae000 fd:01 67235395 /usr/lib64/libc-2.17.so 3fffd105000-3fffd107000 rw-p 001b2000 fd:01 67235395 /usr/lib64/libc-2.17.so 3fffd107000-3fffd10b000 rw-p 00000000 00:00 0 3fffd10b000-3fffd123000 r-xp 00000000 fd:01 67289710 /usr/lib64/libpthread-2.17.so 3fffd123000-3fffd124000 r--p 00017000 fd:01 67289710 /usr/lib64/libpthread-2.17.so 3fffd124000-3fffd125000 rw-p 00018000 fd:01 67289710 /usr/lib64/libpthread-2.17.so 3fffd125000-3fffd129000 rw-p 00000000 00:00 0 3fffd142000-3fffd145000 rw-p 00000000 00:00 0 3fffd145000-3fffd168000 r-xp 00000000 fd:01 67494691 /usr/lib64/ld-2.17.so 3fffd168000-3fffd169000 rw-p 00000000 00:00 0 3fffd169000-3fffd16a000 r--p 00023000 fd:01 67494691 /usr/lib64/ld-2.17.so 3fffd16a000-3fffd16b000 rw-p 00024000 fd:01 67494691 /usr/lib64/ld-2.17.so 3fffd16b000-3fffd16c000 rw-p 00000000 00:00 0 3ffff5b1000-3ffff5d2000 rw-p 00000000 00:00 0 [heap] 3ffffd25000-3ffffd46000 rw-p 00000000 00:00 0 [stack] sbrk() = 0x3ffff5d2000, testing 0x3ffff5d4800 getpagesize = 0x1000 Segmentation fault Note that the sbrk() corresponds to the top of [heap], and that the testing addr is 1.5 pages away from it. Florian notes that on older kernels, single threaded apps have a larger stack guard - perhaps 1 Mb instead of 4k. Running the test *without* ld-2.17.so (i.e. directly) always puts the heap in the 0x80000000 range with the rest of the program, so this conflict never happens for normal programs. Conclusions so far: 1. The failure is dependent on our test harness (using ld.so directly) 2. The testsuite assumes a one-page stack guard, which is not always a valid assumption. 3. The kernel puts [heap] and [stack] closer together than RLIMIT_STACK plus guard space [hc] 0x3ffffd46000-0x3ffff5b1000 [1] 7,950,336 0x79_5000 036250000 0111.1001.0101.0000.0000.0000 4. pthread_attr_getstack() does not take into account the larger stack guard. So I can't say for sure that the kernel is at fault - the test plus glibc plus kernel simply disagree on their assumptions.
We committed a change to upstream master to move this test into a container to avoid running it directly underneath the dynamic loader. commit 279c68ce1336d84d82ce491a4b77086e574ba380 Author: DJ Delorie <dj> Date: Mon Feb 3 14:57:23 2020 -0500 Run nptl/tst-pthread-getattr in a container See https://bugzilla.redhat.com/show_bug.cgi?id=1653942 This test depends on the kernel's assignment of memory regions, but running under ld.so explicitly changes those assignments, sometimes sufficiently to cause the test to fail (esp with address space randomization). The easiest way to "fix" the test, is to run it the way the user would - without ld.so. Running it in a container does that. Reviewed-by: Carlos O'Donell <carlos> This ensures the kernel gives the process the correct VMA layout for a dynamically linked process. We are going to sync Fedora Rawhide to master and that closes this bug. I'm going to mark this CLOSED/RAWHIDE since this is effectively done.