Bug 1527887 - glibc: PTHREAD_STACK_MIN is too small on x86-64
Summary: glibc: PTHREAD_STACK_MIN is too small on x86-64
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: glibc
Version: 27
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Florian Weimer
QA Contact: Fedora Extras Quality Assurance
URL: http://faf.lab.eng.brq.redhat.com/faf...
Whiteboard: abrt_hash:0bcbd845325851b2a8bc116b799...
: 1534340 1564527 (view as bug list)
Depends On: 1527903
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-20 10:55 UTC by Lukas Slebodnik
Modified: 2018-04-17 09:49 UTC (History)
15 users (show)

Fixed In Version: glibc-2.26.9000-38.fc28 glibc-2.26-24.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1527904 (view as bug list)
Environment:
Last Closed: 2018-01-24 08:47:07 UTC
Type: ---


Attachments (Terms of Use)
File: backtrace (73.41 KB, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: cgroup (178 bytes, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: core_backtrace (5.19 KB, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: cpuinfo (1.51 KB, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: dso_list (999 bytes, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: environ (158 bytes, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: exploitable (82 bytes, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: limits (1.29 KB, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: maps (5.21 KB, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: mountinfo (4.10 KB, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: namespaces (125 bytes, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: open_fds (135 bytes, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
File: proc_pid_status (1.23 KB, text/plain)
2017-12-20 10:55 UTC, Lukas Slebodnik
no flags Details
coredump (784.00 KB, application/x-core)
2017-12-20 11:24 UTC, Lukas Slebodnik
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Sourceware 22636 0 P2 RESOLVED PTHREAD_STACK_MIN is too small on x86-64 2020-03-05 15:39:28 UTC
Sourceware 22637 0 P2 RESOLVED guard size is subtracted from thread stack size instead of adding it on top 2020-03-05 15:39:29 UTC

Description Lukas Slebodnik 2017-12-20 10:55:22 UTC
Version-Release number of selected component:
ntp-4.2.8p10-3.fc27

Additional info:
reporter:       libreport-2.9.3
backtrace_rating: 0
cmdline:        /usr/sbin/ntpd -u ntp:ntp -g -x
crash_function: _dl_lookup_symbol_x
executable:     /usr/sbin/ntpd
global_pid:     25929
kernel:         4.14.6-300.fc27.x86_64
runlevel:       N 3
type:           CCpp
uid:            0

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 _dl_lookup_symbol_x
 #1 _dl_fixup
 #2 _dl_runtime_resolve_xsavec
 #3 ??
 #4 ??
 #5 dl_iterate_phdr
 #6 _Unwind_Find_FDE
 #7 ??
 #8 ??
 #9 _Unwind_ForcedUnwind

Comment 1 Lukas Slebodnik 2017-12-20 10:55:28 UTC
Created attachment 1370398 [details]
File: backtrace

Comment 2 Lukas Slebodnik 2017-12-20 10:55:30 UTC
Created attachment 1370399 [details]
File: cgroup

Comment 3 Lukas Slebodnik 2017-12-20 10:55:31 UTC
Created attachment 1370400 [details]
File: core_backtrace

Comment 4 Lukas Slebodnik 2017-12-20 10:55:31 UTC
Created attachment 1370401 [details]
File: cpuinfo

Comment 5 Lukas Slebodnik 2017-12-20 10:55:32 UTC
Created attachment 1370402 [details]
File: dso_list

Comment 6 Lukas Slebodnik 2017-12-20 10:55:34 UTC
Created attachment 1370403 [details]
File: environ

Comment 7 Lukas Slebodnik 2017-12-20 10:55:35 UTC
Created attachment 1370404 [details]
File: exploitable

Comment 8 Lukas Slebodnik 2017-12-20 10:55:36 UTC
Created attachment 1370405 [details]
File: limits

Comment 9 Lukas Slebodnik 2017-12-20 10:55:37 UTC
Created attachment 1370406 [details]
File: maps

Comment 10 Lukas Slebodnik 2017-12-20 10:55:38 UTC
Created attachment 1370407 [details]
File: mountinfo

Comment 11 Lukas Slebodnik 2017-12-20 10:55:39 UTC
Created attachment 1370408 [details]
File: namespaces

Comment 12 Lukas Slebodnik 2017-12-20 10:55:40 UTC
Created attachment 1370409 [details]
File: open_fds

Comment 13 Lukas Slebodnik 2017-12-20 10:55:41 UTC
Created attachment 1370410 [details]
File: proc_pid_status

Comment 14 Lukas Slebodnik 2017-12-20 11:19:43 UTC
sh$ rpm -q glibc ntp
glibc-2.26-20.fc27.x86_64
ntp-4.2.8p10-3.fc27.x86_64

Comment 15 Lukas Slebodnik 2017-12-20 11:21:27 UTC
Adding glibc developers to CC

Comment 16 Lukas Slebodnik 2017-12-20 11:24:23 UTC
Created attachment 1370424 [details]
coredump

Comment 18 Florian Weimer 2017-12-20 12:19:58 UTC
PTHREAD_STACK_MIN is too small:

my_pthread_warmup () at ntpd.c:316
316		rc = pthread_attr_setstacksize(&thr_attr, PTHREAD_STACK_MIN);
(gdb) s
__pthread_attr_setstacksize (attr=0x7fffffffda90, stacksize=16384) at pthread_attr_setstacksize.c:38
38	  int ret = check_stacksize_attr (stacksize);
(gdb) print stacksize
$5 = 16384

CPU flags contains: avx512f avx512dq

The following thread is canceled:

292	static void*
293	my_pthread_warmup_worker(
294		void *thread_args)
295	{
296		(void)thread_args;
297		for (;;)
298			sleep(10);
299		return NULL;
300	}

Both the signal handler frame and the dynamic linker trampoline need an XSAVE area, which is about 2688 bytes.  Despite for the request of a 16 KiB stack, the actually usable stack appears to be only 8 KiB (4 KiB subtraction due to the guard page is expected).  The libgcc functions have fairly large stack frames, too.

We need to fix the guard page accounting in glibc (so that we actually get 16 KiB of stack) and link libgcc with BIND_NOW.

Comment 19 Florian Weimer 2017-12-20 13:19:34 UTC
Miroslav, Lukas, if you need a quick fix in ntp, you can just patch out the explicit stack size setting for the cancel warmup:

    313 #if defined(HAVE_PTHREAD_ATTR_GETSTACKSIZE) && \
    314     defined(HAVE_PTHREAD_ATTR_SETSTACKSIZE) && \
    315     defined(PTHREAD_STACK_MIN)
    316         rc = pthread_attr_setstacksize(&thr_attr, PTHREAD_STACK_MIN);
    317         if (0 != rc)
    318                 msyslog(LOG_ERR,
    319                         "my_pthread_warmup: pthread_attr_setstacksize() -> %s",
    320                         strerror(rc));
    321 #endif

It will take some time until we have gcc/glibc fixes in all Fedora branches.

Comment 20 Lukas Slebodnik 2017-12-21 14:53:33 UTC
(In reply to Florian Weimer from comment #19)
> Miroslav, Lukas, if you need a quick fix in ntp, you can just patch out the
> explicit stack size setting for the cancel warmup:
> 

I noticed the crash just by a chance with unrelated testing.
It did not happen in other similar tests on f27; therefore I filed this BZ.

Florian,
thank you very much for looking into this.

Comment 21 Carlos O'Donell 2017-12-21 16:59:35 UTC
(In reply to Florian Weimer from comment #19)
> Miroslav, Lukas, if you need a quick fix in ntp, you can just patch out the
> explicit stack size setting for the cancel warmup:
> 
>     313 #if defined(HAVE_PTHREAD_ATTR_GETSTACKSIZE) && \
>     314     defined(HAVE_PTHREAD_ATTR_SETSTACKSIZE) && \
>     315     defined(PTHREAD_STACK_MIN)
>     316         rc = pthread_attr_setstacksize(&thr_attr, PTHREAD_STACK_MIN);
>     317         if (0 != rc)
>     318                 msyslog(LOG_ERR,
>     319                         "my_pthread_warmup:
> pthread_attr_setstacksize() -> %s",
>     320                         strerror(rc));
>     321 #endif
> 
> It will take some time until we have gcc/glibc fixes in all Fedora branches.

Lukas,

The value of PTHREAD_STACK_MIN is the minimum amount of stack required to start the thread. This does not include enough stack to cancel the thread. Cancellation is not the same a joining the thread because it must run all cancellation handlers, and the cancellation machinery itself uses stack. How much you use is highly variable and somewhat difficult to compute, sorry about that, but PTHREAD_STACK_MIN is certainly not enough.

We have some compatibility issues here which we will work hard to solve, so that existing use cases continue to work, which might not work anymore after some dynamic loader changes that use more stack.

However, you *need* to increase that stack size in future builds for there to be enough stack to cancel the thread.

Florian,

How do we want to handle this? Could we backport Szabolcs guard page accounting changes to F27 to get back the guard page size?

Comment 22 Florian Weimer 2018-01-02 10:01:07 UTC
(In reply to Carlos O'Donell from comment #21)
> How do we want to handle this? Could we backport Szabolcs guard page
> accounting changes to F27 to get back the guard page size?

We should backport the guard page accounting fix.  In addition, libgcc_s.so should be built with BIND_NOW (following the Fedora packaging guidelines).

Comment 23 Florian Weimer 2018-01-10 12:30:25 UTC
glibc-2.26.9000-38.fc28 effectively adds an additional 4 KiB to the stack size, fixing this issue.  (I checked it on an AVX-512 machine with ntpd from rawhide.)  A subsequent build will also disable lazy binding for libgcc_s.so, increasing compatibility with future CPUs with even larger register files.

Comment 24 Miroslav Lichvar 2018-01-15 08:37:05 UTC
*** Bug 1534340 has been marked as a duplicate of this bug. ***

Comment 25 Ian Donaldson 2018-01-15 12:40:46 UTC
As suggested, as an interim solution, can ntpd be rebuilt with a bigger 
thread stack size?

I'm at a loss as to why this has only showed up as an issue
on one of my systems so far... about 10 others with the same
versions of everything are running ntpd happily; clearly something 
must be different on this (very new) system to cause it to have issues but I'm
not sure what... 
(the system in question also has a really fast clock for some reason;
something I only discovered today since ntpd hasn't been correcting it; 
its gained 55s in the last 6 hours!; clearly something else ain't right)

Comment 26 Ian Donaldson 2018-01-15 12:41:12 UTC
rebuilt for fc27 that is..

Comment 27 Florian Weimer 2018-01-15 12:44:52 UTC
(In reply to Ian Donaldson from comment #25)
> As suggested, as an interim solution, can ntpd be rebuilt with a bigger 
> thread stack size?

I plan to backport various countermeasures into the next glibc update for F27.  A new build will be available early this week, perhaps even today.

> I'm at a loss as to why this has only showed up as an issue
> on one of my systems so far...

Does the system have AVX-512 support?  (For example, does it have a Skylake server CPU?)

Comment 28 Ian Donaldson 2018-01-15 12:51:57 UTC
Hi Florian,
That sounds great.


The new system ntpd is having issues on seems to have avx-512 support,
and the others where it is working fine are way older.

model name      : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti retpoline intel_ppin intel_pt mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke

Ian D

Comment 29 Fedora Update System 2018-01-17 14:03:31 UTC
glibc-2.26-24.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-7714b514e2

Comment 30 Fedora Update System 2018-01-18 02:11:19 UTC
glibc-2.26-24.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-7714b514e2

Comment 31 Fedora Update System 2018-01-23 21:47:34 UTC
glibc-2.26-24.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 32 Christian Heimes 2018-04-17 09:49:08 UTC
*** Bug 1564527 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.