Bug 1527887

Created attachment 1370398 [details]
File: backtrace

Created attachment 1370399 [details]
File: cgroup

Created attachment 1370400 [details]
File: core_backtrace

Created attachment 1370401 [details]
File: cpuinfo

Created attachment 1370402 [details]
File: dso_list

Created attachment 1370403 [details]
File: environ

Created attachment 1370404 [details]
File: exploitable

Created attachment 1370405 [details]
File: limits

Created attachment 1370406 [details]
File: maps

Created attachment 1370407 [details]
File: mountinfo

Created attachment 1370408 [details]
File: namespaces

Created attachment 1370409 [details]
File: open_fds

Created attachment 1370410 [details]
File: proc_pid_status

sh$ rpm -q glibc ntp
glibc-2.26-20.fc27.x86_64
ntp-4.2.8p10-3.fc27.x86_64

Adding glibc developers to CC

Created attachment 1370424 [details]
coredump

PTHREAD_STACK_MIN is too small:

my_pthread_warmup () at ntpd.c:316
316		rc = pthread_attr_setstacksize(&thr_attr, PTHREAD_STACK_MIN);
(gdb) s
__pthread_attr_setstacksize (attr=0x7fffffffda90, stacksize=16384) at pthread_attr_setstacksize.c:38
38	  int ret = check_stacksize_attr (stacksize);
(gdb) print stacksize
$5 = 16384

CPU flags contains: avx512f avx512dq

The following thread is canceled:

292	static void*
293	my_pthread_warmup_worker(
294		void *thread_args)
295	{
296		(void)thread_args;
297		for (;;)
298			sleep(10);
299		return NULL;
300	}

Both the signal handler frame and the dynamic linker trampoline need an XSAVE area, which is about 2688 bytes.  Despite for the request of a 16 KiB stack, the actually usable stack appears to be only 8 KiB (4 KiB subtraction due to the guard page is expected).  The libgcc functions have fairly large stack frames, too.

We need to fix the guard page accounting in glibc (so that we actually get 16 KiB of stack) and link libgcc with BIND_NOW.

Miroslav, Lukas, if you need a quick fix in ntp, you can just patch out the explicit stack size setting for the cancel warmup:

    313 #if defined(HAVE_PTHREAD_ATTR_GETSTACKSIZE) && \
    314     defined(HAVE_PTHREAD_ATTR_SETSTACKSIZE) && \
    315     defined(PTHREAD_STACK_MIN)
    316         rc = pthread_attr_setstacksize(&thr_attr, PTHREAD_STACK_MIN);
    317         if (0 != rc)
    318                 msyslog(LOG_ERR,
    319                         "my_pthread_warmup: pthread_attr_setstacksize() -> %s",
    320                         strerror(rc));
    321 #endif

It will take some time until we have gcc/glibc fixes in all Fedora branches.

(In reply to Florian Weimer from comment #19)
> Miroslav, Lukas, if you need a quick fix in ntp, you can just patch out the
> explicit stack size setting for the cancel warmup:
> 

I noticed the crash just by a chance with unrelated testing.
It did not happen in other similar tests on f27; therefore I filed this BZ.

Florian,
thank you very much for looking into this.

(In reply to Florian Weimer from comment #19)
> Miroslav, Lukas, if you need a quick fix in ntp, you can just patch out the
> explicit stack size setting for the cancel warmup:
> 
>     313 #if defined(HAVE_PTHREAD_ATTR_GETSTACKSIZE) && \
>     314     defined(HAVE_PTHREAD_ATTR_SETSTACKSIZE) && \
>     315     defined(PTHREAD_STACK_MIN)
>     316         rc = pthread_attr_setstacksize(&thr_attr, PTHREAD_STACK_MIN);
>     317         if (0 != rc)
>     318                 msyslog(LOG_ERR,
>     319                         "my_pthread_warmup:
> pthread_attr_setstacksize() -> %s",
>     320                         strerror(rc));
>     321 #endif
> 
> It will take some time until we have gcc/glibc fixes in all Fedora branches.

Lukas,

The value of PTHREAD_STACK_MIN is the minimum amount of stack required to start the thread. This does not include enough stack to cancel the thread. Cancellation is not the same a joining the thread because it must run all cancellation handlers, and the cancellation machinery itself uses stack. How much you use is highly variable and somewhat difficult to compute, sorry about that, but PTHREAD_STACK_MIN is certainly not enough.

We have some compatibility issues here which we will work hard to solve, so that existing use cases continue to work, which might not work anymore after some dynamic loader changes that use more stack.

However, you *need* to increase that stack size in future builds for there to be enough stack to cancel the thread.

Florian,

How do we want to handle this? Could we backport Szabolcs guard page accounting changes to F27 to get back the guard page size?

(In reply to Carlos O'Donell from comment #21)
> How do we want to handle this? Could we backport Szabolcs guard page
> accounting changes to F27 to get back the guard page size?

We should backport the guard page accounting fix.  In addition, libgcc_s.so should be built with BIND_NOW (following the Fedora packaging guidelines).

glibc-2.26.9000-38.fc28 effectively adds an additional 4 KiB to the stack size, fixing this issue.  (I checked it on an AVX-512 machine with ntpd from rawhide.)  A subsequent build will also disable lazy binding for libgcc_s.so, increasing compatibility with future CPUs with even larger register files.

*** Bug 1534340 has been marked as a duplicate of this bug. ***

As suggested, as an interim solution, can ntpd be rebuilt with a bigger 
thread stack size?

I'm at a loss as to why this has only showed up as an issue
on one of my systems so far... about 10 others with the same
versions of everything are running ntpd happily; clearly something 
must be different on this (very new) system to cause it to have issues but I'm
not sure what... 
(the system in question also has a really fast clock for some reason;
something I only discovered today since ntpd hasn't been correcting it; 
its gained 55s in the last 6 hours!; clearly something else ain't right)

rebuilt for fc27 that is..

(In reply to Ian Donaldson from comment #25)
> As suggested, as an interim solution, can ntpd be rebuilt with a bigger 
> thread stack size?

I plan to backport various countermeasures into the next glibc update for F27.  A new build will be available early this week, perhaps even today.

> I'm at a loss as to why this has only showed up as an issue
> on one of my systems so far...

Does the system have AVX-512 support?  (For example, does it have a Skylake server CPU?)

Hi Florian,
That sounds great.


The new system ntpd is having issues on seems to have avx-512 support,
and the others where it is working fine are way older.

model name      : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti retpoline intel_ppin intel_pt mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke

Ian D

glibc-2.26-24.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-7714b514e2

glibc-2.26-24.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-7714b514e2

glibc-2.26-24.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

*** Bug 1564527 has been marked as a duplicate of this bug. ***