Bug 1393909 - [abrt] traceroute: __pthread_initialize_minimal_internal(): traceroute killed by SIGSEGV
Summary: [abrt] traceroute: __pthread_initialize_minimal_internal(): traceroute killed...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: glibc
Version: 24
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Florian Weimer
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:f5dca8f1096719f2f28833dc779...
Depends On:
Blocks: 1410154
TreeView+ depends on / blocked
 
Reported: 2016-11-10 15:19 UTC by Matthew Wilson
Modified: 2017-07-27 11:05 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-27 11:05:11 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: backtrace (28.22 KB, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: cgroup (242 bytes, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: core_backtrace (745 bytes, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: dso_list (902 bytes, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: environ (2.96 KB, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: exploitable (82 bytes, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: limits (1.29 KB, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: maps (4.16 KB, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: mountinfo (3.60 KB, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: namespaces (102 bytes, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: open_fds (786 bytes, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: proc_pid_status (1.10 KB, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
File: var_log_messages (307 bytes, text/plain)
2016-11-10 15:19 UTC, Matthew Wilson
no flags Details
Backtrace from route (7.36 KB, text/plain)
2016-11-10 17:23 UTC, Matthew Wilson
no flags Details
Backtrace from traceroute (8.23 KB, text/plain)
2016-11-10 17:23 UTC, Matthew Wilson
no flags Details
Output of valgrind (4.55 KB, text/plain)
2016-11-10 19:27 UTC, Matthew Wilson
no flags Details
LD_DEBUG=files route (4.09 KB, text/plain)
2016-11-10 20:09 UTC, Matthew Wilson
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Sourceware 16628 0 P2 RESOLVED Segfault after a binary without pthread dlopen()s a library linked with pthread 2020-07-06 09:07:51 UTC

Description Matthew Wilson 2016-11-10 15:19:30 UTC
Description of problem:
traceroute <any address>

Version-Release number of selected component:
traceroute-2.1.0-2.fc24

Additional info:
reporter:       libreport-2.7.2
backtrace_rating: 3
cmdline:        traceroute www.bbc.co.uk
crash_function: __pthread_initialize_minimal_internal
executable:     /usr/bin/traceroute
global_pid:     5888
kernel:         4.8.4-200.fc24.x86_64
pkg_fingerprint: 73BD E983 81B4 6521
pkg_vendor:     Fedora Project
runlevel:       N 5
type:           CCpp
uid:            1000

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 __pthread_initialize_minimal_internal at nptl-init.c:322
 #1 _init at ../sysdeps/x86_64/crti.S:72
 #2 ??
 #3 call_init at dl-init.c:58
 #5 _dl_init at dl-init.c:87
 #6 dl_open_worker at dl-open.c:564
 #7 _dl_catch_error at dl-error.c:187
 #8 _dl_open at dl-open.c:649
 #9 do_dlopen at dl-libc.c:87
 #10 _dl_catch_error at dl-error.c:187

Comment 1 Matthew Wilson 2016-11-10 15:19:35 UTC
Created attachment 1219438 [details]
File: backtrace

Comment 2 Matthew Wilson 2016-11-10 15:19:36 UTC
Created attachment 1219439 [details]
File: cgroup

Comment 3 Matthew Wilson 2016-11-10 15:19:38 UTC
Created attachment 1219440 [details]
File: core_backtrace

Comment 4 Matthew Wilson 2016-11-10 15:19:39 UTC
Created attachment 1219441 [details]
File: dso_list

Comment 5 Matthew Wilson 2016-11-10 15:19:40 UTC
Created attachment 1219442 [details]
File: environ

Comment 6 Matthew Wilson 2016-11-10 15:19:41 UTC
Created attachment 1219443 [details]
File: exploitable

Comment 7 Matthew Wilson 2016-11-10 15:19:43 UTC
Created attachment 1219444 [details]
File: limits

Comment 8 Matthew Wilson 2016-11-10 15:19:44 UTC
Created attachment 1219445 [details]
File: maps

Comment 9 Matthew Wilson 2016-11-10 15:19:45 UTC
Created attachment 1219446 [details]
File: mountinfo

Comment 10 Matthew Wilson 2016-11-10 15:19:47 UTC
Created attachment 1219447 [details]
File: namespaces

Comment 11 Matthew Wilson 2016-11-10 15:19:48 UTC
Created attachment 1219448 [details]
File: open_fds

Comment 12 Matthew Wilson 2016-11-10 15:19:49 UTC
Created attachment 1219449 [details]
File: proc_pid_status

Comment 13 Matthew Wilson 2016-11-10 15:19:51 UTC
Created attachment 1219450 [details]
File: var_log_messages

Comment 14 Matthew Wilson 2016-11-10 15:57:05 UTC
Additional information:  I'm on a VPN (PPTP).

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 enp0s8
10.0.0.0        0.0.0.0         255.255.255.0   U     50     0        0 ppp0
10.0.0.42       0.0.0.0         255.255.255.255 UH    50     0        0 ppp0
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 enp0s8
213.218.195.63  192.168.1.1     255.255.255.255 UGH   0      0        0 enp0s8

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
Segmentation fault (core dumped)

Very similar problem, according to GDB:

#0  0x00007ffff01fecfe in __pthread_initialize_minimal_internal () from /lib64/libpthread.so.0
#1  0x00007ffff01fe549 in _init () from /lib64/libpthread.so.0
#2  0x00007ffff04157b0 in ?? () from /lib64/libnss_mdns4_minimal.so.2
...
#13 0x00007ffff782373a in gethostbyaddr_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
#14 0x00007ffff782b978 in gni_host_inet_name.isra () from /lib64/libc.so.6

Comment 15 Dmitry Butskoy 2016-11-10 16:11:06 UTC
"route -n" vs. "route" -- looks like something wrong in DNS resolving code.
Reassign to glibc.

Comment 16 Florian Weimer 2016-11-10 16:32:27 UTC
Would you please try to reproduce this with glibc-debuginfo and nss-mdns-debuginfo install, and post “bt full” output?  Thanks.

Comment 17 Matthew Wilson 2016-11-10 17:23:05 UTC
Created attachment 1219465 [details]
Backtrace from route

Attached backtrace after running route.

Comment 18 Matthew Wilson 2016-11-10 17:23:48 UTC
Created attachment 1219466 [details]
Backtrace from traceroute

Attached backtrace from running traceroute.

Comment 19 Florian Weimer 2016-11-10 18:22:41 UTC
Could you run “rpm -V glibc nss-mdns” and paste the output, please?

I can't explain the backtraces you get based on the file contents of /lib64/libnss_mdns4_minimal.so.2, so maybe the file is corrupted.

Comment 20 Matthew Wilson 2016-11-10 19:07:30 UTC
# rpm -V glibc nss-mdns
[no output]

# rpm -V glibc-debuginfo nss-mdns-debuginfo
[no output]

(Needed to be root to read two files for glibc in /usr/sbin)

I can reproduce the issue if I add any non-local network destination in my routing tables, e.g.

# route add www.google.co.uk gw 192.168.1.99

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         BrightBox.ee    0.0.0.0         UG    100    0        0 enp0s8
Segmentation fault (core dumped)

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 enp0s8
172.217.23.3    192.168.1.99    255.255.255.255 UGH   0      0        0 enp0s8
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 enp0s8

Comment 21 Florian Weimer 2016-11-10 19:18:14 UTC
Hmm, can you run route under valgrind, using “valgrind route”?  Maybe this is plain old memory corruption.

I will try to replicate this, but the issue may depend on your local network and its mDNS responders.

Comment 22 Matthew Wilson 2016-11-10 19:27:29 UTC
Created attachment 1219510 [details]
Output of valgrind

Attached output of valgrind (success + failure cases).

Comment 23 Florian Weimer 2016-11-10 19:32:38 UTC
Still no idea.  Does this crash as well:

python -c 'import socket; print socket.getnameinfo(("172.217.23.3", 80), 0)'

Or this?

python -c 'import socket; print socket.getnameinfo(("192.168.1.99", 80), 0)'

Comment 24 Matthew Wilson 2016-11-10 19:52:08 UTC
$ python -c 'import socket; print socket.getnameinfo(("172.217.23.3", 80), 0)'
('lhr35s01-in-f3.1e100.net', 'http')

$ python -c 'import socket; print socket.getnameinfo(("192.168.1.99", 80), 0)'
('192.168.1.99', 'http')

From a quick glance at the source along with gdb, it's getting stuck on:

  THREAD_SETMEM (pd, cpuclock_offset, GL(dl_cpuclock_offset));

If setting that field fails, why do the previous ones (lines 317+318) succeed?

Comment 25 Florian Weimer 2016-11-10 20:00:33 UTC
(In reply to mj.wilson.uk from comment #24)
> $ python -c 'import socket; print socket.getnameinfo(("172.217.23.3", 80),
> 0)'
> ('lhr35s01-in-f3.1e100.net', 'http')
> 
> $ python -c 'import socket; print socket.getnameinfo(("192.168.1.99", 80),
> 0)'
> ('192.168.1.99', 'http')

Next candidate:

python -c 'import socket; print socket.getnameinfo(("255.255.255.255", 80), 0)'

> From a quick glance at the source along with gdb, it's getting stuck on:
> 
>   THREAD_SETMEM (pd, cpuclock_offset, GL(dl_cpuclock_offset));
> 
> If setting that field fails, why do the previous ones (lines 317+318)
> succeed?

This code should not run *at all*.  It is for libpthread initialization, but libnss_mdns4_minimal.so.2 isn't linked against libpthread.  Maybe the code is in fact loading *another* NSS module after libnss_mdns4_minimal.so.2, and the backtrace is misleading because the stack has been corrupted.

What's the contents of the “hosts” line in /etc/nsswitch.conf?

What happens if you run “LD_DEBUG=files /sbin/route” in the crashing case?

Comment 26 Matthew Wilson 2016-11-10 20:09:34 UTC
Created attachment 1219526 [details]
LD_DEBUG=files route

/etc/nsswitch.conf:

hosts:      files dns wins mdns4_minimal myhostname mymachines

Seems to be unaltered from the rpm (AFAICT).

Attached the output with LD_DEBUG=files.

Comment 27 Florian Weimer 2016-11-10 20:25:41 UTC
Okay, I can reproduce it after adding “wins“ to the “hosts” line.  Now we are getting somewhere!

Comment 28 Florian Weimer 2016-11-10 20:46:33 UTC
Reduced test case:

#include <dlfcn.h>
#include <err.h>
#include <stdio.h>

int
main (void)
{
  void *handle = dlopen ("libnss_wins.so.2", RTLD_LAZY);
  printf ("handle: %p\n", handle);
  printf ("error: %s\n", dlerror ());
  handle = dlopen ("libnss_mdns4_minimal.so.2", RTLD_LAZY);
  printf ("handle: %p\n", handle);
  printf ("error: %s\n", dlerror ());
}

This is triggered because loading libnss_wins.so.2 fails due to a missing library (libreplace-samba4.so), for which libnss_wins.so.2 as a DT_NEEDED entry.

Comment 29 Fedora End Of Life 2017-07-25 23:50:16 UTC
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 30 Florian Weimer 2017-07-27 11:05:11 UTC
Fixed in glibc 2.25.


Note You need to log in before you can comment on or make changes to this bug.