Bug 977995 - [abrt] openssh-server-6.2p2-3.fc19: _IO_vfprintf_internal: Process /usr/sbin/sshd was killed by signal 11 (SIGSEGV)
[abrt] openssh-server-6.2p2-3.fc19: _IO_vfprintf_internal: Process /usr/sbin/...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: tcp_wrappers (Show other bugs)
19
x86_64 Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Petr Lautrbach
Fedora Extras Quality Assurance
abrt_hash:23c0ade763bd255f61a9f8b78cf...
:
: 986427 987483 (view as bug list)
Depends On:
Blocks: 979009
  Show dependency treegraph
 
Reported: 2013-06-25 14:32 EDT by Felix Kaechele
Modified: 2014-09-22 03:35 EDT (History)
9 users (show)

See Also:
Fixed In Version: tcp_wrappers-7.6-75.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 979009 (view as bug list)
Environment:
Last Closed: 2013-08-20 20:04:09 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: backtrace (257.33 KB, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: cgroup (154 bytes, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: core_backtrace (777 bytes, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: dso_list (2.96 KB, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: environ (557 bytes, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: limits (1.29 KB, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: maps (15.07 KB, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: open_fds (218 bytes, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: proc_pid_status (895 bytes, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
File: var_log_messages (2.39 KB, text/plain)
2013-06-25 14:32 EDT, Felix Kaechele
no flags Details
broken hosts.deny file (338.04 KB, text/plain)
2013-06-25 14:39 EDT, Felix Kaechele
no flags Details

  None (edit)
Description Felix Kaechele 2013-06-25 14:32:15 EDT
Description of problem:
Trying to connect to sshd via IPv6.
Client sees "ssh_exchange_identification: read: Connection reset by peer"

Version-Release number of selected component:
openssh-server-6.2p2-3.fc19

Additional info:
reporter:       libreport-2.1.5
backtrace_rating: 4
cmdline:        'sshd: [accepted]' '' '' '' ''
crash_function: _IO_vfprintf_internal
executable:     /usr/sbin/sshd
kernel:         3.9.6-301.fc19.x86_64
runlevel:       N 3
uid:            0

Truncated backtrace:
Thread no. 1 (7 frames)
 #0 _IO_vfprintf_internal at vfprintf.c:1635
 #1 ___printf_chk at printf_chk.c:36
 #2 printf at /usr/include/bits/stdio2.h:104
 #3 host_match at hosts_access.c:352
 #4 list_match at hosts_access.c:211
 #5 table_match at hosts_access.c:168
 #6 hosts_access at hosts_access.c:127
Comment 1 Felix Kaechele 2013-06-25 14:32:20 EDT
Created attachment 765229 [details]
File: backtrace
Comment 2 Felix Kaechele 2013-06-25 14:32:24 EDT
Created attachment 765231 [details]
File: cgroup
Comment 3 Felix Kaechele 2013-06-25 14:32:27 EDT
Created attachment 765233 [details]
File: core_backtrace
Comment 4 Felix Kaechele 2013-06-25 14:32:30 EDT
Created attachment 765235 [details]
File: dso_list
Comment 5 Felix Kaechele 2013-06-25 14:32:34 EDT
Created attachment 765237 [details]
File: environ
Comment 6 Felix Kaechele 2013-06-25 14:32:37 EDT
Created attachment 765239 [details]
File: limits
Comment 7 Felix Kaechele 2013-06-25 14:32:41 EDT
Created attachment 765241 [details]
File: maps
Comment 8 Felix Kaechele 2013-06-25 14:32:46 EDT
Created attachment 765244 [details]
File: open_fds
Comment 9 Felix Kaechele 2013-06-25 14:32:51 EDT
Created attachment 765245 [details]
File: proc_pid_status
Comment 10 Felix Kaechele 2013-06-25 14:32:54 EDT
Created attachment 765246 [details]
File: var_log_messages
Comment 11 Felix Kaechele 2013-06-25 14:39:21 EDT
Created attachment 765247 [details]
broken hosts.deny file

This bug is triggered by a broken hosts.deny file.
I have attached my hosts.deny file that was present at the time of the crash.
It looks very strange. I have no idea how that happened, though.
Comment 12 Petr Lautrbach 2013-06-27 08:42:18 EDT
There's a bug in hosts_access.c - host_match() function introduced in cf076b1867b6310182b72007111dfb89612eedcf commit.
Comment 13 Henrique Martins 2013-07-30 17:40:51 EDT
*** Bug 986427 has been marked as a duplicate of this bug. ***
Comment 14 Henrique Martins 2013-07-30 17:45:00 EDT
See duplicate bug 986427.  This problem caused rpcbind and sshd to segfault on one of my systems.  However, my hosts.deny contained a single line of the form:

ALL:  machine.domain.com

which should be legit as far as I can tell, thus even a non-broken hosts.deny (or hosts.allow) seems to cause problems.
Comment 15 nvwarr 2013-08-11 02:24:38 EDT
I have experienced the same problem with sshd giving a segfault.

Program terminated with signal 11, Segmentation fault.
#0  0x00007f478907fe29 in _IO_vfprintf_internal (s=s@entry=0x7f478d916e50, format=<optimized out>, format@entry=0x7fff3eef4750 "warning: /etc/hosts.allow, line 11: host name/address mismatch: %s != %.*s", ap=<optimized out>)
    at vfprintf.c:1635
#1  0x00007f47891418a6 in ___vfprintf_chk (fp=fp@entry=0x7f478d916e50, flag=flag@entry=1, format=format@entry=0x7fff3eef4750 "warning: /etc/hosts.allow, line 11: host name/address mismatch: %s != %.*s", ap=ap@entry=0x7fff3eef6788)
    at vfprintf_chk.c:34
#2  0x00007f4789126245 in __GI___vsyslog_chk (pri=<optimized out>, pri@entry=3, flag=flag@entry=1, fmt=fmt@entry=0x7fff3eef4750 "warning: /etc/hosts.allow, line 11: host name/address mismatch: %s != %.*s", ap=ap@entry=0x7fff3eef6788)
    at ../misc/syslog.c:222
#3  0x00007f478b85db61 in vsyslog (__ap=0x7fff3eef6788, __fmt=0x7fff3eef4750 "warning: /etc/hosts.allow, line 11: host name/address mismatch: %s != %.*s", __pri=3) at /usr/include/bits/syslog.h:47
#4  tcpd_diag (tag=tag@entry=0x7f478b85e0e9 "warning", format=format@entry=0x7f478b85e418 "host name/address mismatch: %s != %.*s", ap=ap@entry=0x7fff3eef6788, severity=3) at diag.c:45
#5  0x00007f478b85dc48 in tcpd_warn (format=format@entry=0x7f478b85e418 "host name/address mismatch: %s != %.*s") at diag.c:55
#6  0x00007f478b85cfeb in sock_hostname (host=0x7fff3eef73d0) at socket.c:277
#7  0x00007f478b85c539 in eval_hostname (host=host@entry=0x7fff3eef73d0) at eval.c:77
#8  0x00007f478b85ad88 in host_match (tok=0x7fff3eef69b5 "LOCAL", host=0x7fff3eef73d0) at hosts_access.c:305
#9  0x00007f478b85a54b in list_match (list=<optimized out>, request=0x7fff3eef72c0, match_fn=0x7f478b85b080 <client_match>) at hosts_access.c:211
#10 0x00007f478b85a729 in table_match (table=<optimized out>, request=request@entry=0x7fff3eef72c0) at hosts_access.c:168
#11 0x00007f478b85a84e in hosts_access (request=request@entry=0x7fff3eef72c0) at hosts_access.c:125
#12 0x00007f478be947dd in main (ac=<optimized out>, av=<optimized out>) at sshd.c:2066

I tried looking at a few values using gdb and I found it was triggered by a hostile attack.

In socket.c in the function sock_hostname, if I look at the contents of host, I see host->name gives indoautotech.com. There is absolutely no legitimate reason why this address would connect to my system, so this is clearly hostile. The message passed to tcp_warn is "host name/address mismatch: %s != %.*s" and in the code there is the comment "The host name does not map to the initial address. Perhaps someone has messed up. Perhaps someone compromised a name server." If I look up this name I get:

Name:	indoautotech.com
Address: 67.205.2.42

However, if I look at the contents of sin I get:
{sg_addr = {_sg_sa = {sa_family = 2, 
      sa_data = "\345\324z\264~K\000\000\000\000\000\000\000"}, _sg_sin = {
      sin_family = 2, sin_port = 54501, sin_addr = {s_addr = 1266594938}, 
      sin_zero = "\000\000\000\000\000\000\000"}, _sg_sin6 = {sin6_family = 2, 
      sin6_port = 54501, sin6_flowinfo = 1266594938, sin6_addr = {__in6_u = {
      __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 
      0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}
and s_addr corresponds to 122.180.126.75

Looking up this address I get:
Non-authoritative answer:
75.126.180.122.in-addr.arpa	name = indoautotech.com.

So indeed, it looks like it is a spoof of some sort. So this is a serious security issue.

I also note that if I look in tcpd_diag, I see that tcpd_context.file has the value "/etc/hosts.allow" and tcpd_context.line is 11. If I look in my hosts.allow file, that line is simply:
ALL: LOCAL
which should be legitimate, right? I suspect this is spurious. It is simply the first non-comment line in the file which is given in such cases, where the name and address don't match, right?

tcpd_warn is called with arguments:
  "host name/address mismatch: %s != %.*s", inet_ntop(SGFAM(sin), SGADDRP(sin), buf, sizeof(buf)), STRING_LENGTH, hp->h_name

The contents of buf at this point are "122.180.126.75", so I presume that inet_ntop has correctly returned a pointer to buf, but actually this is not guaranteed. If an error occurs, it will return NULL and set errno, so this is a possible cause of the error. I guess this should be trapped, just in case.
Unfortunately hp has been optimised out at this point, so I don't know what it is passing as hp. However, the code path should not be executed if it is NULL, so presumably there is something valid in hp->h_name and since the length is explicitly restricted to STRING_LENGTH (128) I don't see how this could cause a problem.

So unfortunately, I can't see what exactly causes this crash, but it is a security issue, so it is rather serious. I hope this information helps. Unfortunately, it is virtually impossible to reproduce controllably, because it depends on some evil cracker in India trying to break into my system:)
Comment 16 Petr Lautrbach 2013-08-13 08:31:39 EDT
Which version of tcp_wrappers do you use? Will update to the latest tcp_wrappers-7.6-74.fc19 update help?
Comment 17 nvwarr 2013-08-13 11:43:50 EDT
Sorry, I forgot to mention which version I was using. I was already using the latest tcp_wrappers-7.6-74.fc19. So, no that doesn't fix it.
Comment 18 Petr Lautrbach 2013-08-15 12:59:38 EDT
There was a missing #include <arpa/inet.h> in socket.c so a compiler guessed return value of inet_ntop() to be int instead of const char * 

rawhide build http://koji.fedoraproject.org/koji/taskinfo?taskID=5819690
Comment 19 Michael 2013-08-15 21:08:37 EDT
For what it's worth, that exact IP dropped my openssh today, with a backtrace identical to what nvwarr showed above. The hosts.allow line that it got fetched up on is:

ALL:localhost
Comment 20 nvwarr 2013-08-16 01:45:48 EDT
Yes, I think that is the correct explanation, Petr. I wrote a simple test program which uses the data extracted from the coredump and passed it to tcpd_warn. I did this yesterday, before you posted and wasn't able to reproduce the problem, but after I saw your post, I realised that I had included arpa/inet.h. Sure enough, when I removed that include I got the same crash as with sshd. So the sooner an update with this include added reaches the repositories, the better. Thanks for your effort.
Comment 21 Fedora Update System 2013-08-16 04:07:58 EDT
tcp_wrappers-7.6-75.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/tcp_wrappers-7.6-75.fc19
Comment 22 Petr Lautrbach 2013-08-16 04:10:31 EDT
*** Bug 987483 has been marked as a duplicate of this bug. ***
Comment 23 Fedora Update System 2013-08-16 18:59:32 EDT
Package tcp_wrappers-7.6-75.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing tcp_wrappers-7.6-75.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-14882/tcp_wrappers-7.6-75.fc19
then log in and leave karma (feedback).
Comment 24 nvwarr 2013-08-20 03:05:11 EDT
For some reason, it still hasn't reached my local mirror yet, but I've downloaded the rpm directly, installed it and restarted sshd. Now, we just have to wait until some evil person tests it:)

I can say, that with my own patch, which just included arpa/inet.h, I got the message "warning: /etc/hosts.allow, line 11: host name/address mismatch: 123.30.214.190 != static.vdc.vn" in the early hours of this morning, which is the point where it segfaulted without the patch. This is the first time I've got this message since I applied the patch. This again seems to be one of those nameserver misconfiguration things. 123.30.214.190 translates as static.vdc.vn, but if I look up static.vdc.vn, I get 203.162.0.78.

I won't give karma for your version yet, because I didn't install it until after this attempted breakin, but your tcp_wrappers-7.6-warnings.patch includes the important change, so I expect it to solve the issue. I guess compiler warnings do matter:)
Comment 25 Fedora Update System 2013-08-20 20:04:09 EDT
tcp_wrappers-7.6-75.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 26 Mahmoud Alshinhab 2014-09-19 15:14:51 EDT
Dears, 

I have the same problem but it causes the BackupExec Agent to shutdown 

my version is tcp_wrappers-7.6-77.el7.x86_64

kernel: beremote[]: segfault at 1f5840fc07d ip 00007f1554e1850c sp 00007f15531297c8 error 4 in libc-2.17.so[7f1554d98000+1b6000]


is there any other solution ?
Comment 27 Petr Lautrbach 2014-09-22 03:35:30 EDT
Mahmoud, 

this bug was filled against Fedora 19 and it is already closed. If you have any problem with your RHEL-7, please raise a ticket through your regular Red Hat support channels to make certain  it receives the proper attention and prioritization to assure a timely resolution.

Note You need to log in before you can comment on or make changes to this bug.