| Summary: | sshd segmentation fault | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | mailinglists |
| Component: | systemd | Assignee: | systemd-maint |
| Status: | CLOSED ERRATA | QA Contact: | Petr Sklenar <psklenar> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.2 | CC: | bblaskov, jscotka, jsynacek, psklenar, qe-baseos-security, systemd-maint-list, systemd-maint, szidek |
| Target Milestone: | rc | Keywords: | EasyFix, Patch |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | systemd-219-22.el7 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-04 00:53:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Thank you for the report. I probably don't have appropriate network setup. Can you provide backtrace and information about variables from gdb at the time of crash? strace report shows a lot of recvmsg, but I am not sure where they come from. Note that you need to set follow-fork-mode child to get into the pre-auth child and catch the segfault. When I can and learn how to, I will provide details as asked. In the mean time I found the exact same bug in CentOS 7 bug tracker: https://bugs.centos.org/view.php?id=10429&history=1 I also noticed segfaults with utilities like traceroute, tracepath, ping, .. That is not much helpful for our use case. The backtrace from iptables is also missing debug symbols: debuginfo-install openssh iptables systemd-libs (or whatever gdb complains about missing) I found similar report from last year without any resolution: https://bugs.freedesktop.org/show_bug.cgi?id=88340 If you are able to reliably reproduce the bug, at least backtrace would be very appreciated (can be extracted from created core dump). The problem is most probably not in openssh, but narrowing the problem down would be nice. If we are hitting the same issue as the linked above, it should go to systemd: $ rpm -qf /usr/lib64/libnss_myhostname.so.2 systemd-libs-219-19.el7_2.7.x86_64 Any thoughts from systemd? I am still able to reliably reproduce the bug. I guess you can to, by inserting 600000 (irrelevant) routes into your routing table. Just write a simple shell script. Disabling myhostname in nsswitch.conf fixes the problem. So you are correct, the problem is within systemd and their nss library when there are many routes and PTR records for DNS query are nonexistent. I have updated the bug report to reflect this. The issue was reported to systemd developers but was just ignored. It might be that they have fixed it already, but due to RH forking before major code changes upstream in systemd, it might not have been back ported. (source of info #CentOS on IRC). I will provide additional details when I have the time to do so, but only if you are unable to reproduce issue by inserting large number of routes on your local machine. Reproducible:
# cat add-routes.sh
#!/bin/bash
for a in $(seq 190 195); do
for b in $(seq 1 254); do
for c in $(seq 1 254); do
ip route add $a.$b.$c.0/24 via 192.168.124.100 dev eth0
done
done
done
# sh add-routes.sh
<runs for about 10 minutes on my machine>
Then try to ssh from another machine.
The segfault was fixed upstream: https://github.com/systemd/systemd/commit/82e4eda664d40ef60829e27d84b1610c2f4070cd pushed to staging -> https://github.com/lnykryn/systemd-rhel/commit/164a98ea6b24fea3433516dcc0df496929674cdd -> post Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2216.html |
Description of problem: When connecting from client (OSX 10.9.5) to EL7 7.2.1511 (Core) sshd seg faults. Version-Release number of selected component (if applicable): openssh 6.6.1p1 25.el7_2 How reproducible: Every time when UseDNS is set to yes in sshd_config. Steps to Reproduce: ssh to server connecting client has no PTR record Actual results: Connection closed by server Expected results: shell Additional info: tail of sshd -ddd output: .... debug3: mm_request_receive_expect entering: type 121 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 120 debug3: mm_request_send entering: type 121 debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 [preauth] debug3: mm_request_send entering: type 120 [preauth] debug3: mm_request_receive_expect entering: type 121 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 120 debug3: mm_request_send entering: type 121 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth] debug3: mm_request_send entering: type 0 [preauth] debug3: mm_choose_dh: waiting for MONITOR_ANS_MODULI [preauth] debug3: mm_request_receive_expect entering: type 1 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 0 debug3: mm_answer_moduli: got parameters: 1024 1024 8192 debug3: mm_request_send entering: type 1 debug2: monitor_read: 0 used once, disabling now debug3: mm_choose_dh: remaining 0 [preauth] debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth] debug2: bits set: 508/1024 [preauth] debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth] debug2: bits set: 526/1024 [preauth] debug3: mm_key_sign entering [preauth] debug3: mm_request_send entering: type 6 [preauth] debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth] debug3: mm_request_receive_expect entering: type 7 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 6 debug3: mm_answer_sign debug3: mm_answer_sign: signature 0x7fa86fe824c0(271) debug3: mm_request_send entering: type 7 debug2: monitor_read: 6 used once, disabling now debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent [preauth] debug2: kex_derive_keys [preauth] debug2: set_newkeys: mode 1 [preauth] debug1: SSH2_MSG_NEWKEYS sent [preauth] debug1: expecting SSH2_MSG_NEWKEYS [preauth] debug2: set_newkeys: mode 0 [preauth] debug1: SSH2_MSG_NEWKEYS received [preauth] debug1: KEX done [preauth] debug1: userauth-request for user root service ssh-connection method none [preauth] debug1: attempt 0 failures 0 [preauth] debug3: mm_getpwnamallow entering [preauth] debug3: mm_request_send entering: type 8 [preauth] debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth] debug3: mm_request_receive_expect entering: type 9 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 8 debug3: mm_answer_pwnamallow debug3: Trying to reverse map address 91.X.X.8. mm_log_handler: write: Broken pipe Segmentation fault [root@x1]# tail of sshd -ddd strace: (whole strace can be produced upon request) 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131731000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131730663}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 7500 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"<\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\2\25\0\0\376\v\0\1\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 7500 4201 brk(0) = 0x7fa89742c000 4201 brk(0x7fa89744d000) = 0x7fa89744d000 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131623000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131622669}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 7500 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"<\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\2\30\0\0\376\v\0\1\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 7500 4201 brk(0) = 0x7fa89744d000 4201 brk(0x7fa89746e000) = 0x7fa89746e000 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131515000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131514667}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 7500 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"<\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\2\30\0\0\376\v\0\1\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 7500 4201 brk(0) = 0x7fa89746e000 4201 brk(0x7fa89748f000) = 0x7fa89748f000 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131407000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131406675}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 7500 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"<\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\2\22\0\0\376\v\0\1\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 7500 4201 brk(0) = 0x7fa89748f000 4201 brk(0x7fa8974b0000) = 0x7fa8974b0000 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131297000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131296661}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 7500 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"<\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\2\26\0\0\376\v\0\1\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 7500 4201 brk(0) = 0x7fa8974b0000 4201 brk(0x7fa8974d1000) = 0x7fa8974d1000 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131189000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131188685}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 7500 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"<\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\2\24\0\0\376\v\0\1\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 7500 4201 brk(0) = 0x7fa8974d1000 4201 brk(0x7fa8974f2000) = 0x7fa8974f2000 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131077000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131076704}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 900 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"<\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\2 \0\0\377\2\376\2\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 900 4201 brk(0) = 0x7fa8974f2000 4201 brk(0x7fa897513000) = 0x7fa897513000 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 131006000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 131005739}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 6008 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"l\0\0\0\30\0\2\0\1\0\0\0i\20\0\0\n`\0\0\376\3\0\7\0\0\0\0\10\0\17\0"..., 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 6008 4201 ppoll([{fd=4, events=POLLIN}], 1, {24, 130945000}, NULL, 8) = 1 ([{fd=4, revents=POLLIN}], left {24, 130944711}) 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{NULL, 0}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 20 4201 recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\1\0\0\0i\20\0\0\0\0\0\0", 7544}], msg_controllen=56, {cmsg_len=20, cmsg_level=0x10e /* SOL_??? */, cmsg_type=, ...}, msg_flags=0}, MSG_TRUNC) = 20 4201 close(4) = 0 4201 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7ffd6f13cff8} --- 4201 +++ killed by SIGSEGV +++ 4202 <... read resumed> "", 4) = 0 4202 write(7, "\0\0\0\22\0\0\0\5\0\0\0\ndo_cleanup", 22) = -1 EPIPE (Broken pipe) 4202 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=4202, si_uid=74} --- 4202 write(2, "mm_log_handler: write: Broken pi"..., 36) = 36 4202 exit_group(255) = ? 4202 +++ exited with 255 +++