Description of problem: Trying to ssh into a Fedora 25 system via public key authentication fails with 'Authentication failed' in approximately 1 out ~10 times. When those failures occur the journal reports segfaults of the sshd process, e.g.: Jan 01 21:06:20 example.org audit[12593]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 pid=12593 comm="sshd" exe="/usr/sbin/sshd" sig=11 Jan 01 21:06:20 example.org kernel: sshd[12593]: segfault at 556149000fc8 ip 00007ff3fe9ddfa8 sp 00007ffe0335dc90 error 4 in libc-2.24.so[7ff3fe95d000+1bd000] Or: Jan 01 21:20:16 example.org kernel: sshd[13790]: segfault at 55a0ae00efe8 ip 00007fac81cecfa8 sp 00007ffe4640f9c0 error 4 in libc-2.24.so[7fac81c6c000+1bd000] Or: Jan 01 21:24:02 example.org audit[14120]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 pid=14120 comm="sshd" exe="/usr/sbin/sshd" sig=11 Jan 01 21:24:02 example.org kernel: traps: sshd[14120] general protection ip:7f0e9976afd1 sp:7ffc215dec50 error:0 in libc-2.24.so[7f0e996ea000+1bd000] Or: Jan 01 21:26:34 example.org kernel: traps: sshd[16383] general protection ip:7fda32987fd1 sp:7ffd7296d950 error:0 in libc-2.24.so[7fda32907000+1bd000] Version-Release number of selected component (if applicable): openssh-7.3p1-7.fc25.x86_64 glibc-2.24-4.fc25.x86_64 How reproducible: - on average: 1 out of ~10 ssh logins - 100% when running the ssh command in a loop with 100 iterations Steps to Reproduce: 1. on the server: add the client's ed25519 public key to the authorized_keys file 2. make sure that sshd is running 3. on the client: for i in $(seq 100); do echo $i; if ssh juser echo ; then : ; else break; fi; done Actual results: on the client: 1 2 3 4 ... Authentication failed. on the server: segfault of sshd process, see messages above Expected results: on the client: 1 2 3 4 ... 100 on the server: no segfaults Additional info: Client info: - aarch64 system - openssh 7.4.p1 - public key is a ed25519 one - generated via: `ssh-keygen -t ed25519` - client connects over IPv6 to the Fedora 25 sshd ABRT doesn't collect core files for the sshd segfaults - don't if this a bug or features - any ideas how to enable core file collection? (ABRT does collect core files for user processes)
Hello, thank you for the bug report. OpenSSH 7.4 is not in Fedora 25 so I am unable to reproduce your behavior since I don't have your packages (and I don't see this behavior with my builds). I plan to add OpenSSH 7.4 to Fedora 25 soon, so I would be glad for some feedback, as it will get into Updates testing. The reason why the core is not picked by abrt is most probably because you installed the OpenSSH from sources or from unsigned RPM (not official Fedora). I was trying to find respective documentation, but without success so far.
(In reply to Jakub Jelen from comment #1) > Hello, thank you for the bug report. OpenSSH 7.4 is not in Fedora 25 so I am > unable to reproduce your behavior since I don't have your packages (and I > don't see this behavior with my builds). You are misunderstanding my report. As I clearly stated in the report: I am just using openssh 7.4 on the client. On the Fedora 25 server, I am using the `sshd` from the official openssh-7.3p1-7.fc25.x86_64 - which is installed, by default. > I plan to add OpenSSH 7.4 to Fedora 25 soon, so I would be glad for some > feedback, as it will get into Updates testing. > The reason why the core is not picked by abrt is most probably because you > installed the OpenSSH from sources or from unsigned RPM (not official > Fedora). I was trying to find respective documentation, but without success > so far. No, this is NOT the reason. I have installed openssh from the official Fedora repository. The sshd is the one provided by the system. Installed by default. Started via standard means, i.e. via systemd during normal system startup. rpm shows that it is from openssh-7.3p1-7.fc25.x86_64. By the way, the system is a pretty standard Fedora Workstation install. It was freshly installed via the official network iso and is regularly updated (and restarted). Especially I haven't tinkered with any ABRT settings. Thus, the system should also run with default ABRT and SELinux settings.
Sorry, I was reading too fast. I see the crash too (after more tries) and can not reproduce it with Fedora rawhide. The gdb on the core reports nice backtrace going down to the PAM stack: (gdb) bt #0 0x00007fa5aad4ffa8 in malloc_consolidate () from /lib64/libc.so.6 #1 0x00007fa5aad52eaf in _int_malloc () from /lib64/libc.so.6 #2 0x00007fa5aad54f14 in malloc () from /lib64/libc.so.6 #3 0x00007fa5aad3d4b2 in __GI__IO_file_doallocate () from /lib64/libc.so.6 #4 0x00007fa5aad4c9a6 in __GI__IO_doallocbuf () from /lib64/libc.so.6 #5 0x00007fa5aad4b894 in __GI__IO_file_underflow () from /lib64/libc.so.6 #6 0x00007fa5aad3eef8 in getdelim () from /lib64/libc.so.6 #7 0x00007fa5a83a9b94 in search_key.constprop () from /usr/lib64/security/pam_unix.so #8 0x00007fa5a83a9d9d in _set_ctrl () from /usr/lib64/security/pam_unix.so #9 0x00007fa5a83a7bc3 in pam_sm_acct_mgmt () from /usr/lib64/security/pam_unix.so #10 0x00007fa5ad15aefa in _pam_dispatch () from /lib64/libpam.so.0 #11 0x000055fc2e031a3e in do_pam_account () at auth-pam.c:942 #12 0x000055fc2e027d44 in mm_answer_pam_account (sock=6, m=0x7ffe86eb2750) at monitor.c:1167 #13 0x000055fc2e029038 in monitor_child_preauth (_authctxt=<optimized out>, pmonitor=0x55fc2fdaa990) at monitor.c:460 #14 0x000055fc2e009fd3 in privsep_preauth (authctxt=0x55fc2fdaa5a0) at sshd.c:759 #15 main (ac=<optimized out>, av=<optimized out>) at sshd.c:2408 I will have a look into that.
With more debuginfo for PAM shows more information, but nothing useful. The failure comes from getline() called from search_key() under PAM, which seems unrelated and smells of mangled memory from earlier. Running the same tests with RSA keys does not show this segfault, so something around ed25519 code is probably somehow mangling the heap. Trying to find what is going on with valgrind I found one bad memory handling, which might have cause this error in openssh-6.6p1-role-mls.patch (use-after-free), which was introduced with the latest rebase: [...] free(p); } [...] cp = buffer_get_cstring(&b, NULL); if ((r = strchr(p, '/')) != NULL) /* should be cp instead of p */ calling strchr() on wrong (already freed) buffer. Giving a shot with a build having this fixed does not show this behavior. I will issue the updates soon. Can you verify the fix is solving the problem for you with this scratch build? https://koji.fedoraproject.org/koji/taskinfo?taskID=17148652
Sorry, Fedora 25 build will be here: https://koji.fedoraproject.org/koji/taskinfo?taskID=17148690
(In reply to Jakub Jelen from comment #5) > Sorry, Fedora 25 build will be here: > https://koji.fedoraproject.org/koji/taskinfo?taskID=17148690 I followed the link to: https://koji.fedoraproject.org/koji/taskinfo?taskID=17148692 Then downloaded the packages https://kojipkgs.fedoraproject.org//work/tasks/8692/17148692/openssh-server-7.3p1-8.fc25.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/8692/17148692/openssh-7.3p1-8.fc25.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/8692/17148692/openssh-clients-7.3p1-8.fc25.x86_64.rpm and did a local install: dnf install /home/juser/openssh-7.3p1-8.fc25.x86_64.rpm /home/juser/openssh-clients-7.3p1-8.fc25.x86_64.rpm /home/juser/openssh-server-7.3p1-8.fc25.x86_64.rpm (btw, is there a simpler way?) And with that version this issue is resolved for me. That means that the client system can now login via ssh with ed25519 public key auth many many times (in a loop) without producing a segfault anymore.
(In reply to Georg Sauthoff from comment #6) > (btw, is there a simpler way?) If you have fedora-review installed, you should be able to $ koji-download-scratch 17148690 (but it is somehow broken at the moment for me) and then install from that directory $ dnf update *.rpm > And with that version this issue is resolved for me. > > That means that the client system can now login via ssh with ed25519 public > key auth many many times (in a loop) without producing a segfault anymore. Thanks for verification. I will update soon.
openssh-7.4p1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4767e2991d
openssh-7.4p1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-4767e2991d
openssh-7.4p1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.