Bug 208102

Summary: openssh client segmentation fault
Product: [Fedora] Fedora Reporter: jbernstein
Component: opensshAssignee: Tomas Mraz <tmraz>
Status: CLOSED CURRENTRELEASE QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 5CC: bojan, nalin, pawsa, security-response-team, triage
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard: bzcl34nup
Fixed In Version: F8 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-04-04 05:37:40 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Attachments:
Description Flags
backtrace results
none
Testing srpm which disables spnego patch
none
gdb output for ssh debug
none
Testing srpm with fixed spnego patch
none
Fixed krb5 src rpm none

Description jbernstein 2006-09-26 10:03:21 EDT
Description of problem:
OpenSSH connection from Fedora Core 5 client to RHEL4 server fails with a
segmentation fault.  Connections work to devices other then RHEL4.  Connection
also works to RHEL4 if it is forced to use SSH v1.

This function worked properly until last week.  No recent known changes to RHEL4
server.

Version-Release number of selected component (if applicable):
openssh-clients-4.3p2-4.1

How reproducible:
Problem is consistant to the same RHEL4 servers, and does not appear when
connecting to other devices.

Steps to Reproduce:
1. ssh to RHEL4 server
2. segmentation fault
3. repeat
  
Actual results:
segmentation fault

Expected results:
successful connection

Additional info:
See attached backtrace file
Comment 1 jbernstein 2006-09-26 10:03:22 EDT
Created attachment 137138 [details]
backtrace results
Comment 2 Tomas Mraz 2006-09-26 10:30:27 EDT
Could you try to install openssh-debuginfo package and run the ssh client from
gdb? This would generate even better backtrace file.
Comment 3 Tomas Mraz 2006-09-26 11:08:55 EDT
Created attachment 137146 [details]
Testing srpm which disables spnego patch

Could you also try to rebuild and test this srpm whether it helps?
Comment 4 jbernstein 2006-09-26 15:20:02 EDT
Created attachment 137164 [details]
gdb output for ssh debug
Comment 5 jbernstein 2006-09-26 15:58:06 EDT
I rebuilt the src rpm that you sent, and it works now.  I'm interested in any
further info that you develop.
Comment 6 Tomas Mraz 2006-09-27 06:09:20 EDT
Created attachment 137207 [details]
Testing srpm with fixed spnego patch

Can you try this srpm. It contains fix in spnego patch.
Comment 7 Tomas Mraz 2006-10-06 04:00:35 EDT
*** Bug 209529 has been marked as a duplicate of this bug. ***
Comment 8 Tomas Mraz 2006-10-06 04:06:11 EDT
I cannot reproduce this crash with MIT KDC here, but I'll continue with
investigation.

Could you also try stracing the ssh client when it's crashing?

Also removing the Security keyword because it is just a simple client crash
without exploit possible.
Comment 9 Pawel Salek 2006-10-06 06:27:14 EDT
I got this one when recompiling the code:
gss-serv-krb5.c: In function 'ssh_gssapi_krb5_storecreds':
gss-serv-krb5.c:156: warning: implicit declaration of function
'gss_krb5_copy_ccache'
and the double free/corruption is still there.
Comment 10 Pawel Salek 2006-10-06 06:28:57 EDT
When running ssh under gdb I get segfault with following stack trace:
Program received signal SIGSEGV, Segmentation fault.
0x00231f35 in free () from /lib/libc.so.6
(gdb) where
#0  0x00231f35 in free () from /lib/libc.so.6
#1  0x001961ef in krb5_free_addresses () from /usr/lib/libkrb5.so.3
#2  0x00181908 in krb5_get_notification_message () from /usr/lib/libkrb5.so.3
#3  0x00184bf7 in krb5_cc_next_cred () from /usr/lib/libkrb5.so.3
#4  0x0017edc6 in krb5int_cc_creds_match_request () from /usr/lib/libkrb5.so.3
#5  0x0017f01e in krb5_cc_retrieve_cred_default () from /usr/lib/libkrb5.so.3
#6  0x001822d9 in krb5_get_notification_message () from /usr/lib/libkrb5.so.3
#7  0x00184b6e in krb5_cc_retrieve_cred () from /usr/lib/libkrb5.so.3
#8  0x00191319 in krb5_get_credentials () from /usr/lib/libkrb5.so.3
#9  0x0068a139 in krb5_gss_init_sec_context ()
   from /usr/lib/libgssapi_krb5.so.2
#10 0x0068e731 in gss_init_sec_context () from /usr/lib/libgssapi_krb5.so.2
#11 0x005e2aa3 in ssh_gssapi_init_ctx (ctx=0x81cd090, deleg_creds=0,
    recv_tok=0x0, send_tok=0xbfc0b640, flags=0x0) at gss-genr.c:194
#12 0x005e2c7c in ssh_gssapi_check_mechanism (ctx=0xbfc0b678, oid=0x81cd0d0,
    host=0x81cd490 "detroit") at gss-genr.c:304
#13 0x005c4dad in userauth_gssapi (authctxt=0xbfc0b764) at sshconnect2.c:498
#14 0x005c4f5d in userauth (authctxt=0xbfc0b764, authlist=Variable "authlist" is
not available.
)
    at sshconnect2.c:341
#15 0x005c562a in input_userauth_failure (type=51, seq=5, ctxt=0xbfc0b764)
    at sshconnect2.c:407
#16 0x005dc98b in dispatch_run (mode=0, done=0xbfc0b778, ctxt=0xbfc0b764)
    at dispatch.c:93
Comment 11 Tomas Mraz 2006-10-06 06:53:59 EDT
Thanks for the debugging. Could you please also install krb5-debuginfo package
and attach the backtrace with that installed?
Comment 12 jbernstein 2006-10-06 09:39:27 EDT
Sorry, I was away for a few days.  What further info do you need from me? 
Should I still try the attached SRPM from 27 Sept?
Comment 13 Tomas Mraz 2006-10-06 11:47:48 EDT
Not necessary as it probably doesn't resolve the crash. I still cannot reproduce
it here. What kind of KDC do you use?
Comment 14 jbernstein 2006-10-06 15:57:30 EDT
microsoft active directory
Comment 15 Tomas Mraz 2006-10-06 16:10:59 EDT
CCing nalin as krb5 package owner.
Comment 16 Pawel Salek 2006-10-06 21:18:49 EDT
(gdb) bt
#0  0x002d4f35 in free () from /lib/libc.so.6
#1  0x00f461ef in krb5_free_addresses (context=0x8808190, val=0x880a0e8)
    at kfree.c:46
#2  0x00f31908 in krb5_fcc_next_cred (context=0x8808190, id=0x880a718, 
    cursor=0xbfb3ea18, creds=0xbfb3e9c0) at cc_file.c:551
#3  0x00f34bf7 in krb5_cc_next_cred (context=0x8808190, cache=0x880a718, 
    cursor=0xbfb3ea18, creds=0xbfb3e9c0) at ccfns.c:97
#4  0x00f2edc6 in krb5_cc_retrieve_cred_seq (context=0x8808190, id=0x880a718, 
    whichfields=545, mcreds=0xbfb3eae0, creds=0x880a248, nktypes=7, 
    ktypes=0x880a348) at cc_retr.c:215
#5  0x00f2f01e in krb5_cc_retrieve_cred_default (context=0x8808190, 
    id=0x880a718, flags=545, mcreds=0xbfb3eae0, creds=0x880a248) at cc_retr.c:266
#6  0x00f322d9 in krb5_fcc_retrieve (context=0x8808190, id=0x880a718, 
    whichfields=545, mcreds=0xbfb3eae0, creds=0x880a248) at cc_file.c:2102
#7  0x00f34b6e in krb5_cc_retrieve_cred (context=0x8808190, cache=0x880a718, 
    flags=545, mcreds=0xbfb3eae0, creds=0x880a248) at ccfns.c:76
#8  0x00f41319 in krb5_get_credentials (context=0x8808190, options=Variable "options" is not 
available.
)
    at get_creds.c:131
#9  0x00d5d139 in krb5_gss_init_sec_context (minor_status=0x8808094, 
    claimant_cred_handle=0x0, context_handle=0x8808098, target_name=0x88099d0, 
    mech_type=0x88080e8, req_flags=306, time_req=0, input_chan_bindings=0x0, 
    input_token=0x0, actual_mech_type=0x0, output_token=0xbfb3eda0, 
    ret_flags=0x0, time_rec=0x0) at init_sec_context.c:116
#10 0x00d61731 in gss_init_sec_context (minor_status=0x8808094, 
    claimant_cred_handle=0x0, context_handle=0x8808098, target_name=0x88099d0, 
    mech_type=0x88080e8, req_flags=34, time_req=0, input_chan_bindings=0x0, 
    input_token=0x0, actual_mech_type=0x0, output_token=0xbfb3eda0, 
    ret_flags=0x0, time_rec=0x0) at krb5_gss_glue.c:263
#11 0x004cdaa3 in ssh_gssapi_init_ctx (ctx=0x8808090, deleg_creds=0, 
    recv_tok=0x0, send_tok=0xbfb3eda0, flags=0x0) at gss-genr.c:194
(gdb) up
#1  0x00f461ef in krb5_free_addresses (context=0x8808190, val=0x880a0e8)
    at kfree.c:46
46                  krb5_xfree((*temp)->contents);
(gdb) print temp
$1 = (krb5_address **) 0x880a0e8
(gdb) print *temp
$2 = (krb5_address *) 0x880a320
(gdb) print (*temp)->contents
$3 = (krb5_octet *) 0xae79e008 <Address 0xae79e008 out of bounds>
Comment 17 Tomas Mraz 2006-10-11 14:49:01 EDT
Created attachment 138269 [details]
Fixed krb5 src rpm

So the doublefree is in krb5 code. Pawel or jbernstein can you please recompile
and test the attached krb5 src.rpm?
Comment 18 Pawel Salek 2006-10-13 05:58:17 EDT
The backtrace is different now:

debug1: Next authentication method: gssapi-with-mic

Program received signal SIGSEGV, Segmentation fault.
0x00737bc1 in _int_malloc () from /lib/libc.so.6
(gdb) where
#0  0x00737bc1 in _int_malloc () from /lib/libc.so.6
#1  0x007397f4 in malloc () from /lib/libc.so.6
#2  0x00c9a3f0 in krb5_fcc_read_data (context=0x96e1188, id=0x96e3590, 
data=0x96e32ac) at cc_file.c:627
#3  0x00c9a584 in krb5_fcc_read_principal (context=0x96e1188, 
id=0x96e3590, princ=0xbfe66a58) at cc_file.c:491
#4  0x00c9c1b5 in krb5_fcc_start_seq_get (context=0x96e1188, 
id=0x96e3590, cursor=0xbfe66b88) at cc_file.c:1398
#5  0x00c9ebc0 in krb5_cc_start_seq_get (context=0x96e1188, 
cache=0x96e3590, cursor=0xbfe66b88) at ccfns.c:90
#6  0x00c98d71 in krb5_cc_retrieve_cred_seq (context=0x96e1188, 
id=0x96e3590, whichfields=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c, 
    nktypes=7, ktypes=0x96e3340) at cc_retr.c:211
#7  0x00c9901e in krb5_cc_retrieve_cred_default (context=0x96e1188, 
id=0x96e3590, flags=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c)
    at cc_retr.c:266
#8  0x00c9c2d9 in krb5_fcc_retrieve (context=0x96e1188, id=0x96e3590, 
whichfields=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c)
    at cc_file.c:2104
#9  0x00c9eb6e in krb5_cc_retrieve_cred (context=0x96e1188, 
cache=0x96e3590, flags=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c)
    at ccfns.c:76
#10 0x00ca9bfb in krb5_get_cred_from_kdc_opt (context=0x96e1188, 
ccache=0x96e3590, in_cred=0xbfe66eec, out_cred=0xbfe66fd8, 
    tgts=0xbfe66e48, kdcopt=0) at gc_frm_kdc.c:125
#11 0x00cab389 in krb5_get_credentials (context=0x96e1188, 
options=Variable "options" is not available.
) at get_creds.c:148
#12 0x00d22f19 in krb5_gss_init_sec_context (minor_status=0x96e1094, 
claimant_cred_handle=0x0, context_handle=0x96e1098, 
    target_name=0x96e29c8, mech_type=0x96e10e8, req_flags=306, 
time_req=0, input_chan_bindings=0x0, input_token=0x0, 
    actual_mech_type=0x0, output_token=0xbfe670b0, ret_flags=0x0, 
time_rec=0x0) at init_sec_context.c:116
#13 0x00d27511 in gss_init_sec_context (minor_status=0x96e1094, 
claimant_cred_handle=0x0, context_handle=0x96e1098, 
    target_name=0x96e29c8, mech_type=0x96e10e8, req_flags=34, 
time_req=0, input_chan_bindings=0x0, input_token=0x0, 
    actual_mech_type=0x0, output_token=0xbfe670b0, ret_flags=0x0, 
time_rec=0x0) at krb5_gss_glue.c:263
#14 0x00664aa3 in ssh_gssapi_init_ctx (ctx=0x96e1090, deleg_creds=0, 
recv_tok=0x0, send_tok=0xbfe670b0, flags=0x0) at gss-genr.c:194

krb5_fcc_read_data is pretty strange, particularly this code:
618          data->length = len;
619          if (data->length != len || data->length + 1 == 0)
620              return KRB5_CC_NOMEM;
The first condition always fails.
When I step through krb5_fcc_read_data, the first call to this routine succeeds, so does second, etc, 
only 26th call to this routine causes segfault.
Comment 19 Tomas Mraz 2006-10-16 11:10:27 EDT
The code in the previous comment is harmless.

I'm still investigating but I can't find anything suspicious in
krb5_fcc_read_data or krb5_fcc_read_principal. Without a reproducer I'm afraid
that's all I can do for now. If I got an access to a system where the ssh client
crashes I could try more.
Comment 20 Tomas Mraz 2007-09-24 15:17:05 EDT
Can you still reproduce the problem with current openssh and krb5 packages from
Fedora development?
Comment 21 Pawel Salek 2007-09-24 16:05:54 EDT
Myself, I haven't seen that recently. As a matter of fact, I haven't seen that since fc6.
Comment 22 Bug Zapper 2008-04-03 23:51:23 EDT
Fedora apologizes that these issues have not been resolved yet. We're
sorry it's taken so long for your bug to be properly triaged and acted
on. We appreciate the time you took to report this issue and want to
make sure no important bugs slip through the cracks.

If you're currently running a version of Fedora Core between 1 and 6,
please note that Fedora no longer maintains these releases. We strongly
encourage you to upgrade to a current Fedora release. In order to
refocus our efforts as a project we are flagging all of the open bugs
for releases which are no longer maintained and closing them.
http://fedoraproject.org/wiki/LifeCycle/EOL

If this bug is still open against Fedora Core 1 through 6, thirty days
from now, it will be closed 'WONTFIX'. If you can reporduce this bug in
the latest Fedora version, please change to the respective version. If
you are unable to do this, please add a comment to this bug requesting
the change.

Thanks for your help, and we apologize again that we haven't handled
these issues to this point.

The process we are following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp

We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.

And if you'd like to join the bug triage team to help make things
better, check out http://fedoraproject.org/wiki/BugZappers