Bug 208102

Created attachment 137138 [details]
backtrace results

Could you try to install openssh-debuginfo package and run the ssh client from
gdb? This would generate even better backtrace file.

Created attachment 137146 [details]
Testing srpm which disables spnego patch

Could you also try to rebuild and test this srpm whether it helps?

Created attachment 137164 [details]
gdb output for ssh debug

I rebuilt the src rpm that you sent, and it works now.  I'm interested in any
further info that you develop.

Created attachment 137207 [details]
Testing srpm with fixed spnego patch

Can you try this srpm. It contains fix in spnego patch.

*** Bug 209529 has been marked as a duplicate of this bug. ***

I cannot reproduce this crash with MIT KDC here, but I'll continue with
investigation.

Could you also try stracing the ssh client when it's crashing?

Also removing the Security keyword because it is just a simple client crash
without exploit possible.

I got this one when recompiling the code:
gss-serv-krb5.c: In function 'ssh_gssapi_krb5_storecreds':
gss-serv-krb5.c:156: warning: implicit declaration of function
'gss_krb5_copy_ccache'
and the double free/corruption is still there.

When running ssh under gdb I get segfault with following stack trace:
Program received signal SIGSEGV, Segmentation fault.
0x00231f35 in free () from /lib/libc.so.6
(gdb) where
#0  0x00231f35 in free () from /lib/libc.so.6
#1  0x001961ef in krb5_free_addresses () from /usr/lib/libkrb5.so.3
#2  0x00181908 in krb5_get_notification_message () from /usr/lib/libkrb5.so.3
#3  0x00184bf7 in krb5_cc_next_cred () from /usr/lib/libkrb5.so.3
#4  0x0017edc6 in krb5int_cc_creds_match_request () from /usr/lib/libkrb5.so.3
#5  0x0017f01e in krb5_cc_retrieve_cred_default () from /usr/lib/libkrb5.so.3
#6  0x001822d9 in krb5_get_notification_message () from /usr/lib/libkrb5.so.3
#7  0x00184b6e in krb5_cc_retrieve_cred () from /usr/lib/libkrb5.so.3
#8  0x00191319 in krb5_get_credentials () from /usr/lib/libkrb5.so.3
#9  0x0068a139 in krb5_gss_init_sec_context ()
   from /usr/lib/libgssapi_krb5.so.2
#10 0x0068e731 in gss_init_sec_context () from /usr/lib/libgssapi_krb5.so.2
#11 0x005e2aa3 in ssh_gssapi_init_ctx (ctx=0x81cd090, deleg_creds=0,
    recv_tok=0x0, send_tok=0xbfc0b640, flags=0x0) at gss-genr.c:194
#12 0x005e2c7c in ssh_gssapi_check_mechanism (ctx=0xbfc0b678, oid=0x81cd0d0,
    host=0x81cd490 "detroit") at gss-genr.c:304
#13 0x005c4dad in userauth_gssapi (authctxt=0xbfc0b764) at sshconnect2.c:498
#14 0x005c4f5d in userauth (authctxt=0xbfc0b764, authlist=Variable "authlist" is
not available.
)
    at sshconnect2.c:341
#15 0x005c562a in input_userauth_failure (type=51, seq=5, ctxt=0xbfc0b764)
    at sshconnect2.c:407
#16 0x005dc98b in dispatch_run (mode=0, done=0xbfc0b778, ctxt=0xbfc0b764)
    at dispatch.c:93

Thanks for the debugging. Could you please also install krb5-debuginfo package
and attach the backtrace with that installed?

Sorry, I was away for a few days.  What further info do you need from me? 
Should I still try the attached SRPM from 27 Sept?

Not necessary as it probably doesn't resolve the crash. I still cannot reproduce
it here. What kind of KDC do you use?

microsoft active directory

CCing nalin as krb5 package owner.

(gdb) bt
#0  0x002d4f35 in free () from /lib/libc.so.6
#1  0x00f461ef in krb5_free_addresses (context=0x8808190, val=0x880a0e8)
    at kfree.c:46
#2  0x00f31908 in krb5_fcc_next_cred (context=0x8808190, id=0x880a718, 
    cursor=0xbfb3ea18, creds=0xbfb3e9c0) at cc_file.c:551
#3  0x00f34bf7 in krb5_cc_next_cred (context=0x8808190, cache=0x880a718, 
    cursor=0xbfb3ea18, creds=0xbfb3e9c0) at ccfns.c:97
#4  0x00f2edc6 in krb5_cc_retrieve_cred_seq (context=0x8808190, id=0x880a718, 
    whichfields=545, mcreds=0xbfb3eae0, creds=0x880a248, nktypes=7, 
    ktypes=0x880a348) at cc_retr.c:215
#5  0x00f2f01e in krb5_cc_retrieve_cred_default (context=0x8808190, 
    id=0x880a718, flags=545, mcreds=0xbfb3eae0, creds=0x880a248) at cc_retr.c:266
#6  0x00f322d9 in krb5_fcc_retrieve (context=0x8808190, id=0x880a718, 
    whichfields=545, mcreds=0xbfb3eae0, creds=0x880a248) at cc_file.c:2102
#7  0x00f34b6e in krb5_cc_retrieve_cred (context=0x8808190, cache=0x880a718, 
    flags=545, mcreds=0xbfb3eae0, creds=0x880a248) at ccfns.c:76
#8  0x00f41319 in krb5_get_credentials (context=0x8808190, options=Variable "options" is not 
available.
)
    at get_creds.c:131
#9  0x00d5d139 in krb5_gss_init_sec_context (minor_status=0x8808094, 
    claimant_cred_handle=0x0, context_handle=0x8808098, target_name=0x88099d0, 
    mech_type=0x88080e8, req_flags=306, time_req=0, input_chan_bindings=0x0, 
    input_token=0x0, actual_mech_type=0x0, output_token=0xbfb3eda0, 
    ret_flags=0x0, time_rec=0x0) at init_sec_context.c:116
#10 0x00d61731 in gss_init_sec_context (minor_status=0x8808094, 
    claimant_cred_handle=0x0, context_handle=0x8808098, target_name=0x88099d0, 
    mech_type=0x88080e8, req_flags=34, time_req=0, input_chan_bindings=0x0, 
    input_token=0x0, actual_mech_type=0x0, output_token=0xbfb3eda0, 
    ret_flags=0x0, time_rec=0x0) at krb5_gss_glue.c:263
#11 0x004cdaa3 in ssh_gssapi_init_ctx (ctx=0x8808090, deleg_creds=0, 
    recv_tok=0x0, send_tok=0xbfb3eda0, flags=0x0) at gss-genr.c:194
(gdb) up
#1  0x00f461ef in krb5_free_addresses (context=0x8808190, val=0x880a0e8)
    at kfree.c:46
46                  krb5_xfree((*temp)->contents);
(gdb) print temp
$1 = (krb5_address **) 0x880a0e8
(gdb) print *temp
$2 = (krb5_address *) 0x880a320
(gdb) print (*temp)->contents
$3 = (krb5_octet *) 0xae79e008 <Address 0xae79e008 out of bounds>

Created attachment 138269 [details]
Fixed krb5 src rpm

So the doublefree is in krb5 code. Pawel or jbernstein can you please recompile
and test the attached krb5 src.rpm?

The backtrace is different now:

debug1: Next authentication method: gssapi-with-mic

Program received signal SIGSEGV, Segmentation fault.
0x00737bc1 in _int_malloc () from /lib/libc.so.6
(gdb) where
#0  0x00737bc1 in _int_malloc () from /lib/libc.so.6
#1  0x007397f4 in malloc () from /lib/libc.so.6
#2  0x00c9a3f0 in krb5_fcc_read_data (context=0x96e1188, id=0x96e3590, 
data=0x96e32ac) at cc_file.c:627
#3  0x00c9a584 in krb5_fcc_read_principal (context=0x96e1188, 
id=0x96e3590, princ=0xbfe66a58) at cc_file.c:491
#4  0x00c9c1b5 in krb5_fcc_start_seq_get (context=0x96e1188, 
id=0x96e3590, cursor=0xbfe66b88) at cc_file.c:1398
#5  0x00c9ebc0 in krb5_cc_start_seq_get (context=0x96e1188, 
cache=0x96e3590, cursor=0xbfe66b88) at ccfns.c:90
#6  0x00c98d71 in krb5_cc_retrieve_cred_seq (context=0x96e1188, 
id=0x96e3590, whichfields=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c, 
    nktypes=7, ktypes=0x96e3340) at cc_retr.c:211
#7  0x00c9901e in krb5_cc_retrieve_cred_default (context=0x96e1188, 
id=0x96e3590, flags=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c)
    at cc_retr.c:266
#8  0x00c9c2d9 in krb5_fcc_retrieve (context=0x96e1188, id=0x96e3590, 
whichfields=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c)
    at cc_file.c:2104
#9  0x00c9eb6e in krb5_cc_retrieve_cred (context=0x96e1188, 
cache=0x96e3590, flags=576, mcreds=0xbfe66cf8, creds=0xbfe66d4c)
    at ccfns.c:76
#10 0x00ca9bfb in krb5_get_cred_from_kdc_opt (context=0x96e1188, 
ccache=0x96e3590, in_cred=0xbfe66eec, out_cred=0xbfe66fd8, 
    tgts=0xbfe66e48, kdcopt=0) at gc_frm_kdc.c:125
#11 0x00cab389 in krb5_get_credentials (context=0x96e1188, 
options=Variable "options" is not available.
) at get_creds.c:148
#12 0x00d22f19 in krb5_gss_init_sec_context (minor_status=0x96e1094, 
claimant_cred_handle=0x0, context_handle=0x96e1098, 
    target_name=0x96e29c8, mech_type=0x96e10e8, req_flags=306, 
time_req=0, input_chan_bindings=0x0, input_token=0x0, 
    actual_mech_type=0x0, output_token=0xbfe670b0, ret_flags=0x0, 
time_rec=0x0) at init_sec_context.c:116
#13 0x00d27511 in gss_init_sec_context (minor_status=0x96e1094, 
claimant_cred_handle=0x0, context_handle=0x96e1098, 
    target_name=0x96e29c8, mech_type=0x96e10e8, req_flags=34, 
time_req=0, input_chan_bindings=0x0, input_token=0x0, 
    actual_mech_type=0x0, output_token=0xbfe670b0, ret_flags=0x0, 
time_rec=0x0) at krb5_gss_glue.c:263
#14 0x00664aa3 in ssh_gssapi_init_ctx (ctx=0x96e1090, deleg_creds=0, 
recv_tok=0x0, send_tok=0xbfe670b0, flags=0x0) at gss-genr.c:194

krb5_fcc_read_data is pretty strange, particularly this code:
618          data->length = len;
619          if (data->length != len || data->length + 1 == 0)
620              return KRB5_CC_NOMEM;
The first condition always fails.
When I step through krb5_fcc_read_data, the first call to this routine succeeds, so does second, etc, 
only 26th call to this routine causes segfault.

The code in the previous comment is harmless.

I'm still investigating but I can't find anything suspicious in
krb5_fcc_read_data or krb5_fcc_read_principal. Without a reproducer I'm afraid
that's all I can do for now. If I got an access to a system where the ssh client
crashes I could try more.

Can you still reproduce the problem with current openssh and krb5 packages from
Fedora development?

Myself, I haven't seen that recently. As a matter of fact, I haven't seen that since fc6.

Fedora apologizes that these issues have not been resolved yet. We're
sorry it's taken so long for your bug to be properly triaged and acted
on. We appreciate the time you took to report this issue and want to
make sure no important bugs slip through the cracks.

If you're currently running a version of Fedora Core between 1 and 6,
please note that Fedora no longer maintains these releases. We strongly
encourage you to upgrade to a current Fedora release. In order to
refocus our efforts as a project we are flagging all of the open bugs
for releases which are no longer maintained and closing them.
http://fedoraproject.org/wiki/LifeCycle/EOL

If this bug is still open against Fedora Core 1 through 6, thirty days
from now, it will be closed 'WONTFIX'. If you can reporduce this bug in
the latest Fedora version, please change to the respective version. If
you are unable to do this, please add a comment to this bug requesting
the change.

Thanks for your help, and we apologize again that we haven't handled
these issues to this point.

The process we are following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp

We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.

And if you'd like to join the bug triage team to help make things
better, check out http://fedoraproject.org/wiki/BugZappers