Bug 1605756

Summary:	krb5 memory ccache cursors are invalidated by initialize, likely crashes
Product:	Red Hat Enterprise Linux 7	Reporter:	Norman Sardella <sardella>
Component:	krb5	Assignee:	Robbie Harwood <rharwood>
Status:	CLOSED CURRENTRELEASE	QA Contact:	BaseOS QE Security Team <qe-baseos-security>
Severity:	high	Docs Contact:	Mirek Jahoda <mjahoda>
Priority:	high
Version:	7.3	CC:	dpal, ebenes, lmanasko, mkosek, pkis, qe-baseos-security, rharwood, sardella, tscherf
Target Milestone:	rc	Keywords:	ZStream
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	krb5-1.15.1-35.el7	Doc Type:	Bug Fix
Doc Text:	.`krb5` memory caches are now thread-safe Previously, the memory caches of the Kerberos V5 login program (`krb5`) were not completely thread-safe. As a consequence, multi-threaded access terminated unexpectedly in some cases. With this update, the memory caches are cleaned up to be more thread-safe. As a result, no more crashes occur.	Story Points:	---
Clone Of:
Clones:	1657890 1671329 (view as bug list)		Environment:
Last Closed:	2019-08-06 15:50:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1647919, 1657890

Description Norman Sardella 2018-07-20 16:50:30 UTC

Description of problem:
I like Redhat to consider backporting the follow patch that applies cleanly
to RHEL7.3 krb5. This resolves crashes due to a dangling pointer


http://krbdev.mit.edu/rt/Ticket/Display.html?id=8202

 Tue Jun 16 23:05:45 2015  ghudson - Ticket created      
      Subject: memory ccache cursors are invalidated by initialize

Memory ccache objects contain a linked list of credentials.  The
iteration cursor is a pointer to one of the list elements.  If the
cache is initialized by one thread while another thread is iterating
over it, the second thread's cursor contains a dangling pointer and the
process will likely crash.

(This behavior can also be demonstrated in a single-threaded caller,
but the caller would have to be doing something obtuse.)
 



Version-Release number of selected component (if applicable):


How reproducible:
fix and test case can be found here

https://github.com/krb5/krb5/commit/146dadec8fe7ccc4149eb2e3f577cc320aee6efb
Author: Greg Hudson <ghudson>
Commit: 146dadec8fe7ccc4149eb2e3f577cc320aee6efb
Branch: master
 src/lib/krb5/ccache/cc_memory.c |  164 ++++++++++++++++++++++++--------------
 src/lib/krb5/ccache/t_cc.c      |   51 ++++++++++++
 2 files changed, 154 insertions(+), 61 deletions(-)


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Robbie Harwood 2018-07-20 17:48:15 UTC

Hi, under what circumstances are you seeing this crash?

Comment 3 Norman Sardella 2018-07-20 18:07:58 UTC

I don't have a repro of hand other that what described in the krbdev ticket

here is a crash I see without the fix in http://krbdev.mit.edu/rt/Ticket/Display.html?id=8202
https://github.com/krb5/krb5/pull/804
Greg Hudson can explain in more detail along with Robbie Harwood from Redhat that reviewed the commit



(gdb) bt                                                                                                                                                                                                                                                    
#0  0x00002af5e777aed5 in __memcpy_ssse3_back () from /lib64/libc.so.6                                                                                                                                                                                      
#1  0x00002af5db158949 in memcpy (__len=972, __src=<optimized out>, __dest=<optimized out>) at /usr/include/bits/string3.h:51                                                                                                                               
#2  krb5int_copy_data_contents (context=context@entry=0x2af6200e02c0, indata=indata@entry=0x2af7940ced30, outdata=outdata@entry=0x2af6200df850) at copy_data.c:73                                                                                           
#3  0x00002af5db15899d in krb5_copy_data (context=context@entry=0x2af6200e02c0, indata=indata@entry=0x2af7940ced30, outdata=outdata@entry=0x2af7f98fc0b0) at copy_data.c:51                                                                                 
#4  0x00002af5db15874c in k5_copy_creds_contents (context=0x2af6200e02c0, incred=0x2af7940cece0, tempcred=0x2af7f98fc170) at copy_creds.c:78                                                                                                                
#5  0x00002af5db14953f in krb5_mcc_next_cred (context=<optimized out>, id=<optimized out>, cursor=0x2af7f98fc168, creds=<optimized out>) at cc_memory.c:375                                                                                                 
#6  0x00002af5db145162 in krb5_cc_retrieve_cred_seq (context=context@entry=0x2af6200e02c0, id=id@entry=0x2af6200df100, whichfields=whichfields@entry=545, mcreds=mcreds@entry=0x2af7f98fc430, creds=creds@entry=0x2af6200df120, nktypes=6,                  
    ktypes=0x2af6200df210) at cc_retr.c:221                                                                                                                                                                                                                 
#7  0x00002af5db145593 in k5_cc_retrieve_cred_default (context=0x2af6200e02c0, id=0x2af6200df100, flags=545, mcreds=0x2af7f98fc430, creds=0x2af6200df120) at cc_retr.c:272                                                                                  
#8  0x00002af5db14c324 in krb5_cc_retrieve_cred (context=context@entry=0x2af6200e02c0, cache=cache@entry=0x2af6200df100, flags=flags@entry=545, mcreds=mcreds@entry=0x2af7f98fc430, creds=creds@entry=0x2af6200df120) at ccfns.c:120                        
#9  0x00002af5db15ce5f in cache_get (context=context@entry=0x2af6200e02c0, ccache=0x2af6200df100, flags=545, in_creds=in_creds@entry=0x2af7f98fc430, out_creds=out_creds@entry=0x2af6200df010) at get_creds.c:242                                           
#10 0x00002af5db15e550 in check_cache (ctx=0x2af6200deee0, context=0x2af6200e02c0) at get_creds.c:1042                                                                                                                                                      
#11 begin (ctx=0x2af6200deee0, context=0x2af6200e02c0) at get_creds.c:1065                                                                                                                                                                                  
#12 krb5_tkt_creds_step (context=context@entry=0x2af6200e02c0, ctx=ctx@entry=0x2af6200deee0, in=in@entry=0x2af7f98fc520, out=out@entry=0x2af7f98fc510, realm=realm@entry=0x2af7f98fc530, flags=flags@entry=0x2af7f98fc508) at get_creds.c:1246              
#13 0x00002af5db15ece7 in krb5_tkt_creds_get (context=context@entry=0x2af6200e02c0, ctx=0x2af6200deee0) at get_creds.c:1190                                                                                                                                 
#14 0x00002af5db15ee0c in krb5_get_credentials (context=context@entry=0x2af6200e02c0, options=options@entry=0, ccache=<optimized out>, in_creds=in_creds@entry=0x2af7f98fc680, out_creds=out_creds@entry=0x2af7f98fc678) at get_creds.c:1279                
#15 0x00002af5db42484c in get_credentials (server=<optimized out>, out_creds=<synthetic pointer>, endtime=<optimized out>, now=1524496366, cred=0x2af6200de9c0, context=0x2af6200e02c0) at init_sec_context.c:199                                           
#16 kg_new_connection (exts=0x2af7f98fc910, context=0x2af6200e02c0, time_rec=0x0, ret_flags=0x2af7f98fca9c, output_token=0x2af7f98fcab0, actual_mech_type=0x0, input_token=<optimized out>, input_chan_bindings=0x0, time_req=<optimized out>,              
    req_flags=58, mech_type=0x2af5db6435c0 <krb5_gss_oid_array>, target_name=<optimized out>, context_handle=0x2af6200e0480, cred=0x2af6200de9c0, minor_status=0x2af7f98fca94) at init_sec_context.c:589                                                    
#17 krb5_gss_init_sec_context_ext (minor_status=0x2af7f98fca94, claimant_cred_handle=0x2af6200de9c0, context_handle=0x2af6200e0480, target_name=<optimized out>, mech_type=0x2af5db6435c0 <krb5_gss_oid_array>, req_flags=58, time_req=time_req@entry=0,    
    input_chan_bindings=input_chan_bindings@entry=0x0, input_token=input_token@entry=0x0, actual_mech_type=actual_mech_type@entry=0x0, output_token=output_token@entry=0x2af7f98fcab0, ret_flags=ret_flags@entry=0x2af7f98fca9c,                            
    time_rec=time_rec@entry=0x0, exts=exts@entry=0x2af7f98fc910) at init_sec_context.c:1009                                                                                                                                                                 
#18 0x00002af5db425137 in krb5_gss_init_sec_context (minor_status=<optimized out>, claimant_cred_handle=<optimized out>, context_handle=<optimized out>, target_name=<optimized out>, mech_type=<optimized out>, req_flags=<optimized out>, time_req=0,     
    input_chan_bindings=0x0, input_token=0x0, actual_mech_type=0x0, output_token=0x2af7f98fcab0, ret_flags=0x2af7f98fca9c, time_rec=0x0) at init_sec_context.c:1111                                                                                         
#19 0x00002af5db40e10b in gss_init_sec_context (minor_status=0x2af7f98fca94, claimant_cred_handle=<optimized out>, context_handle=0x2af6200df3e8, target_name=0x2af6200dc680, req_mech_type=<optimized out>, req_flags=58, time_req=0,                      
    input_chan_bindings=0x0, input_token=0x0, actual_mech_type=0x0, output_token=0x2af7f98fcab0, ret_flags=0x2af7f98fca9c, time_rec=0x0) at g_init_sec_context.c:247
#20 0x00002af7fcd07d51 in gssapi_client_mech_step () from /usr/lib64/sasl2/libgssapiv2.so                                                                                                                                                                   
#21 0x00002af5e24b4835 in sasl_client_step () from /lib64/libsasl2.so.3                                                                                                                                                                                     
#22 0x00002af5e24b4bb6 in sasl_client_start () from /lib64/libsasl2.so.3                                                                                                                                                                                    
#23 0x00002af5dc0c14ab in ldap_int_sasl_bind () from /lib64/libldap_r-2.4.so.2                                                                                                                                                                              
#24 0x00002af5dc0c4ce7 in ldap_sasl_interactive_bind () from /lib64/libldap_r-2.4.so.2                                                                                                                                                                      
#25 0x00002af5dc0c4eed in ldap_sasl_interactive_bind_s () from /lib64/libldap_r-2.4.so.2                                                                                                                                                                    
                              
#33 0x00002af5dd62de25 in start_thread () from /lib64/libpthread.so.0                                
#34 0x00002af5e77283dd in clone () from /lib64/libc.so.6

Comment 4 Robbie Harwood 2018-07-20 20:45:35 UTC

Please note that I am Robbie Harwood, from Red Hat :)

It sounds like you aren't hitting this crash.  Please update the bug if that changes.

Comment 5 Norman Sardella 2018-07-20 21:04:48 UTC

>It sounds like you aren't hitting this crash.  Please update the bug if that changes.


Our application calls ldap_sasl_bind_interactive() bind to an MS AD Server.This calls  krb5_ccahe to obtain  the TGS ticket required for the bind operation., we have multiple threads to handle the ldap requests. 

In every thread,we check the cache if it contains the required TGS for the configured principal.If it doesn't, then we authenticate the user again and get a new TGS ticket for ldap service. Every thread creates its own krb5_context to authenticate the user,but all the threads are using the same ccache object.
Everything works fine while krb5 FILE type of ccache is in use. 
When we switch to MEMORY, our application crashes intermittently.

The concurrency issue is best explained in http://krbdev.mit.edu/rt/Ticket/Display.html?id=8202
After we apply that patch, we no longer see crashes