RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1605756 - krb5 memory ccache cursors are invalidated by initialize, likely crashes
Summary: krb5 memory ccache cursors are invalidated by initialize, likely crashes
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: krb5
Version: 7.3
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Robbie Harwood
QA Contact: BaseOS QE Security Team
Mirek Jahoda
URL:
Whiteboard:
Depends On:
Blocks: 1647919 1657890
TreeView+ depends on / blocked
 
Reported: 2018-07-20 16:50 UTC by Norman Sardella
Modified: 2019-08-06 15:50 UTC (History)
9 users (show)

Fixed In Version: krb5-1.15.1-35.el7
Doc Type: Bug Fix
Doc Text:
.`krb5` memory caches are now thread-safe Previously, the memory caches of the Kerberos V5 login program (`krb5`) were not completely thread-safe. As a consequence, multi-threaded access terminated unexpectedly in some cases. With this update, the memory caches are cleaned up to be more thread-safe. As a result, no more crashes occur.
Clone Of:
: 1657890 1671329 (view as bug list)
Environment:
Last Closed: 2019-08-06 15:50:38 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Norman Sardella 2018-07-20 16:50:30 UTC 
Description of problem:
I like Redhat to consider backporting the follow patch that applies cleanly
to RHEL7.3 krb5. This resolves crashes due to a dangling pointer


http://krbdev.mit.edu/rt/Ticket/Display.html?id=8202

 Tue Jun 16 23:05:45 2015  ghudson - Ticket created      
      Subject: memory ccache cursors are invalidated by initialize

Memory ccache objects contain a linked list of credentials.  The
iteration cursor is a pointer to one of the list elements.  If the
cache is initialized by one thread while another thread is iterating
over it, the second thread's cursor contains a dangling pointer and the
process will likely crash.

(This behavior can also be demonstrated in a single-threaded caller,
but the caller would have to be doing something obtuse.)
 



Version-Release number of selected component (if applicable):


How reproducible:
fix and test case can be found here

https://github.com/krb5/krb5/commit/146dadec8fe7ccc4149eb2e3f577cc320aee6efb
Author: Greg Hudson <ghudson>
Commit: 146dadec8fe7ccc4149eb2e3f577cc320aee6efb
Branch: master
 src/lib/krb5/ccache/cc_memory.c |  164 ++++++++++++++++++++++++--------------
 src/lib/krb5/ccache/t_cc.c      |   51 ++++++++++++
 2 files changed, 154 insertions(+), 61 deletions(-)


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Robbie Harwood 2018-07-20 17:48:15 UTC 
Hi, under what circumstances are you seeing this crash?
Comment 3 Norman Sardella 2018-07-20 18:07:58 UTC 
I don't have a repro of hand other that what described in the krbdev ticket

here is a crash I see without the fix in http://krbdev.mit.edu/rt/Ticket/Display.html?id=8202
https://github.com/krb5/krb5/pull/804
Greg Hudson can explain in more detail along with Robbie Harwood from Redhat that reviewed the commit



(gdb) bt                                                                                                                                                                                                                                                    
#0  0x00002af5e777aed5 in __memcpy_ssse3_back () from /lib64/libc.so.6                                                                                                                                                                                      
#1  0x00002af5db158949 in memcpy (__len=972, __src=<optimized out>, __dest=<optimized out>) at /usr/include/bits/string3.h:51                                                                                                                               
#2  krb5int_copy_data_contents (context=context@entry=0x2af6200e02c0, indata=indata@entry=0x2af7940ced30, outdata=outdata@entry=0x2af6200df850) at copy_data.c:73                                                                                           
#3  0x00002af5db15899d in krb5_copy_data (context=context@entry=0x2af6200e02c0, indata=indata@entry=0x2af7940ced30, outdata=outdata@entry=0x2af7f98fc0b0) at copy_data.c:51                                                                                 
#4  0x00002af5db15874c in k5_copy_creds_contents (context=0x2af6200e02c0, incred=0x2af7940cece0, tempcred=0x2af7f98fc170) at copy_creds.c:78                                                                                                                
#5  0x00002af5db14953f in krb5_mcc_next_cred (context=<optimized out>, id=<optimized out>, cursor=0x2af7f98fc168, creds=<optimized out>) at cc_memory.c:375                                                                                                 
#6  0x00002af5db145162 in krb5_cc_retrieve_cred_seq (context=context@entry=0x2af6200e02c0, id=id@entry=0x2af6200df100, whichfields=whichfields@entry=545, mcreds=mcreds@entry=0x2af7f98fc430, creds=creds@entry=0x2af6200df120, nktypes=6,                  
    ktypes=0x2af6200df210) at cc_retr.c:221                                                                                                                                                                                                                 
#7  0x00002af5db145593 in k5_cc_retrieve_cred_default (context=0x2af6200e02c0, id=0x2af6200df100, flags=545, mcreds=0x2af7f98fc430, creds=0x2af6200df120) at cc_retr.c:272                                                                                  
#8  0x00002af5db14c324 in krb5_cc_retrieve_cred (context=context@entry=0x2af6200e02c0, cache=cache@entry=0x2af6200df100, flags=flags@entry=545, mcreds=mcreds@entry=0x2af7f98fc430, creds=creds@entry=0x2af6200df120) at ccfns.c:120                        
#9  0x00002af5db15ce5f in cache_get (context=context@entry=0x2af6200e02c0, ccache=0x2af6200df100, flags=545, in_creds=in_creds@entry=0x2af7f98fc430, out_creds=out_creds@entry=0x2af6200df010) at get_creds.c:242                                           
#10 0x00002af5db15e550 in check_cache (ctx=0x2af6200deee0, context=0x2af6200e02c0) at get_creds.c:1042                                                                                                                                                      
#11 begin (ctx=0x2af6200deee0, context=0x2af6200e02c0) at get_creds.c:1065                                                                                                                                                                                  
#12 krb5_tkt_creds_step (context=context@entry=0x2af6200e02c0, ctx=ctx@entry=0x2af6200deee0, in=in@entry=0x2af7f98fc520, out=out@entry=0x2af7f98fc510, realm=realm@entry=0x2af7f98fc530, flags=flags@entry=0x2af7f98fc508) at get_creds.c:1246              
#13 0x00002af5db15ece7 in krb5_tkt_creds_get (context=context@entry=0x2af6200e02c0, ctx=0x2af6200deee0) at get_creds.c:1190                                                                                                                                 
#14 0x00002af5db15ee0c in krb5_get_credentials (context=context@entry=0x2af6200e02c0, options=options@entry=0, ccache=<optimized out>, in_creds=in_creds@entry=0x2af7f98fc680, out_creds=out_creds@entry=0x2af7f98fc678) at get_creds.c:1279                
#15 0x00002af5db42484c in get_credentials (server=<optimized out>, out_creds=<synthetic pointer>, endtime=<optimized out>, now=1524496366, cred=0x2af6200de9c0, context=0x2af6200e02c0) at init_sec_context.c:199                                           
#16 kg_new_connection (exts=0x2af7f98fc910, context=0x2af6200e02c0, time_rec=0x0, ret_flags=0x2af7f98fca9c, output_token=0x2af7f98fcab0, actual_mech_type=0x0, input_token=<optimized out>, input_chan_bindings=0x0, time_req=<optimized out>,              
    req_flags=58, mech_type=0x2af5db6435c0 <krb5_gss_oid_array>, target_name=<optimized out>, context_handle=0x2af6200e0480, cred=0x2af6200de9c0, minor_status=0x2af7f98fca94) at init_sec_context.c:589                                                    
#17 krb5_gss_init_sec_context_ext (minor_status=0x2af7f98fca94, claimant_cred_handle=0x2af6200de9c0, context_handle=0x2af6200e0480, target_name=<optimized out>, mech_type=0x2af5db6435c0 <krb5_gss_oid_array>, req_flags=58, time_req=time_req@entry=0,    
    input_chan_bindings=input_chan_bindings@entry=0x0, input_token=input_token@entry=0x0, actual_mech_type=actual_mech_type@entry=0x0, output_token=output_token@entry=0x2af7f98fcab0, ret_flags=ret_flags@entry=0x2af7f98fca9c,                            
    time_rec=time_rec@entry=0x0, exts=exts@entry=0x2af7f98fc910) at init_sec_context.c:1009                                                                                                                                                                 
#18 0x00002af5db425137 in krb5_gss_init_sec_context (minor_status=<optimized out>, claimant_cred_handle=<optimized out>, context_handle=<optimized out>, target_name=<optimized out>, mech_type=<optimized out>, req_flags=<optimized out>, time_req=0,     
    input_chan_bindings=0x0, input_token=0x0, actual_mech_type=0x0, output_token=0x2af7f98fcab0, ret_flags=0x2af7f98fca9c, time_rec=0x0) at init_sec_context.c:1111                                                                                         
#19 0x00002af5db40e10b in gss_init_sec_context (minor_status=0x2af7f98fca94, claimant_cred_handle=<optimized out>, context_handle=0x2af6200df3e8, target_name=0x2af6200dc680, req_mech_type=<optimized out>, req_flags=58, time_req=0,                      
    input_chan_bindings=0x0, input_token=0x0, actual_mech_type=0x0, output_token=0x2af7f98fcab0, ret_flags=0x2af7f98fca9c, time_rec=0x0) at g_init_sec_context.c:247
#20 0x00002af7fcd07d51 in gssapi_client_mech_step () from /usr/lib64/sasl2/libgssapiv2.so                                                                                                                                                                   
#21 0x00002af5e24b4835 in sasl_client_step () from /lib64/libsasl2.so.3                                                                                                                                                                                     
#22 0x00002af5e24b4bb6 in sasl_client_start () from /lib64/libsasl2.so.3                                                                                                                                                                                    
#23 0x00002af5dc0c14ab in ldap_int_sasl_bind () from /lib64/libldap_r-2.4.so.2                                                                                                                                                                              
#24 0x00002af5dc0c4ce7 in ldap_sasl_interactive_bind () from /lib64/libldap_r-2.4.so.2                                                                                                                                                                      
#25 0x00002af5dc0c4eed in ldap_sasl_interactive_bind_s () from /lib64/libldap_r-2.4.so.2                                                                                                                                                                    
                              
#33 0x00002af5dd62de25 in start_thread () from /lib64/libpthread.so.0                                
#34 0x00002af5e77283dd in clone () from /lib64/libc.so.6
Comment 4 Robbie Harwood 2018-07-20 20:45:35 UTC 
Please note that I am Robbie Harwood, from Red Hat :)

It sounds like you aren't hitting this crash.  Please update the bug if that changes.
Comment 5 Norman Sardella 2018-07-20 21:04:48 UTC 
>It sounds like you aren't hitting this crash.  Please update the bug if that changes.


Our application calls ldap_sasl_bind_interactive() bind to an MS AD Server.This calls  krb5_ccahe to obtain  the TGS ticket required for the bind operation., we have multiple threads to handle the ldap requests. 

In every thread,we check the cache if it contains the required TGS for the configured principal.If it doesn't, then we authenticate the user again and get a new TGS ticket for ldap service. Every thread creates its own krb5_context to authenticate the user,but all the threads are using the same ccache object.
Everything works fine while krb5 FILE type of ccache is in use. 
When we switch to MEMORY, our application crashes intermittently.

The concurrency issue is best explained in http://krbdev.mit.edu/rt/Ticket/Display.html?id=8202
After we apply that patch, we no longer see crashes
Note You need to log in before you can comment on or make changes to this bug.