Hide Forgot
Description of problem: The version of the sssd client included in RHEL 6 "Santiago" is not thread safe for some calls. In particular, under heavy concurrency of getpwuid_r on a system slaved to LDAP, the process making the calls eventually crashes with an invalid free. The relevant Fedora bug is here: https://fedorahosted.org/sssd/ticket/640 fixed in SSSD 1.5.0. Version-Release number of selected component (if applicable): RHEL6.0 includes sssd-client 1.2.1 How reproducible: Very. Steps to Reproduce: 1. Call getpwuid_r at a high rate from several threads 2. Eventually process will crash I can reproduce this reliably when load testing a piece of software that relies on these calls. If necessary I can probably write a simple C test case to reproduce the problem, but the upstream bug should be sufficient. Actual results: getpwuid_r returns strange error codes, and soon memory corruption results, for example due to invalid frees: *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3575675676] /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb] /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd] Expected results: Should function correctly, since getpwuid_r is a reentrant-safe call.
1. Configure sssd 1.2 running on RHEL6.0 2. run ./nss-thread-test (compiled as "gcc -lpthread nss-thread-test.c -o nss-thread-test") The script doesn't finish executing and crashes with following error: <snip> 7f9b317fd000-7f9b321fd000 rw-p 00000000 00:00 0 7f9b321fd000-7f9b321fe000 ---p 00000000 00:00 0 7f9b321fe000-7f9b32bfe000 rw-p 00000000 00:00 0 7f9b32bfe000-7f9b32bff000 ---p 00000000 00:00 0 7f9b32bff000-7f9b335ff000 rw-p 00000000 00:00 0 7f9b335ff000-7f9b33600000 ---p 00000000 00:00 0 Aborted (core dumped) </snip> 3. On another RHEL6.1 machine with sssd 1.5 installed and configured. 4. Repeat step 2. Result: The multi-threaded script executes successfully. <snip> Completed join with thread 43 status= 0 Completed join with thread 44 status= 0 Completed join with thread 45 status= 0 Completed join with thread 46 status= 0 Completed join with thread 47 status= 0 Completed join with thread 48 status= 0 Completed join with thread 49 status= 0 </snip> Version: rpm -qi sssd | head Name : sssd Relocations: (not relocatable) Version : 1.5.1 Vendor: Red Hat, Inc. Release : 14.el6 Build Date: Thu 10 Mar 2011 01:00:12 AM IST Install Date: Fri 11 Mar 2011 02:36:04 AM IST Build Host: x86-009.build.bos.redhat.com Group : Applications/System Source RPM: sssd-1.5.1-14.el6.src.rpm Size : 3418526 License: GPLv3+ Signature : (none) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> URL : http://fedorahosted.org/sssd/ Summary : System Security Services Daemon Marking bug as 'VERIFIED' ----------nss-thread-test.c----------- #define _GNU_SOURCE #include <pthread.h> #include <stdio.h> #include <stdlib.h> #include <pwd.h> #define NUM_THREADS 50 #define BUFLEN 4096 struct thread_data { int tid; int min; int max; }; struct thread_data td[NUM_THREADS]; void *thread_main(void *arg) { struct thread_data *data = (struct thread_data *) arg; int i; struct passwd pw, *pwp; char buf[BUFLEN]; int num = 0; unsigned long long uid; for (i=0; i<NUM_THREADS; i++) { uid = random() % (data->max - data->min) + data->min; printf("calling getpwuid from %d for %llu (%d/%d)\n", data->tid, uid, i, NUM_THREADS); getpwuid_r(uid, &pw, buf, BUFLEN, &pwp); } printf("Thread %d done\n", data->tid); pthread_exit((void *) NULL); } int main(int argc, char *argv[]) { pthread_t thread[NUM_THREADS]; pthread_attr_t attr; int ret, t; void *status; pthread_attr_init(&attr); pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE); srandom(time(NULL)); for(t=0; t < NUM_THREADS; t++) { printf("Creating thread %d\n", t); td[t].tid = t; td[t].min = 501; td[t].max = 9999; ret = pthread_create(&thread[t], &attr, thread_main, &td[t]); if (ret) { printf("ERROR; return code from pthread_create() is %d\n", ret); goto done; } } for(t=0; t<NUM_THREADS; t++) { ret = pthread_join(thread[t], &status); if (ret) { printf("ERROR return code from pthread_join() is %d\n", ret); goto done; } printf("Completed join with thread %d status= %ld\n", t, (long) status); } ret = 0; done: pthread_attr_destroy(&attr); pthread_exit(NULL); return ret; }
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0560.html