Bug 680367

Summary: sssd not thread-safe
Product: Red Hat Enterprise Linux 6 Reporter: Todd Lipcon <todd>
Component: sssdAssignee: Stephen Gallagher <sgallagh>
Status: CLOSED ERRATA QA Contact: Chandrasekar Kannan <ckannan>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.0CC: benl, dpal, eli, grajaiya, jgalipea, kbanerje, kmaiti, prc
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: sssd-1.5.1-10.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 11:39:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Todd Lipcon 2011-02-25 09:34:15 UTC
Description of problem:

The version of the sssd client included in RHEL 6 "Santiago" is not thread safe for some calls. In particular, under heavy concurrency of getpwuid_r on a system slaved to LDAP, the process making the calls eventually crashes with an invalid free.

The relevant Fedora bug is here:
https://fedorahosted.org/sssd/ticket/640
fixed in SSSD 1.5.0.

Version-Release number of selected component (if applicable):

RHEL6.0 includes sssd-client 1.2.1

How reproducible:

Very.

Steps to Reproduce:
1. Call getpwuid_r at a high rate from several threads
2. Eventually process will crash

I can reproduce this reliably when load testing a piece of software that relies on these calls. If necessary I can probably write a simple C test case to reproduce the problem, but the upstream bug should be sufficient.
  
Actual results:

getpwuid_r returns strange error codes, and soon memory corruption results, for example due to invalid frees:
*** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3575675676]
/lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
/lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

Expected results:

Should function correctly, since getpwuid_r is a reentrant-safe call.

Comment 4 Kaushik Banerjee 2011-03-18 11:32:46 UTC
1. Configure sssd 1.2 running on RHEL6.0
2. run ./nss-thread-test (compiled as "gcc -lpthread nss-thread-test.c -o nss-thread-test")

The script doesn't finish executing and crashes with following error:
<snip>

7f9b317fd000-7f9b321fd000 rw-p 00000000 00:00 0 
7f9b321fd000-7f9b321fe000 ---p 00000000 00:00 0 
7f9b321fe000-7f9b32bfe000 rw-p 00000000 00:00 0 
7f9b32bfe000-7f9b32bff000 ---p 00000000 00:00 0 
7f9b32bff000-7f9b335ff000 rw-p 00000000 00:00 0 
7f9b335ff000-7f9b33600000 ---p 00000000 00:00 0 Aborted (core dumped)

</snip>

3. On another RHEL6.1 machine with sssd 1.5 installed and configured.
4. Repeat step 2.

Result:
The multi-threaded script executes successfully.

<snip>

Completed join with thread 43 status= 0
Completed join with thread 44 status= 0
Completed join with thread 45 status= 0
Completed join with thread 46 status= 0
Completed join with thread 47 status= 0
Completed join with thread 48 status= 0
Completed join with thread 49 status= 0

</snip>

Version: rpm -qi sssd | head
Name        : sssd                         Relocations: (not relocatable)
Version     : 1.5.1                             Vendor: Red Hat, Inc.
Release     : 14.el6                        Build Date: Thu 10 Mar 2011 01:00:12 AM IST
Install Date: Fri 11 Mar 2011 02:36:04 AM IST      Build Host: x86-009.build.bos.redhat.com
Group       : Applications/System           Source RPM: sssd-1.5.1-14.el6.src.rpm
Size        : 3418526                          License: GPLv3+
Signature   : (none)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://fedorahosted.org/sssd/
Summary     : System Security Services Daemon


Marking bug as 'VERIFIED'


----------nss-thread-test.c-----------

#define _GNU_SOURCE

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <pwd.h>

#define NUM_THREADS 50
#define BUFLEN 4096

struct thread_data {
    int  tid;
    int  min;
    int  max;
};

struct thread_data td[NUM_THREADS];

void *thread_main(void *arg)
{
    struct thread_data *data = (struct thread_data *) arg;
    int i;
    struct passwd pw, *pwp;
    char buf[BUFLEN];
    int num = 0;
    unsigned long long uid;

    for (i=0; i<NUM_THREADS; i++) {
        uid = random() % (data->max - data->min) + data->min;
        printf("calling getpwuid from %d for %llu (%d/%d)\n",
                data->tid, uid, i, NUM_THREADS);
        getpwuid_r(uid, &pw, buf, BUFLEN, &pwp);
    }

    printf("Thread %d done\n", data->tid);
    pthread_exit((void *) NULL);
}


int main(int argc, char *argv[])
{
    pthread_t thread[NUM_THREADS];
    pthread_attr_t attr;
    int ret, t;
    void *status;

    pthread_attr_init(&attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

    srandom(time(NULL));

    for(t=0; t < NUM_THREADS; t++) {
        printf("Creating thread %d\n", t);

        td[t].tid = t;
        td[t].min = 501;
        td[t].max = 9999;

        ret = pthread_create(&thread[t], &attr, thread_main, &td[t]);
        if (ret) {
            printf("ERROR; return code from pthread_create() is %d\n", ret);
            goto done;
        }
    }

    for(t=0; t<NUM_THREADS; t++) {
        ret = pthread_join(thread[t], &status);
        if (ret) {
            printf("ERROR return code from pthread_join() is %d\n", ret);
            goto done;
        }
        printf("Completed join with thread %d status= %ld\n", t, (long) status);
    }


    ret = 0;
done:
    pthread_attr_destroy(&attr);
    pthread_exit(NULL);
    return ret;
}

Comment 7 errata-xmlrpc 2011-05-19 11:39:49 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html

Comment 8 errata-xmlrpc 2011-05-19 13:09:42 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html