Bug 680367 - sssd not thread-safe
sssd not thread-safe
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: sssd (Show other bugs)
6.0
Unspecified Linux
urgent Severity urgent
: rc
: ---
Assigned To: Stephen Gallagher
Chandrasekar Kannan
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-25 04:34 EST by Todd Lipcon
Modified: 2017-02-06 09:35 EST (History)
8 users (show)

See Also:
Fixed In Version: sssd-1.5.1-10.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-19 07:39:49 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Todd Lipcon 2011-02-25 04:34:15 EST
Description of problem:

The version of the sssd client included in RHEL 6 "Santiago" is not thread safe for some calls. In particular, under heavy concurrency of getpwuid_r on a system slaved to LDAP, the process making the calls eventually crashes with an invalid free.

The relevant Fedora bug is here:
https://fedorahosted.org/sssd/ticket/640
fixed in SSSD 1.5.0.

Version-Release number of selected component (if applicable):

RHEL6.0 includes sssd-client 1.2.1

How reproducible:

Very.

Steps to Reproduce:
1. Call getpwuid_r at a high rate from several threads
2. Eventually process will crash

I can reproduce this reliably when load testing a piece of software that relies on these calls. If necessary I can probably write a simple C test case to reproduce the problem, but the upstream bug should be sufficient.
  
Actual results:

getpwuid_r returns strange error codes, and soon memory corruption results, for example due to invalid frees:
*** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3575675676]
/lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
/lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

Expected results:

Should function correctly, since getpwuid_r is a reentrant-safe call.
Comment 4 Kaushik Banerjee 2011-03-18 07:32:46 EDT
1. Configure sssd 1.2 running on RHEL6.0
2. run ./nss-thread-test (compiled as "gcc -lpthread nss-thread-test.c -o nss-thread-test")

The script doesn't finish executing and crashes with following error:
<snip>

7f9b317fd000-7f9b321fd000 rw-p 00000000 00:00 0 
7f9b321fd000-7f9b321fe000 ---p 00000000 00:00 0 
7f9b321fe000-7f9b32bfe000 rw-p 00000000 00:00 0 
7f9b32bfe000-7f9b32bff000 ---p 00000000 00:00 0 
7f9b32bff000-7f9b335ff000 rw-p 00000000 00:00 0 
7f9b335ff000-7f9b33600000 ---p 00000000 00:00 0 Aborted (core dumped)

</snip>

3. On another RHEL6.1 machine with sssd 1.5 installed and configured.
4. Repeat step 2.

Result:
The multi-threaded script executes successfully.

<snip>

Completed join with thread 43 status= 0
Completed join with thread 44 status= 0
Completed join with thread 45 status= 0
Completed join with thread 46 status= 0
Completed join with thread 47 status= 0
Completed join with thread 48 status= 0
Completed join with thread 49 status= 0

</snip>

Version: rpm -qi sssd | head
Name        : sssd                         Relocations: (not relocatable)
Version     : 1.5.1                             Vendor: Red Hat, Inc.
Release     : 14.el6                        Build Date: Thu 10 Mar 2011 01:00:12 AM IST
Install Date: Fri 11 Mar 2011 02:36:04 AM IST      Build Host: x86-009.build.bos.redhat.com
Group       : Applications/System           Source RPM: sssd-1.5.1-14.el6.src.rpm
Size        : 3418526                          License: GPLv3+
Signature   : (none)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://fedorahosted.org/sssd/
Summary     : System Security Services Daemon


Marking bug as 'VERIFIED'


----------nss-thread-test.c-----------

#define _GNU_SOURCE

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <pwd.h>

#define NUM_THREADS 50
#define BUFLEN 4096

struct thread_data {
    int  tid;
    int  min;
    int  max;
};

struct thread_data td[NUM_THREADS];

void *thread_main(void *arg)
{
    struct thread_data *data = (struct thread_data *) arg;
    int i;
    struct passwd pw, *pwp;
    char buf[BUFLEN];
    int num = 0;
    unsigned long long uid;

    for (i=0; i<NUM_THREADS; i++) {
        uid = random() % (data->max - data->min) + data->min;
        printf("calling getpwuid from %d for %llu (%d/%d)\n",
                data->tid, uid, i, NUM_THREADS);
        getpwuid_r(uid, &pw, buf, BUFLEN, &pwp);
    }

    printf("Thread %d done\n", data->tid);
    pthread_exit((void *) NULL);
}


int main(int argc, char *argv[])
{
    pthread_t thread[NUM_THREADS];
    pthread_attr_t attr;
    int ret, t;
    void *status;

    pthread_attr_init(&attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

    srandom(time(NULL));

    for(t=0; t < NUM_THREADS; t++) {
        printf("Creating thread %d\n", t);

        td[t].tid = t;
        td[t].min = 501;
        td[t].max = 9999;

        ret = pthread_create(&thread[t], &attr, thread_main, &td[t]);
        if (ret) {
            printf("ERROR; return code from pthread_create() is %d\n", ret);
            goto done;
        }
    }

    for(t=0; t<NUM_THREADS; t++) {
        ret = pthread_join(thread[t], &status);
        if (ret) {
            printf("ERROR return code from pthread_join() is %d\n", ret);
            goto done;
        }
        printf("Completed join with thread %d status= %ld\n", t, (long) status);
    }


    ret = 0;
done:
    pthread_attr_destroy(&attr);
    pthread_exit(NULL);
    return ret;
}
Comment 7 errata-xmlrpc 2011-05-19 07:39:49 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html
Comment 8 errata-xmlrpc 2011-05-19 09:09:42 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html

Note You need to log in before you can comment on or make changes to this bug.