Bug 963235

Summary: sssd_be crashing with nested ldap groups
Product: Red Hat Enterprise Linux 6 Reporter: Aaron Bliss <abliss>
Component: sssdAssignee: Jakub Hrozek <jhrozek>
Status: CLOSED ERRATA QA Contact: Kaushik Banerjee <kbanerje>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.4CC: abliss, chad, ddas, grajaiya, hgraham, jgalipea, lslebodn, mkosek, mleary, nc, nkarandi, pbrezina
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: sssd-1.9.2-116.el6 Doc Type: Bug Fix
Doc Text:
Cause: When a group was processed whose members were all outside the configured search bases, this search request was terminated in a wrong way, causing a use-after-free memory access. Consequence: As a consequence of the use-after-free access, the sssd_be process might have crashed. Fix: The faulty search request was fixed so that it terminates correctly even if all group members are outside the configured search bases. Result: The sssd_be process no longer crashes in the above scenario.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 22:17:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
coredump generated by abrtd none

Description Aaron Bliss 2013-05-15 12:57:27 UTC 
Description of problem:
I have several fully patched RedHat boxes (20 or more), with the following sssd rpms installed:

sssd-client-1.9.2-82.7.el6_4.x86_64
sssd-1.9.2-82.7.el6_4.x86_64 

Whenever a lookup is done (for example opening an SSH session or running groups username) to figure out a users' group membership and that particular user is a member of a ldap group that is nested in another ldap group, sssd_be aborts with the following logged to /var/log/messages:

kernel: sssd_be[32294]: segfault at 0 ip (null) sp 00007fff4a2f2eb8 error 14 in sssd_be[400000+87000]

I do make use of the ldap_schema = rfc2307bis and ldap_group_member = uniqueMember options, as our ldap provider is Oracle Enterprise Directory Server (formally Sun Directory Server).

I have also confirmed that this issue was introduced with an update to sssd released sometime after sssd-1.9.2-82.el6.x86_64, as in order to further troubleshoot this, I did a clean build of a RedHat 6.4 client, used the same /etc/sssd/sssd.conf file and have yet to have any trouble with the sssd daemon crashing.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. install update to sssd higher than sssd-1.9.2-82.el6.x86_64 
2. use nested ldap groups
3. run groups username command
  
Actual results:
sssd crashes with the following logged to /var/log/messages:

kernel: sssd_be[32294]: segfault at 0 ip (null) sp 00007fff4a2f2eb8 error 14 in sssd_be[400000+87000]

Expected results:

with client running sssd-1.9.2-82.7.el6_4.x86_64 or lowered (i.e. fresh RedHat 6.4 install or previous version of 6.x) sssd does not crash

Additional info:
Comment 2 Ondrej Kos 2013-05-15 13:53:36 UTC 
Hi,

I just briefly tested the given situation

RHEL 6.4 machine
sssd-1.9.2-82.el6.x86_64
sssd-client-1.9.2-82.el6.x86_64

I tried to retrieve groups for users in the top group, for users in the nested groups, and groups itself, several times - with or without cache cleaned.

I couldn't reproduce the reported issue, could you please provide sanitized sssd.conf and logs with raised debug level?
Comment 3 Jakub Hrozek 2013-05-15 13:57:45 UTC 
Hello Aaron,

thank you for reporting the issue. Can you please provide us with a couple more information to troubleshoot the problem further? In particular, we'd like to see the sssd.conf file, a core file and the debug logs.

The sssd.conf file is located at /etc/sssd/sssd.conf. Please remove any sensitive information before attaching it.

In order to generate the core file:
1) install the debug info packages
# debuginfo-install sssd
2) install gdb
# yum -y install gdb
3) attach the gdb to the sssd_be process and resume its operation
# gdb program $(pidof sssd_be)
resume the sssd_be operation by typing "continue" in the gdb prompt.
(gdb) continue
4) when the SSSD crashes, save the core file
(gdb) generate-core-file
5) also generate the full backtrace
(gdb) set logging on
(gdb) bt full

The gdb session would generate the core file to a file named "core.XXXXX" where XXXXX is the PID of the sssd_be process that crashed and the backtrace would be located in a file named "gdb.txt" by default. Please attach these files to this bug report.
Comment 4 Aaron Bliss 2013-05-15 14:07:01 UTC 
Jakub,
Your results are consistent with mine.  The package versions you tested I've confirmed are working for me as well without issue.  As I noted above, I'm having trouble with an update to sssd released after sssd-1.9.2-82.el6.x86_64.  If you could, please update to any version of sssd higher than sssd-1.9.2-82.el6.x86_64 .  sssd-1.9.2-82.el6.x86_64 is shipped with 6.4 and works, however the issue started occurring following an update released for sssd up to and including the latest package version available, which is sssd-1.9.2-82.7.el6_4.x86_64.  Please let me know if you still need the information you requested following updating to sssd-1.9.2-82.7.el6_4.x86_64 and sssd-client-1.9.2-82.7.el6_4.x86_64.  Thanks.

Aaron
Comment 5 Jakub Hrozek 2013-05-15 14:17:48 UTC 
Yes, we still need the backtrace and/or the core dump.
Comment 6 Jakub Hrozek 2013-05-15 14:21:29 UTC 
By the way if the steps using gdb are quite complex for your environment then simply installing and enabling abrt should do the very same thing automatically.
Comment 7 Aaron Bliss 2013-05-15 14:51:29 UTC 
Jakub,
I do have output available from abrt.  Do you just need the abrt full crash report or do you need the sosreport that was generated? Also, here is the sanitized sssd.conf file:

[sssd]
config_file_version = 2
services = nss, pam
domains = LDAP
[nss]
[pam]
[domain/LDAP]
auth_provider = ldap
ldap_id_use_start_tls = True
chpass_provider = ldap
cache_credentials = True
ldap_search_base = o=example.com,dc=example,dc=com
id_provider = ldap
ldap_uri = ldap://server1.example.com/,ldap://server2.example.com/
ldap_tls_cacertdir = /etc/openldap/cacerts
ldap_schema = rfc2307bis
ldap_group_member = uniqueMember
ldap_group_search_base = ou=Groups,o=example.com,dc=example,dc=com
Comment 8 Aaron Bliss 2013-05-15 14:56:34 UTC 
I'm assuming not the sosreport file that was generated, as I just extracted it and it contains configs of the entire box.  As such, here is the full crash report:

abrt_version:   2.0.8
cgroup:         
cmdline:        /usr/libexec/sssd/sssd_be --domain LDAP --debug-to-files
executable:     /usr/libexec/sssd/sssd_be
kernel:         2.6.32-358.6.1.el6.x86_64
pid:            1667
pwd:            /
time:           Wed 15 May 2013 10:40:37 AM EDT
uid:            0
username:       root

sosreport.tar.xz: Binary file, 1358368 bytes

environ:
:TERM=linux
:PATH=/sbin:/usr/sbin:/bin:/usr/bin
:runlevel=3
:RUNLEVEL=3
:LANGSH_SOURCED=1
:PWD=/
:LANG=en_US.UTF-8
:previous=N
:PREVLEVEL=N
:CONSOLETYPE=vt
:SHLVL=3
:UPSTART_INSTANCE=
:UPSTART_EVENTS=runlevel
:UPSTART_JOB=rc
:_=/usr/sbin/sssd
:_SSS_LOOPS=NO
:KRB5RCACHEDIR=/var/cache/krb5rcache

limits:
:Limit                     Soft Limit           Hard Limit           Units     
:Max cpu time              unlimited            unlimited            seconds   
:Max file size             unlimited            unlimited            bytes     
:Max data size             unlimited            unlimited            bytes     
:Max stack size            10485760             unlimited            bytes     
:Max core file size        0                    unlimited            bytes     
:Max resident set          unlimited            unlimited            bytes     
:Max processes             62707                62707                processes 
:Max open files            1024                 4096                 files     
:Max locked memory         65536                65536                bytes     
:Max address space         unlimited            unlimited            bytes     
:Max file locks            unlimited            unlimited            locks     
:Max pending signals       62707                62707                signals   
:Max msgqueue size         819200               819200               bytes     
:Max nice priority         0                    0                    
:Max realtime priority     0                    0                    
:Max realtime timeout      unlimited            unlimited            us        

maps:
:00400000-00487000 r-xp 00000000 08:02 916749                             /usr/libexec/sssd/sssd_be (deleted)
:00687000-00689000 rw-p 00087000 08:02 916749                             /usr/libexec/sssd/sssd_be (deleted)
:011f7000-0131f000 rw-p 00000000 00:00 0                                  [heap]
:37da600000-37da6db000 r-xp 00000000 08:02 783465                         /lib64/libkrb5.so.3.3
:37da6db000-37da8da000 ---p 000db000 08:02 783465                         /lib64/libkrb5.so.3.3
:37da8da000-37da8e4000 r--p 000da000 08:02 783465                         /lib64/libkrb5.so.3.3
:37da8e4000-37da8e6000 rw-p 000e4000 08:02 783465                         /lib64/libkrb5.so.3.3
:37daa00000-37daa29000 r-xp 00000000 08:02 783464                         /lib64/libk5crypto.so.3.1
:37daa29000-37dac29000 ---p 00029000 08:02 783464                         /lib64/libk5crypto.so.3.1
:37dac29000-37dac2a000 r--p 00029000 08:02 783464                         /lib64/libk5crypto.so.3.1
:37dac2a000-37dac2b000 rw-p 0002a000 08:02 783464                         /lib64/libk5crypto.so.3.1
:37dac2b000-37dac2c000 rw-p 00000000 00:00 0 
:37dae00000-37dae0a000 r-xp 00000000 08:02 783446                         /lib64/libkrb5support.so.0.1
:37dae0a000-37db009000 ---p 0000a000 08:02 783446                         /lib64/libkrb5support.so.0.1
:37db009000-37db00a000 r--p 00009000 08:02 783446                         /lib64/libkrb5support.so.0.1
:37db00a000-37db00b000 rw-p 0000a000 08:02 783446                         /lib64/libkrb5support.so.0.1
:387ae00000-387ae20000 r-xp 00000000 08:02 783373                         /lib64/ld-2.12.so
:387b01f000-387b020000 r--p 0001f000 08:02 783373                         /lib64/ld-2.12.so
:387b020000-387b021000 rw-p 00020000 08:02 783373                         /lib64/ld-2.12.so
:387b021000-387b022000 rw-p 00000000 00:00 0 
:387b200000-387b38a000 r-xp 00000000 08:02 783391                         /lib64/libc-2.12.so
:387b38a000-387b589000 ---p 0018a000 08:02 783391                         /lib64/libc-2.12.so
:387b589000-387b58d000 r--p 00189000 08:02 783391                         /lib64/libc-2.12.so
:387b58d000-387b58e000 rw-p 0018d000 08:02 783391                         /lib64/libc-2.12.so
:387b58e000-387b593000 rw-p 00000000 00:00 0 
:387b600000-387b617000 r-xp 00000000 08:02 783396                         /lib64/libpthread-2.12.so
:387b617000-387b817000 ---p 00017000 08:02 783396                         /lib64/libpthread-2.12.so
:387b817000-387b818000 r--p 00017000 08:02 783396                         /lib64/libpthread-2.12.so
:387b818000-387b819000 rw-p 00018000 08:02 783396                         /lib64/libpthread-2.12.so
:387b819000-387b81d000 rw-p 00000000 00:00 0 
:387ba00000-387ba02000 r-xp 00000000 08:02 783404                         /lib64/libdl-2.12.so
:387ba02000-387bc02000 ---p 00002000 08:02 783404                         /lib64/libdl-2.12.so
:387bc02000-387bc03000 r--p 00002000 08:02 783404                         /lib64/libdl-2.12.so
:387bc03000-387bc04000 rw-p 00003000 08:02 783404                         /lib64/libdl-2.12.so
:387be00000-387be07000 r-xp 00000000 08:02 783473                         /lib64/librt-2.12.so
:387be07000-387c006000 ---p 00007000 08:02 783473                         /lib64/librt-2.12.so
:387c006000-387c007000 r--p 00006000 08:02 783473                         /lib64/librt-2.12.so
:387c007000-387c008000 rw-p 00007000 08:02 783473                         /lib64/librt-2.12.so
:387c200000-387c215000 r-xp 00000000 08:02 783445                         /lib64/libz.so.1.2.3
:387c215000-387c414000 ---p 00015000 08:02 783445                         /lib64/libz.so.1.2.3
:387c414000-387c415000 r--p 00014000 08:02 783445                         /lib64/libz.so.1.2.3
:387c415000-387c416000 rw-p 00015000 08:02 783445                         /lib64/libz.so.1.2.3
:387c600000-387c617000 r-xp 00000000 08:02 783484                         /lib64/libaudit.so.1.0.0
:387c617000-387c816000 ---p 00017000 08:02 783484                         /lib64/libaudit.so.1.0.0
:387c816000-387c817000 r--p 00016000 08:02 783484                         /lib64/libaudit.so.1.0.0
:387c817000-387c81c000 rw-p 00017000 08:02 783484                         /lib64/libaudit.so.1.0.0
:387ca00000-387ca0e000 r-xp 00000000 08:02 920301                         /usr/lib64/libcares.so.2.0.0
:387ca0e000-387cc0e000 ---p 0000e000 08:02 920301                         /usr/lib64/libcares.so.2.0.0
:387cc0e000-387cc0f000 rw-p 0000e000 08:02 920301                         /usr/lib64/libcares.so.2.0.0
:387ce00000-387cee4000 r-xp 00000000 08:02 783488                         /lib64/libglib-2.0.so.0.2200.5
:387cee4000-387d0e3000 ---p 000e4000 08:02 783488                         /lib64/libglib-2.0.so.0.2200.5
:387d0e3000-387d0e5000 rw-p 000e3000 08:02 783488                         /lib64/libglib-2.0.so.0.2200.5
:387d200000-387d240000 r-xp 00000000 08:02 783476                         /lib64/libdbus-1.so.3.4.0
:387d240000-387d43f000 ---p 00040000 08:02 783476                         /lib64/libdbus-1.so.3.4.0
:387d43f000-387d440000 r--p 0003f000 08:02 783476                         /lib64/libdbus-1.so.3.4.0
:387d440000-387d441000 rw-p 00040000 08:02 783476                         /lib64/libdbus-1.so.3.4.0
:387d600000-387d616000 r-xp 00000000 08:02 783423                         /lib64/libresolv-2.12.so
:387d616000-387d816000 ---p 00016000 08:02 783423                         /lib64/libresolv-2.12.so
:387d816000-387d817000 r--p 00016000 08:02 783423                         /lib64/libresolv-2.12.so
:387d817000-387d818000 rw-p 00017000 08:02 783423                         /lib64/libresolv-2.12.so
:387d818000-387d81a000 rw-p 00000000 00:00 0 
:387da00000-387da07000 r-xp 00000000 08:02 783455                         /lib64/libcrypt-2.12.so
:387da07000-387dc07000 ---p 00007000 08:02 783455                         /lib64/libcrypt-2.12.so
:387dc07000-387dc08000 r--p 00007000 08:02 783455                         /lib64/libcrypt-2.12.so
:387dc08000-387dc09000 rw-p 00008000 08:02 783455                         /lib64/libcrypt-2.12.so
:387dc09000-387dc37000 rw-p 00000000 00:00 0 
:387de00000-387de5d000 r-xp 00000000 08:02 783453                         /lib64/libfreebl3.so
:387de5d000-387e05c000 ---p 0005d000 08:02 783453                         /lib64/libfreebl3.so
:387e05c000-387e05d000 r--p 0005c000 08:02 783453                         /lib64/libfreebl3.so
:387e05d000-387e05e000 rw-p 0005d000 08:02 783453                         /lib64/libfreebl3.so
:387e05e000-387e062000 rw-p 00000000 00:00 0 
:387e200000-387e204000 r-xp 00000000 08:02 783440                         /lib64/libplc4.so
:387e204000-387e403000 ---p 00004000 08:02 783440                         /lib64/libplc4.so
:387e403000-387e404000 r--p 00003000 08:02 783440                         /lib64/libplc4.so
:387e404000-387e405000 rw-p 00004000 08:02 783440                         /lib64/libplc4.so
:387e600000-387e639000 r-xp 00000000 08:02 783439                         /lib64/libnspr4.so
:387e639000-387e838000 ---p 00039000 08:02 783439                         /lib64/libnspr4.so
:387e838000-387e839000 r--p 00038000 08:02 783439                         /lib64/libnspr4.so
:387e839000-387e83b000 rw-p 00039000 08:02 783439                         /lib64/libnspr4.so
:387e83b000-387e83d000 rw-p 00000000 00:00 0 
:387ea00000-387ea25000 r-xp 00000000 08:02 914797                         /usr/lib64/libnssutil3.so
:387ea25000-387ec24000 ---p 00025000 08:02 914797                         /usr/lib64/libnssutil3.so
:387ec24000-387ec2a000 r--p 00024000 08:02 914797                         /usr/lib64/libnssutil3.so
:387ec2a000-387ec2b000 rw-p 0002a000 08:02 914797                         /usr/lib64/libnssutil3.so
:387ee00000-387ee33000 r-xp 00000000 08:02 915560                         /usr/lib64/libssl3.so
:387ee33000-387f033000 ---p 00033000 08:02 915560                         /usr/lib64/libssl3.so
:387f033000-387f035000 r--p 00033000 08:02 915560                         /usr/lib64/libssl3.so
:387f035000-387f036000 rw-p 00035000 08:02 915560                         /usr/lib64/libssl3.so
:387f200000-387f333000 r-xp 00000000 08:02 915556                         /usr/lib64/libnss3.so
:387f333000-387f532000 ---p 00133000 08:02 915556                         /usr/lib64/libnss3.so
:387f532000-387f537000 r--p 00132000 08:02 915556                         /usr/lib64/libnss3.so
:387f537000-387f539000 rw-p 00137000 08:02 915556                         /usr/lib64/libnss3.so
:387f539000-387f53b000 rw-p 00000000 00:00 0 
:387f600000-387f603000 r-xp 00000000 08:02 783443                         /lib64/libplds4.so
:387f603000-387f802000 ---p 00003000 08:02 783443                         /lib64/libplds4.so
:387f802000-387f803000 r--p 00002000 08:02 783443                         /lib64/libplds4.so
:387f803000-387f804000 rw-p 00003000 08:02 783443                         /lib64/libplds4.so
:387fa00000-387fa28000 r-xp 00000000 08:02 915729                         /usr/lib64/libsmime3.so
:387fa28000-387fc28000 ---p 00028000 08:02 915729                         /usr/lib64/libsmime3.so
:387fc28000-387fc2b000 r--p 00028000 08:02 915729                         /usr/lib64/libsmime3.so
:387fc2b000-387fc2c000 rw-p 0002b000 08:02 915729                         /usr/lib64/libsmime3.so
:387fe00000-387fe19000 r-xp 00000000 08:02 915731                         /usr/lib64/libsasl2.so.2.0.23
:387fe19000-3880018000 ---p 00019000 08:02 915731                         /usr/lib64/libsasl2.so.2.0.23
:3880018000-3880019000 r--p 00018000 08:02 915731                         /usr/lib64/libsasl2.so.2.0.23
:3880019000-388001a000 rw-p 00019000 08:02 915731                         /usr/lib64/libsasl2.so.2.0.23
:3880200000-3880208000 r-xp 00000000 08:02 783482                         /lib64/libpopt.so.0.0.0
:3880208000-3880408000 ---p 00008000 08:02 783482                         /lib64/libpopt.so.0.0.0
:3880408000-3880409000 rw-p 00008000 08:02 783482                         /lib64/libpopt.so.0.0.0
:3880a00000-3880a0c000 r-xp 00000000 08:02 783486                         /lib64/libpam.so.0.82.2
:3880a0c000-3880c0c000 ---p 0000c000 08:02 783486                         /lib64/libpam.so.0.82.2
:3880c0c000-3880c0d000 r--p 0000c000 08:02 783486                         /lib64/libpam.so.0.82.2
:3880c0d000-3880c0e000 rw-p 0000d000 08:02 783486                         /lib64/libpam.so.0.82.2
:3881600000-388160b000 r-xp 00000000 08:02 919462                         /usr/lib64/libcollection.so.2.0.0
:388160b000-388180a000 ---p 0000b000 08:02 919462                         /usr/lib64/libcollection.so.2.0.0
:388180a000-388180b000 rw-p 0000a000 08:02 919462                         /usr/lib64/libcollection.so.2.0.0
:3881a00000-3881a02000 r-xp 00000000 08:02 783422                         /lib64/libkeyutils.so.1.3
:3881a02000-3881c01000 ---p 00002000 08:02 783422                         /lib64/libkeyutils.so.1.3
:3881c01000-3881c02000 r--p 00001000 08:02 783422                         /lib64/libkeyutils.so.1.3
:3881c02000-3881c03000 rw-p 00002000 08:02 783422                         /lib64/libkeyutils.so.1.3
:3881e00000-3881e08000 r-xp 00000000 08:02 919534                         /usr/lib64/libini_config.so.2.0.0
:3881e08000-3882007000 ---p 00008000 08:02 919534                         /usr/lib64/libini_config.so.2.0.0
:3882007000-3882008000 rw-p 00007000 08:02 919534                         /usr/lib64/libini_config.so.2.0.0
:3882200000-3882202000 r-xp 00000000 08:02 919532                         /usr/lib64/libref_array.so.1.0.0
:3882202000-3882401000 ---p 00002000 08:02 919532                         /usr/lib64/libref_array.so.1.0.0
:3882401000-3882402000 rw-p 00001000 08:02 919532                         /usr/lib64/libref_array.so.1.0.0
:3882600000-388260c000 r-xp 00000000 08:02 919286                         /usr/lib64/libtevent.so.0.9.17
:388260c000-388280b000 ---p 0000c000 08:02 919286                         /usr/lib64/libtevent.so.0.9.17
:388280b000-388280c000 rw-p 0000b000 08:02 919286                         /usr/lib64/libtevent.so.0.9.17
:3882a00000-3882a0a000 r-xp 00000000 08:02 917590                         /usr/lib64/libtalloc.so.2.0.7
:3882a0a000-3882c09000 ---p 0000a000 08:02 917590                         /usr/lib64/libtalloc.so.2.0.7
:3882c09000-3882c0a000 rw-p 00009000 08:02 917590                         /usr/lib64/libtalloc.so.2.0.7
:3882e00000-3882e03000 r-xp 00000000 08:02 919539                         /usr/lib64/libdhash.so.1.0.0
:3882e03000-3883002000 ---p 00003000 08:02 919539                         /usr/lib64/libdhash.so.1.0.0
:3883002000-3883003000 rw-p 00002000 08:02 919539                         /usr/lib64/libdhash.so.1.0.0
:3883200000-388322c000 r-xp 00000000 08:02 919305                         /usr/lib64/libldb.so.1.1.13
:388322c000-388342c000 ---p 0002c000 08:02 919305                         /usr/lib64/libldb.so.1.1.13
:388342c000-388342d000 rw-p 0002c000 08:02 919305                         /usr/lib64/libldb.so.1.1.13
:3883600000-3883611000 r-xp 00000000 08:02 919303                         /usr/lib64/libtdb.so.1.2.10
:3883611000-3883810000 ---p 00011000 08:02 919303                         /usr/lib64/libtdb.so.1.2.10
:3883810000-3883811000 rw-p 00010000 08:02 919303                         /usr/lib64/libtdb.so.1.2.10
:3883a00000-3883a03000 r-xp 00000000 08:02 919463                         /usr/lib64/libpath_utils.so.1.0.0
:3883a03000-3883c02000 ---p 00003000 08:02 919463                         /usr/lib64/libpath_utils.so.1.0.0
:3883c02000-3883c03000 rw-p 00002000 08:02 919463                         /usr/lib64/libpath_utils.so.1.0.0
:3981400000-398142c000 r-xp 00000000 08:02 783459                         /lib64/libpcre.so.0.0.1
:398142c000-398162b000 ---p 0002c000 08:02 783459                         /lib64/libpcre.so.0.0.1
:398162b000-398162c000 rw-p 0002b000 08:02 783459                         /lib64/libpcre.so.0.0.1
:7fc68aab4000-7fc68aad7000 r-xp 00000000 08:02 926950                     /usr/lib64/libnsspem.so
:7fc68aad7000-7fc68acd7000 ---p 00023000 08:02 926950                     /usr/lib64/libnsspem.so
:7fc68acd7000-7fc68acd8000 r--p 00023000 08:02 926950                     /usr/lib64/libnsspem.so
:7fc68acd8000-7fc68acd9000 rw-p 00024000 08:02 926950                     /usr/lib64/libnsspem.so
:7fc68acd9000-7fc68ad65000 r-xp 00000000 08:02 915451                     /usr/lib64/libsqlite3.so.0.8.6
:7fc68ad65000-7fc68af64000 ---p 0008c000 08:02 915451                     /usr/lib64/libsqlite3.so.0.8.6
:7fc68af64000-7fc68af67000 rw-p 0008b000 08:02 915451                     /usr/lib64/libsqlite3.so.0.8.6
:7fc68af67000-7fc68af68000 rw-p 00000000 00:00 0 
:7fc68af6f000-7fc68afab000 r-xp 00000000 08:02 916705                     /usr/lib64/libsoftokn3.so
:7fc68afab000-7fc68b1ab000 ---p 0003c000 08:02 916705                     /usr/lib64/libsoftokn3.so
:7fc68b1ab000-7fc68b1ac000 r--p 0003c000 08:02 916705                     /usr/lib64/libsoftokn3.so
:7fc68b1ac000-7fc68b1ad000 rw-p 0003d000 08:02 916705                     /usr/lib64/libsoftokn3.so
:7fc68b1ad000-7fc68b1b4000 r-xp 00000000 08:02 783397                     /lib64/libnss_sss.so.2
:7fc68b1b4000-7fc68b3b3000 ---p 00007000 08:02 783397                     /lib64/libnss_sss.so.2
:7fc68b3b3000-7fc68b3b4000 rw-p 00006000 08:02 783397                     /lib64/libnss_sss.so.2
:7fc68b3b4000-7fc68b3b9000 r-xp 00000000 08:02 783485                     /lib64/libnss_dns-2.12.so
:7fc68b3b9000-7fc68b5b8000 ---p 00005000 08:02 783485                     /lib64/libnss_dns-2.12.so
:7fc68b5b8000-7fc68b5b9000 r--p 00004000 08:02 783485                     /lib64/libnss_dns-2.12.so
:7fc68b5b9000-7fc68b5ba000 rw-p 00005000 08:02 783485                     /lib64/libnss_dns-2.12.so
:7fc68b5ba000-7fc68b5c6000 r-xp 00000000 08:02 783550                     /lib64/libnss_files-2.12.so
:7fc68b5c6000-7fc68b7c6000 ---p 0000c000 08:02 783550                     /lib64/libnss_files-2.12.so
:7fc68b7c6000-7fc68b7c7000 r--p 0000c000 08:02 783550                     /lib64/libnss_files-2.12.so
:7fc68b7c7000-7fc68b7c8000 rw-p 0000d000 08:02 783550                     /lib64/libnss_files-2.12.so
:7fc68b7c8000-7fc68b7e5000 r-xp 00000000 08:02 783425                     /lib64/libselinux.so.1
:7fc68b7e5000-7fc68b9e4000 ---p 0001d000 08:02 783425                     /lib64/libselinux.so.1
:7fc68b9e4000-7fc68b9e5000 r--p 0001c000 08:02 783425                     /lib64/libselinux.so.1
:7fc68b9e5000-7fc68b9e6000 rw-p 0001d000 08:02 783425                     /lib64/libselinux.so.1
:7fc68b9e6000-7fc68b9e7000 rw-p 00000000 00:00 0 
:7fc68b9e7000-7fc68b9ea000 r-xp 00000000 08:02 919255                     /usr/lib64/libsss_idmap.so.0.0.1.#prelink#.w8EoBh (deleted)
:7fc68b9ea000-7fc68bbea000 ---p 00003000 08:02 919255                     /usr/lib64/libsss_idmap.so.0.0.1.#prelink#.w8EoBh (deleted)
:7fc68bbea000-7fc68bbeb000 rw-p 00003000 08:02 919255                     /usr/lib64/libsss_idmap.so.0.0.1.#prelink#.w8EoBh (deleted)
:7fc68bbeb000-7fc68bbee000 r-xp 00000000 08:02 783435                     /lib64/libcom_err.so.2.1
:7fc68bbee000-7fc68bded000 ---p 00003000 08:02 783435                     /lib64/libcom_err.so.2.1
:7fc68bded000-7fc68bdee000 r--p 00002000 08:02 783435                     /lib64/libcom_err.so.2.1
:7fc68bdee000-7fc68bdef000 rw-p 00003000 08:02 783435                     /lib64/libcom_err.so.2.1
:7fc68bdef000-7fc68bdf6000 r--s 00000000 08:02 1045346                    /usr/lib64/gconv/gconv-modules.cache
:7fc68bdf6000-7fc68bf46000 r-xp 00000000 08:02 927528                     /usr/lib64/sssd/libsss_ldap.so
:7fc68bf46000-7fc68c145000 ---p 00150000 08:02 927528                     /usr/lib64/sssd/libsss_ldap.so
:7fc68c145000-7fc68c14b000 rw-p 0014f000 08:02 927528                     /usr/lib64/sssd/libsss_ldap.so
:7fc68c14b000-7fc68c52a000 rw-s 00000000 08:07 208                        /var/lib/sss/db/cache_LDAP.ldb
:7fc68c52a000-7fc68c548000 rw-s 00000000 08:07 1342                       /var/lib/sss/db/config.ldb
:7fc68c548000-7fc68c556000 r-xp 00000000 08:02 927127                     /usr/lib64/ldb/modules/ldb/tdb.so
:7fc68c556000-7fc68c755000 ---p 0000e000 08:02 927127                     /usr/lib64/ldb/modules/ldb/tdb.so
:7fc68c755000-7fc68c756000 rw-p 0000d000 08:02 927127                     /usr/lib64/ldb/modules/ldb/tdb.so
:7fc68c756000-7fc68c758000 r-xp 00000000 08:02 927126                     /usr/lib64/ldb/modules/ldb/skel.so
:7fc68c758000-7fc68c957000 ---p 00002000 08:02 927126                     /usr/lib64/ldb/modules/ldb/skel.so
:7fc68c957000-7fc68c958000 rw-p 00001000 08:02 927126                     /usr/lib64/ldb/modules/ldb/skel.so
:7fc68c958000-7fc68c95a000 r-xp 00000000 08:02 927125                     /usr/lib64/ldb/modules/ldb/server_sort.so
:7fc68c95a000-7fc68cb59000 ---p 00002000 08:02 927125                     /usr/lib64/ldb/modules/ldb/server_sort.so
:7fc68cb59000-7fc68cb5a000 rw-p 00001000 08:02 927125                     /usr/lib64/ldb/modules/ldb/server_sort.so
:7fc68cb5a000-7fc68cb5b000 r-xp 00000000 08:02 927124                     /usr/lib64/ldb/modules/ldb/sample.so
:7fc68cb5b000-7fc68cd5a000 ---p 00001000 08:02 927124                     /usr/lib64/ldb/modules/ldb/sample.so
:7fc68cd5a000-7fc68cd5b000 rw-p 00000000 08:02 927124                     /usr/lib64/ldb/modules/ldb/sample.so
:7fc68cd5b000-7fc68cd5d000 r-xp 00000000 08:02 927123                     /usr/lib64/ldb/modules/ldb/rdn_name.so
:7fc68cd5d000-7fc68cf5d000 ---p 00002000 08:02 927123                     /usr/lib64/ldb/modules/ldb/rdn_name.so
:7fc68cf5d000-7fc68cf5e000 rw-p 00002000 08:02 927123                     /usr/lib64/ldb/modules/ldb/rdn_name.so
:7fc68cf5e000-7fc68cf60000 r-xp 00000000 08:02 927122                     /usr/lib64/ldb/modules/ldb/paged_searches.so
:7fc68cf60000-7fc68d160000 ---p 00002000 08:02 927122                     /usr/lib64/ldb/modules/ldb/paged_searches.so
:7fc68d160000-7fc68d161000 rw-p 00002000 08:02 927122                     /usr/lib64/ldb/modules/ldb/paged_searches.so
:7fc68d161000-7fc68d163000 r-xp 00000000 08:02 927121                     /usr/lib64/ldb/modules/ldb/paged_results.so
:7fc68d163000-7fc68d363000 ---p 00002000 08:02 927121                     /usr/lib64/ldb/modules/ldb/paged_results.so
:7fc68d363000-7fc68d364000 rw-p 00002000 08:02 927121                     /usr/lib64/ldb/modules/ldb/paged_results.so
:7fc68d364000-7fc68d372000 r-xp 00000000 08:02 927184                     /usr/lib64/ldb/modules/ldb/memberof.so
:7fc68d372000-7fc68d572000 ---p 0000e000 08:02 927184                     /usr/lib64/ldb/modules/ldb/memberof.so
:7fc68d572000-7fc68d573000 rw-p 0000e000 08:02 927184                     /usr/lib64/ldb/modules/ldb/memberof.so
:7fc68d573000-7fc68d575000 r-xp 00000000 08:02 927120                     /usr/lib64/ldb/modules/ldb/asq.so
:7fc68d575000-7fc68d774000 ---p 00002000 08:02 927120                     /usr/lib64/ldb/modules/ldb/asq.so
:7fc68d774000-7fc68d775000 rw-p 00001000 08:02 927120                     /usr/lib64/ldb/modules/ldb/asq.so
:7fc68d775000-7fc693606000 r--p 00000000 08:02 927020                     /usr/lib/locale/locale-archive
:7fc693606000-7fc693613000 rw-p 00000000 00:00 0 
:7fc693613000-7fc69365c000 r-xp 00000000 08:02 783371                     /lib64/libldap-2.4.so.2.5.6.#prelink#.QgySWG (deleted)
:7fc69365c000-7fc69385c000 ---p 00049000 08:02 783371                     /lib64/libldap-2.4.so.2.5.6.#prelink#.QgySWG (deleted)
:7fc69385c000-7fc69385e000 r--p 00049000 08:02 783371                     /lib64/libldap-2.4.so.2.5.6.#prelink#.QgySWG (deleted)
:7fc69385e000-7fc693860000 rw-p 0004b000 08:02 783371                     /lib64/libldap-2.4.so.2.5.6.#prelink#.QgySWG (deleted)
:7fc693860000-7fc69386e000 r-xp 00000000 08:02 783368                     /lib64/liblber-2.4.so.2.5.6.#prelink#.Y5n5fQ (deleted)
:7fc69386e000-7fc693a6d000 ---p 0000e000 08:02 783368                     /lib64/liblber-2.4.so.2.5.6.#prelink#.Y5n5fQ (deleted)
:7fc693a6d000-7fc693a6e000 r--p 0000d000 08:02 783368                     /lib64/liblber-2.4.so.2.5.6.#prelink#.Y5n5fQ (deleted)
:7fc693a6e000-7fc693a6f000 rw-p 0000e000 08:02 783368                     /lib64/liblber-2.4.so.2.5.6.#prelink#.Y5n5fQ (deleted)
:7fc693a6f000-7fc693a73000 rw-p 00000000 00:00 0 
:7fc693a78000-7fc693a7b000 rw-p 00000000 00:00 0 
:7fff0a42e000-7fff0a443000 rw-p 00000000 00:00 0                          [stack]
:7fff0a5ff000-7fff0a600000 r-xp 00000000 00:00 0                          [vdso]
:ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

open_fds:
:0:/dev/null
:pos:	0
:flags:	0100002
:1:/dev/null
:pos:	0
:flags:	0100002
:2:/dev/null
:pos:	0
:flags:	0100002
:3:anon_inode:[eventpoll]
:pos:	0
:flags:	02
:4:pipe:[10845]
:pos:	0
:flags:	04000
:5:pipe:[10845]
:pos:	0
:flags:	04001
:6:anon_inode:[eventpoll]
:pos:	0
:flags:	02
:7:pipe:[10840]
:pos:	0
:flags:	04000
:8:pipe:[10840]
:pos:	0
:flags:	04001
:9:/var/lib/sss/db/config.ldb
:pos:	168
:flags:	0100002
:10:inotify
:pos:	0
:flags:	04000
:11:/var/log/sssd/sssd_LDAP.log
:pos:	0
:flags:	0102001
:12:socket:[10844]
:pos:	0
:flags:	04002
:13:anon_inode:[eventpoll]
:pos:	0
:flags:	02
:14:/var/lib/sss/db/cache_LDAP.ldb
:pos:	168
:flags:	0100002
:15:socket:[10847]
:pos:	0
:flags:	04002
:16:socket:[10851]
:pos:	0
:flags:	04002
:17:/var/log/sssd/ldap_child.log
:pos:	0
:flags:	0102001
:18:socket:[10870]
:pos:	0
:flags:	04002
:19:socket:[10882]
:pos:	0
:flags:	04002
:20:socket:[3275109]
:pos:	0
:flags:	02
Comment 9 Jakub Hrozek 2013-05-15 15:11:19 UTC 
(In reply to comment #8)
> I'm assuming not the sosreport file that was generated, as I just extracted
> it and it contains configs of the entire box.  As such, here is the full
> crash report:
> 
> abrt_version:   2.0.8
> cgroup:         
> cmdline:        /usr/libexec/sssd/sssd_be --domain LDAP --debug-to-files
> executable:     /usr/libexec/sssd/sssd_be
> kernel:         2.6.32-358.6.1.el6.x86_64
> pid:            1667
> pwd:            /
> time:           Wed 15 May 2013 10:40:37 AM EDT
> uid:            0
> username:       root
> 
> sosreport.tar.xz: Binary file, 1358368 bytes

Is there also a subdirectory directory with crash data inside /var/spool/abrt ? It should contain the corefile in a file called "coredump" and a bunch of other files. Could you attach contents of that directory?
Comment 10 Aaron Bliss 2013-05-15 15:16:33 UTC 
Created attachment 748369 [details]
coredump generated by abrtd
Comment 11 Aaron Bliss 2013-05-15 15:17:13 UTC 
Jakub,
coredump has been attached.  Thanks.

Aaron
Comment 12 Ondrej Kos 2013-05-16 09:49:48 UTC 
Hi,

I examined the provided corefile, it's very helpful, but we're still not able to reproduce the issue.

Could you please provide us sanitized logs with debug_level = 10?
Comment 13 Aaron Bliss 2013-05-16 10:23:39 UTC 
Ondrej,
Just want to confirm that you are asking me to set the following in sssd.conf debug_level = 0xFFF0, reproduce the issue and upload /var/log/sssd/sssd_LDAP.log? Please advise and thanks.

Aaron
Comment 14 Ondrej Kos 2013-05-16 10:46:03 UTC 
Hi Aaron,

Yes, that is correct, place debug_level = 0xFFF0 to domain/LDAP section, remove cached data (/var/lib/sss/db/*) and logs (/var/log/sssd/*), reproduce the issue and upload the sssd_LDAP.log with removed confidential data.

Ondra
Comment 15 Aaron Bliss 2013-05-16 14:54:47 UTC 
Ondrej,
I'm working on getting the sanitized file for you.  In the meantime, after much trial and error, sssd_be seems to crash when a lookup is done to determine a users group membership and there is a member of that group that doesn't exist.  I originally believed this to be caused by group nesting, however I've been able to re-create the issue even with groups that aren't nested.  Can you attempt to re-create on your end by adding a memeber to a group that doesn't exist, clear cashe as you described above and then do a lookup against that group.  Also please be sure to be using sssd version sssd-1.9.2-82.7.el6_4.x86_64 as sssd version sssd-1.9.2-82.el6.x86_64 is not affected by this problem.  Thanks.

Aaron
Comment 16 Aaron Bliss 2013-05-16 16:50:42 UTC 
Ondrej,
After cleaning up users from several ldap groups that were no longer valid (user accounts that have been previously purged but not removed from several group), sssd_be is no longer crashing, even when processing nested groups.  While I certainly should be taking better care of group memberships when purging accounts, sssd_be shouldn't be crashing when encountering a group that contains user objects that no longer exist.  Do you still need anything further from me or have you been able to reproduce this on your end?

Aaron
Comment 17 Jakub Hrozek 2013-05-16 16:54:22 UTC 
Hey Aaron, I'm glad SSSD works for you now! You're right that the SSSD should never crash, even with a dangling member attribute. We'll prepare a fix for this issue.
Comment 18 Jakub Hrozek 2013-05-16 17:00:38 UTC 
Upstream ticket:
https://fedorahosted.org/sssd/ticket/1932
Comment 19 Lukas Slebodnik 2013-06-05 12:51:44 UTC 
Aaron,
I would like to apologize for late response.
I tried to reproduce this bug with following steps:

1. create one group "corgroup"  in ldap (rfc2307bis)
2. create few members of previous created group "buser1, buser2, ..."
3. run sssd with empty cache (directory /var/lib/sss/db/ is empty)
4. call few bash commands (id buser1 && id buser2 && groups buser1)
5. remove user buser1 from ldap, but group corgroup still contain attribute member: uid=bbuser2,cn=users,cn=accounts,<DOMAIN_DN>
6. run command "groups buser1"

Package info:
bash$ rpm -q sssd
sssd-1.9.2-82.7.el6_4.x86_64

Remote ldap server
bash$ rpm -q 389-ds-base
389-ds-base-1.3.0.5-1.fc18.x86_64

Could you please provide more informations. Which ldap server was used (version).Could you provide minimal LDIF dump of ldap database, which cause this bug? Sanitized sssd_LDAP.log will be also helpful.
Comment 20 Jakub Hrozek 2013-07-09 15:06:54 UTC 
Aaron, we are unable to reproduce the problem in-house. Can you please provide the info requested by Lukas? Otherwise I'm afraid we'd have to close this bug..
Comment 21 Jakub Hrozek 2013-07-18 08:07:42 UTC 
There was no update to this bugzilla in over a month and we need the information requested in comment #19 in order to come up with a fix. I'm sorry, but I'll close this bug report as CLOSED/INSUFFICIENT_DATA. 

We'll be glad to work with you again on providing a fix for this problem once we have the information. Feel free to reopen this bug report, then.
Comment 23 Najmuddin Chirammal 2013-08-05 15:01:18 UTC 
I am able to re-produce the issue, Here is the sample ldif file.

# luser1, People, gsslab.pnq.redhat.com
dn: uid=luser1,ou=People,dc=gsslab,dc=pnq,dc=redhat,dc=com
givenName: ldap
sn: user1
loginShell: /bin/bash
uidNumber: 10001
gidNumber: 1500
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
objectClass: posixAccount
uid: luser1
gecos: Test User
cn: ldap user1
homeDirectory: /home/luser1

# LinuxUsers, group, gsslab.pnq.redhat.com
dn: cn=LinuxUsers,ou=group,dc=gsslab,dc=pnq,dc=redhat,dc=com
uniqueMember: uid=luser1,ou=People,dc=gsslab,dc=pnq,dc=redhat,dc=com
uniqueMember: uid=luser120,ou=People,dc=gsslab,dc=pnq,dc=redhat,dc=com
gidNumber: 25000
objectClass: top
objectClass: groupofuniquenames
objectClass: posixgroup
cn: LinuxUsers

----------------
# id luser1
uid=10001(luser1) gid=1500(DefaultGroup) groups=1500(DefaultGroup),25000(LinuxUsers)
Aug  5 20:19:58 dhcp210-111 kernel: sssd_be[29961]: segfault at 30 ip 00007ff05e7d96b3 sp 00007fff72e56728 error 4 in libtevent.so.0.9.17[7ff05e7d6000+c000]

----------------

Note: uid=luser120,ou=People,dc=gsslab,dc=pnq,dc=redhat,dc=com does not exist in ldap server.

The crash happens almost 90% of the time, the only thing I noticed is it occurs only if "ldap_group_search_base (ou=group,$suffix)" is not specified in sssd.conf.
Comment 24 Najmuddin Chirammal 2013-08-05 15:16:34 UTC 
Sorry.. there was a typo. I menat to say, "the issue occurs only if ldap_group_search_base (ou=group,$suffix)" is *specified* in sssd.conf"
Comment 25 Jakub Hrozek 2013-08-06 20:17:38 UTC 
Hi,

thanks again for working with us to provide a reproducer. I've built test RPMs, if the customer would like to try them out:
https://brewweb.devel.redhat.com/taskinfo?taskID=6133848
Comment 26 Jakub Hrozek 2013-08-07 22:48:44 UTC 
Fixed upstream, please dev_ack.
Comment 27 Jakub Hrozek 2013-08-08 08:18:18 UTC 
*** Bug 987536 has been marked as a duplicate of this bug. ***
Comment 29 Jakub Hrozek 2013-08-08 09:07:01 UTC 
*** Bug 969652 has been marked as a duplicate of this bug. ***
Comment 31 Nirupama Karandikar 2013-08-26 14:00:57 UTC 
Verified with version sssd-1.9.2-123.el6.x86_64

Output from beaker automation run:
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: dangling-member_01: BZ 963235
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

adding new entry "uid=sssduser1,ou=Users,dc=example,dc=com"

adding new entry "cn=dang1,ou=Groups,dc=example,dc=com"

Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
[  OK  ]
uid=10011(sssduser1) gid=10011 groups=10011,10016(dang1)
:: [   PASS   ] :: sssd_be did not crash 
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
[  OK  ]
uid=10011(sssduser1) gid=10011 groups=10011,10016(dang1)
:: [   PASS   ] :: sssd_be did not crash 
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
[  OK  ]
uid=10011(sssduser1) gid=10011 groups=10011,10016(dang1)
:: [   PASS   ] :: sssd_be did not crash 
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
[  OK  ]
uid=10011(sssduser1) gid=10011 groups=10011,10016(dang1)
:: [   PASS   ] :: sssd_be did not crash 
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
[  OK  ]
uid=10011(sssduser1) gid=10011 groups=10011,10016(dang1)
:: [   PASS   ] :: sssd_be did not crash
Comment 33 errata-xmlrpc 2013-11-21 22:17:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1680.html