Bug 1367802

Summary: using overides causes segfault in libldb
Product: Red Hat Enterprise Linux 6 Reporter: Jakub Hrozek <jhrozek>
Component: sssdAssignee: SSSD Maintainers <sssd-maint>
Status: CLOSED ERRATA QA Contact: Steeve Goveas <sgoveas>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.0CC: apeetham, grajaiya, jhrozek, lslebodn, mkosek, mniranja, mzidek, pbrezina, sgoveas
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sssd-1.13.3-45.el6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-21 09:57:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Simple LD_PRELOAD for reproducing a crash none

Description Jakub Hrozek 2016-08-17 14:06:33 UTC 
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/sssd/ticket/3118

testing sssd with overrides (users and groups) on centos 7.2 (sssd 1.13.0, libldb 1.1.25), i soon ran in to problems with sssd_nss crashing. dmesg shows:

  sssd_nss[28935]: segfault at 51 ip 00007fa5e39d46af sp 00007ffcd6f18290 error 4 in libldb.so.1.1.25[7fa5e39c4000+2d000]

backtrace:

{{{
  Program received signal SIGSEGV, Segmentation fault.
  0x00007f4b514276af in ldb_dn_from_ldb_val (mem_ctx=mem_ctx@entry=0x7f4b52297300, ldb=0x7f4b5229dad0, strdn=0x25) at ../common/ldb_dn.c:97
  97              if (strdn && strdn->data
  (gdb) bt
  #0  0x00007f4b514276af in ldb_dn_from_ldb_val (mem_ctx=mem_ctx@entry=0x7f4b52297300, ldb=0x7f4b5229dad0, strdn=0x25) at ../common/ldb_dn.c:97
  #1  0x00007f4b5188109a in sysdb_add_group_member_overrides (domain=domain@entry=0x7f4b52296120, obj=0x7f4b522a3e40) at src/db/sysdb_views.c:1308
  #2  0x00007f4b5187373c in sysdb_getgrgid_with_views (mem_ctx=mem_ctx@entry=0x7f4b52295ea0, domain=domain@entry=0x7f4b52296120, gid=65751, res=res@entry=0x7f4b522a3260) at src/db/sysdb_search.c:659
  #3  0x00007f4b51ef292c in nss_cmd_getgrgid_search (dctx=dctx@entry=0x7f4b522a3240) at src/responder/nss/nsssrv_cmd.c:3349
  #4  0x00007f4b51ef672d in nss_cmd_getbyid (cmd=<optimized out>, cctx=0x7f4b522a0900) at src/responder/nss/nsssrv_cmd.c:1975
  #5  0x00007f4b51f01b2e in client_cmd_execute (sss_cmds=0x7f4b521182e0 <nss_cmds>, cctx=0x7f4b522a0900) at src/responder/common/responder_common.c:249
  #6  client_recv (cctx=0x7f4b522a0900) at src/responder/common/responder_common.c:283
  #7  client_fd_handler (ev=<optimized out>, fde=<optimized out>, flags=<optimized out>, ptr=<optimized out>) at src/responder/common/responder_common.c:335
  #8  0x00007f4b4e15bd0b in epoll_event_loop_once () from /lib64/libtevent.so.0
  #9  0x00007f4b4e15a1d7 in std_event_loop_once () from /lib64/libtevent.so.0
  #10 0x00007f4b4e15636d in _tevent_loop_once () from /lib64/libtevent.so.0
  #11 0x00007f4b4e15650b in tevent_common_loop_wait () from /lib64/libtevent.so.0
  #12 0x00007f4b4e15a177 in std_event_loop_wait () from /lib64/libtevent.so.0
  #13 0x00007f4b51891553 in server_loop (main_ctx=0x7f4b5228d2a0) at src/util/server.c:668
  #14 0x00007f4b51eeaf77 in main (argc=8, argv=<optimized out>) at src/responder/nss/nsssrv.c:626
}}}

running sssd with -d 9, and having added ldb tracing to src/db/sysdb.c (line 59 in the unpatched source):

  ret = ldb_connect(ldb, filename, LDB_FLG_ENABLE_TRACING, NULL);

things die just after retrieving the override information for a member of a group, e.g. (username/domain name removed):

{{{
  [sssd[nss]] [ldb] (0x4000): ldb_trace_response: ENTRY
  dn: overrideAnchorUUID=:LOCAL:name\3Dusername\,cn\3Dusers\,cn\3Ddomainname\,cn\3Dsysdb,cn=LOCAL,cn=views,cn=sysdb
  loginShell: /bin/tcsh
  name: username
  objectClass: userOverride
  overrideObjectDN: name=username,cn=users,cn=domainname,cn=sysdb
  uidNumber: 6272

  [sssd[nss]] [ldb] (0x4000): Destroying timer event 0x7f4b522a3cc0 "ltdb_timeout"
  [sssd[nss]] [ldb] (0x4000): Ending timer event 0x7f4b522ae710 "ltdb_callback"
  [sssd[nss]] [sysdb_add_group_member_overrides] (0x4000): Added [username] to [overridememberUid].
}}}

it always seems to fail on the first or second member of the group, and it is always when the group being looked at has overrides (gid).

when dropping back to the previous ldb packages for centos 7.2 (ldb 1.1.20) everything seems to work just fine, so i looked at the differences, and it seems that this change, which was added in ldb 1.1.24 might be significant:

{{{
  --- ldb-1.1.20/ldb_tdb/ldb_search.c     2014-09-16 19:04:31.000000000 +0100                    
  +++ ldb-1.1.25/ldb_tdb/ldb_search.c     2015-12-10 11:01:40.000000000 +0000                    
  @@ -407,10 +407,18 @@                                                                          
          }                                                                                      
                                                                                                 
          talloc_free(msg->elements);                                                            
  -       msg->elements = talloc_realloc(msg, el2, struct ldb_message_element, msg->num_elements);
  +                                                                                               
  +       if (num_elements > 0) {                                                                 
  +               msg->elements = talloc_realloc(msg, el2, struct ldb_message_element,            
  +                                              num_elements);                                   
  +       } else {                                                                                
  +               msg->elements = talloc_array(msg, struct ldb_message_element, 0);               
  +               talloc_free(el2);                                                               
  +       }                                                                                       
          if (msg->elements == NULL) {                                                            
                  return -1;                                                                      
          }
}}}
reverting this changes stops things from crashing, as does just adding 1 to num_elements in the talloc_realloc call, e.g.:

{{{
  --- ldb-1.1.25/ldb_tdb/ldb_search.c     2015-12-10 11:01:40.000000000 +0000
  +++ ldb-1.1.25.test/ldb_tdb/ldb_search.c        2016-08-02 16:37:01.823488833 +0100
  @@ -410,7 +410,7 @@
  
          if (num_elements > 0) {
                  msg->elements = talloc_realloc(msg, el2, struct ldb_message_element,
  -                                              num_elements);
  +                                              num_elements+1);
}}}
i have had a bit of a poke around, but can't say i have been able to work out exactly why this is the case ...

i would like to have been able to give a better report of the exact cause of the problem, but have unfortunately run out of time to look at this for now.

at the moment, i can stick with ldb-1.1.20, but that's not really a long term solution. i did also do some quick testing with sssd-1.14.0, and the problem remains.

let me know if i can provide any more information.

thanks,

richard
Comment 4 Lukas Slebodnik 2016-11-07 14:26:34 UTC 
Created attachment 1218032 [details]
Simple LD_PRELOAD for reproducing a crash

How to reproduce a crash:
* compile a module
  gcc -Wall -fPIC -shared -o slow_realloc.so slow_realloc.c -ldl
* cp slow_realloc.so /usr/lib64/slow_realloc.so
* Use this module in sssd
  echo 'export LD_PRELOAD=/usr/lib64/slow_realloc.so' >> /etc/sysconfig/sssd
* run test for local_overrides
Comment 6 Lukas Slebodnik 2016-11-08 09:37:41 UTC 
sssd-1-13:
* 55fc0bb19e6205af13828a98592b283d3b6d24e0
* 19ba10fcc7dbdfdd7a238fa94f57605cf16fc28e
* 8e19dce22b286f1f815cba7150149ab249a62854
* 5d64343d5ffed9cb42184eb30e5bf1871d8196d5
* ce714745ad28dfb6dcfd9f8f8983e492661a6e2f
* 3bea6818a3432a349a9901a84fd517c052b19f69
Comment 8 Niranjan Mallapadi Raghavender 2016-12-13 16:49:19 UTC 
Versions:

Reproducer:
============
sssd-common-pac-1.13.3-22.el6.x86_64
sssd-ad-1.13.3-22.el6.x86_64
sssd-tools-1.13.3-22.el6.x86_64
sssd-client-1.13.3-22.el6.x86_64
sssd-common-1.13.3-22.el6.x86_64
sssd-proxy-1.13.3-22.el6.x86_64
sssd-krb5-common-1.13.3-22.el6.x86_64
sssd-ipa-1.13.3-22.el6.x86_64
sssd-krb5-1.13.3-22.el6.x86_64
python-sssdconfig-1.13.3-22.el6.noarch
sssd-qe-tests-sssd-rhel68-libs-mniranja-20161213172256-0.noarch
sssd-ldap-1.13.3-22.el6.x86_64
sssd-1.13.3-22.el6.x86_64

sssd.conf
=========

[sssd]
config_file_version = 2
services = nss, pam
domains = LDAP

[nss]
filter_groups = root
filter_users = root

[pam]

[domain/LDAP]
debug_level=0xFFF0
id_provider = ldap
ldap_uri = ldap://vm-idm-011.lab.eng.pnq.redhat.com
ldap_search_base = dc=example,dc=com
ldap_tls_cacert = /etc/openldap/certs/cacert.asc


Ldap server:
OS: RHEL7.3 
Red Hat Directory Server 

[root@auto-hv-02-guest03 mniranja]# py.test --with-beakerlib local_overrides.py -s -v
============================================================================================================ test session starts =============================================================================================================
platform linux2 -- Python 2.7.5, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /opt/rh/python27/root/usr/bin/python2
cachedir: .cache
rootdir: /mnt/tests/sssd/rhel68/client/ldap_provider/local_overrides/mniranja, inifile:
plugins: beakerlib-0.6
collected 12 items 

local_overrides.py::test_simple_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_root_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_replace_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_remove_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_imp_exp_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
user1@LDAP:ov_user1:10010:20010:Overridden User 1:/home/ov/user1:/bin/ov_user1_shell
user2@LDAP:ov_user2:10020:20020:Overridden User 2:/home/ov/user2:/bin/ov_user2_shell
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_simple_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_root_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_replace_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_remove_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_imp_exp_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_regr_2757_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_regr_2790_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
FAILEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]



sssd crashed 

Process /usr/libexec/sssd/sssd_nss was killed by signal 11 (SIGSEGV)

dmesg:
sssd_nss[3595] general protection ip:39d021219f sp:7ffd83642260 error:0 in libldb.so.1.1.25[39d0200000+2d000]



Update sssd to latest 

sssd-ldap-1.13.3-52.el6.x86_64
sssd-krb5-common-1.13.3-52.el6.x86_64
sssd-ad-1.13.3-52.el6.x86_64
sssd-krb5-1.13.3-52.el6.x86_64
sssd-proxy-1.13.3-52.el6.x86_64
sssd-ipa-1.13.3-52.el6.x86_64
sssd-tools-1.13.3-52.el6.x86_64
python-sssdconfig-1.13.3-52.el6.noarch
sssd-common-1.13.3-52.el6.x86_64
sssd-common-pac-1.13.3-52.el6.x86_64
sssd-1.13.3-52.el6.x86_64
sssd-client-1.13.3-52.el6.x86_64


[root@auto-hv-02-guest03 mniranja]# py.test --with-beakerlib local_overrides.py -s -v
=================================================================== test session starts ===========================================
platform linux2 -- Python 2.7.5, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /opt/rh/python27/root/usr/bin/python2
cachedir: .cache
rootdir: /mnt/tests/sssd/rhel68/client/ldap_provider/local_overrides/mniranja, inifile:
plugins: beakerlib-0.6
collected 12 items 

local_overrides.py::test_simple_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_root_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_replace_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_remove_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown
Stopping sssd: [  OK  ]

local_overrides.py::test_imp_exp_user_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
user1@LDAP:ov_user1:10010:20010:Overridden User 1:/home/ov/user1:/bin/ov_user1_shell
user2@LDAP:ov_user2:10020:20020:Overridden User 2:/home/ov/user2:/bin/ov_user2_shell
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]

local_overrides.py::test_simple_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]

local_overrides.py::test_root_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]

local_overrides.py::test_replace_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]

local_overrides.py::test_remove_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]

local_overrides.py::test_imp_exp_group_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]

local_overrides.py::test_regr_2757_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]

local_overrides.py::test_regr_2790_override Starting sssd: [  OK  ]
SSSD needs to be restarted for the changes to take effect.
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
PASSEDcreate_sssd_fixture.teardown  
Stopping sssd: [  OK  ]


================================ 12 passed in 115.10 seconds ======================
[root@auto-hv-02-guest03 mniranja]#
Comment 10 errata-xmlrpc 2017-03-21 09:57:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0632.html