Bug 1461437

Summary: crash in send_ldap_result
Product: Red Hat Enterprise Linux 7 Reporter: German Parente <gparente>
Component: 389-ds-baseAssignee: mreynolds
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.4CC: msauton, nkinder, rmeggins, tbordaz
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.7.5-4.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1489694 (view as bug list) Environment:
Last Closed: 2018-04-10 14:16:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1489694    

Description German Parente 2017-06-14 12:55:29 UTC
Description of problem:

we have seen a server crash at customer site that when sending the result of an operation.

The full stacktrace is:

The whole backtrace:

#0  0x00007fa579a22711 in send_ldap_result_ext (pb=pb@entry=0x7fa528007550, err=err@entry=53, matched=matched@entry=0x0, 
    text=text@entry=0x7fa579a4c708 "This plugin is not configured to access operation target data", nentries=nentries@entry=0, urls=urls@entry=0x0, ber=ber@entry=0x0) at ldap/servers/slapd/result.c:353
#1  0x00007fa579a22fd1 in send_ldap_result (pb=pb@entry=0x7fa528007550, err=err@entry=53, matched=matched@entry=0x0, 
    text=text@entry=0x7fa579a4c708 "This plugin is not configured to access operation target data", nentries=nentries@entry=0, urls=urls@entry=0x0) at ldap/servers/slapd/result.c:194
#2  0x00007fa579a13260 in slapi_search_internal_callback_pb (pb=pb@entry=0x7fa528007550, callback_data=callback_data@entry=0x7fa53bfe5ea0, prc=prc@entry=0x0, 
    psec=psec@entry=0x7fa56a802df0 <backend_shr_note_entry_sdn_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:559
#3  0x00007fa56a804a61 in backend_shr_update_references_cb (group=0x7fa520007d50 "cn=compat,dc=mlbam,dc=net", set=0x7fa52000aa50 "cn=groups", flag=<optimized out>, backend_data=0x7fa52392daa0, 
    cbdata_ptr=<optimized out>) at back-shr.c:1539
#4  0x00007fa56a81203f in map_data_foreach_map (state=state@entry=0x7fa57b585a30, domain_name=domain_name@entry=0x0, fn=fn@entry=0x7fa56a803fd0 <backend_shr_update_references_cb>, 
    cbdata=cbdata@entry=0x7fa53bfe5f50) at map.c:347
#5  0x00007fa56a80279b in backend_shr_update_references (state=0x7fa57b585a30, pb=pb@entry=0x7fa53bfe6a90, e=<optimized out>, mods=mods@entry=0x0, modlist=modlist@entry=0x0) at back-shr.c:1825
#6  0x00007fa56a803a17 in backend_shr_add_cb (pb=pb@entry=0x7fa53bfe6a90) at back-shr.c:1953
#7  0x00007fa56a803b41 in backend_shr_add_cb (pb=0x7fa53bfe6a90) at back-shr.c:1883
#8  backend_shr_betxn_post_add_cb (pb=0x7fa53bfe6a90) at back-shr.c:1965
#9  0x00007fa579a0db18 in plugin_call_func (list=0x7fa57b58e910, operation=operation@entry=560, pb=pb@entry=0x7fa53bfe6a90, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#10 0x00007fa579a0dda3 in plugin_call_list (pb=0x7fa53bfe6a90, operation=560, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#11 plugin_call_plugins (pb=pb@entry=0x7fa53bfe6a90, whichfunction=whichfunction@entry=560) at ldap/servers/slapd/plugin.c:445
#12 0x00007fa56c7b9b94 in ldbm_back_add (pb=0x7fa53bfe6a90) at ldap/servers/slapd/back-ldbm/ldbm_add.c:1150
#13 0x00007fa5799b1ed8 in op_shared_add (pb=pb@entry=0x7fa53bfe6a90) at ldap/servers/slapd/add.c:699
#14 0x00007fa5799b31a0 in do_add (pb=pb@entry=0x7fa53bfe6a90) at ldap/servers/slapd/add.c:226
#15 0x00007fa579ee59c3 in connection_dispatch_operation (pb=0x7fa53bfe6a90, op=0x7fa57b7521b0, conn=0x7fa57bc05118) at ldap/servers/slapd/connection.c:612
#16 connection_threadmain () at ldap/servers/slapd/connection.c:1759
#17 0x00007fa577bdb9bb in _pt_root (arg=0x7fa57bbcf3d0) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#18 0x00007fa57757bdc5 in start_thread (arg=0x7fa53bfe7700) at pthread_create.c:308
#19 0x00007fa5772aa76d in __lseek_nocancel () at ../sysdeps/unix/syscall-template.S:81
#20 0x0000000000000000 in ?? ()

The add operation provokes a compat plugin post operation.

When returning to server the operation is NULL:

#0  0x00007fa579a22711 in send_ldap_result_ext (pb=pb@entry=0x7fa528007550, err=err@entry=53, matched=matched@entry=0x0, 
    text=text@entry=0x7fa579a4c708 "This plugin is not configured to access operation target data", nentries=nentries@entry=0, urls=urls@entry=0x0, ber=ber@entry=0x0) at ldap/servers/slapd/result.c:353
353		if (operation->o_status == SLAPI_OP_STATUS_RESULT_SENT) {

But operation is NULL:

 print operation
$1 = (Slapi_Operation *) 0x0

As we are not checking if operation is null before accessing its filed o_status, the server crashes:

=============================================
in send_ldap_intermediate:

        slapi_pblock_get (pb, SLAPI_OPERATION, &operation);
        slapi_pblock_get (pb, SLAPI_CONNECTION, &connection);

        if (operation->o_status == SLAPI_OP_STATUS_RESULT_SENT) {
                return(rc); /* result already sent */
        }
=============================================


Version-Release number of selected component (if applicable):
      389-ds-base-1.3.5.10-21.el7_3.x86_64

How reproducible: rarely !

Expected results:

Operation should not be NULL but a check could prevent the crash.

Comment 2 thierry bordaz 2017-06-14 16:21:29 UTC
Upstream ticket https://pagure.io/389-ds-base/issue/49291

Comment 3 thierry bordaz 2017-06-15 08:07:38 UTC
upstream slapi-nis ticket https://pagure.io/slapi-nis/issue/16 (improve robustness of slapi-nis)

Comment 4 thierry bordaz 2017-06-16 09:24:18 UTC
upstream slapi-nis ticket https://pagure.io/slapi-nis/issue/17 (format_add_sdn_list was called with empty sdn)

Comment 5 thierry bordaz 2017-06-23 15:41:33 UTC
As a follow up of investigations of https://pagure.io/slapi-nis/issue/17 


    - Because base_sdn_list and base_sdn_list2 contains the same SDN with NULL DN
      we know the NULL DN was added by calling format_add_sdn_list.

    - The NULL DN SDN was the last SDN of base_sdn_list, so it was append to the list

    - format_add_sdn_list is call in several places
      backend_shr_update_references_cb
      backend_shr_note_entry_sdn_cb
      format_referred_r_entry_cb
      format_maybe_add_sdn_list
      format_deref_rx
      format_referred_r


    - In backend_shr_update_references_cb:
      backend_shr_update_references_cb was not called with an entry with NULL dn
      In fact, 'cn=groups' maps contains a 'ignore_subtree' array.
      In such condition the entry DN is tested and if it is NULL,
      backend_shr_update_references_cb exits before calling format_add_sdn_list.
(gdb) print *(struct backend_shr_set_data *) backend_data
$2 = {state = 0x7fa57b585a30, group = 0x7fa521a75df0 "cn=compat,<suffix>", set = 0x7fa5238e28a0 "cn=groups", 
  bases = 0x7fa5238ea0c0, entry_filter = 0x7fa520008220 "(objectclass=posixGroup)", rel_attrs = 0x7fa523eae650, 
  rel_attr_list = 0x7fa528007860 "cn,ipaanchoruuid,gidNumber,memberUid,ipauniqueid,member,uid", rel_attrs_list = 0x7fa523eae650, 
  ref_attrs = 0x0, inref_attrs = 0x0, ref_attr_list = 0x7fa510582480, inref_attr_list = 0x0, skip_uninteresting_updates = 1, 
(gdb) print *((struct backend_shr_set_data *) backend_data)->ignore_subtrees[0]
$5 = {flag = 7 '\a', udn = 0x0, dn = 0x7fa521a53330 "cn=tasks,cn=config", ndn = 0x7fa510f8b240 "cn=tasks,cn=config", ndn_len = 18}
(gdb) print *((struct backend_shr_set_data *) backend_data)->ignore_subtrees[1]
Cannot access memory at address 0x0
      
      So the NULL DN SDN was not added by backend_shr_update_references_cb


    -  In backend_shr_note_entry_sdn_cb
      backend_shr_note_entry_sdn_cb is a callback of a search function.
      A NULL dn entry does not exist. An internal search would not call backend_shr_note_entry_sdn_cb with an NULL DN entry

    - In format_referred_r_entry_cb
      Idem as above

    - In format_maybe_add_sdn_list
      It does not call format_add_sdn_list on NULL dn entry

    - format_deref_rx
      This is possible when looping over parents entries

    - format_referred_r
      This is possible 
            if backend_get_set_config can return NULL DN SDN
            if it is call with an entry containing NULL DN


    In conclusion:
        The most probable place to introduce NULL DN SDN in the list are
            format_deref_rx (while looping other parents)
            format_referred_r (backend_get_set_config / called with NULL DN entry)

Comment 10 thierry bordaz 2017-09-19 07:31:57 UTC
Hi Viktor,

Unfortunately I think the easiest way to reproduce is to use/configure slapi-nis.
We need to have a plugin that triggers an internal search with a base search that is SDN containing a NULL DN.

I do not know if using slapi-nis is an acceptable option to test that bug. If it is you can use the test case described in https://pagure.io/slapi-nis/issue/17 (having a group with members (member attribute) with one of the 'member' value being empty).

If it is not acceptable I will need to dig in our standard plugins to check if one of them is doing an internal search without checking the base search SDN.

Comment 12 Viktor Ashirov 2017-11-21 15:04:41 UTC
Builds tested:
389-ds-base-1.3.7.5-10.el7.x86_64
slapi-nis-0.56.0-7.el7.x86_64

[1] To verify I configured slapi-nis plugin: 
# ldapadd -D "cn=Directory Manager" -w Secret123
dn: cn=NIS Server,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: NIS Server
nsslapd-pluginPath: /usr/lib64/dirsrv/plugins/nisserver-plugin.so
nsslapd-pluginInitfunc: nis_plugin_init
nsslapd-pluginType: postoperation
nsslapd-pluginEnabled: on
nsslapd-pluginDescription: NIS Server Plugin
nsslapd-pluginVendor: redhat.com
nsslapd-pluginVersion: 0.56 (betxn support available and enabled by default)
nsslapd-pluginId: nis-plugin
nis-tcp-wrappers-name: ypserv
nsslapd-pluginarg0: 541

dn: nis-domain=example.com+nis-map=groups,cn=NIS Server,cn=plugins,cn=config
objectClass: extensibleObject
objectClass: top
nis-domain: example.com
nis-map: groups
nis-base: ou=Groups, dc=example, dc=com
nis-filter: (objectClass=groupOfNames)
nis-key-format: %{cn}
nis-value-format: %merge(" ","%deref_f(\"member\",\"(objectclass=ipanisNetgroup)\",\"cn\")","(%link(\"%ifeq(\\\"hostCategory\\\",\\\"all\\\",\\\"\\\",\\\"%collect(\\\\\\\"%{externalHost}\\\\\\\",\\\\\\\"%deref(\\\\\\\\\\\\\\\"memberHost\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"fqdn\\\\\\\\\\\\\\\")\\\\\\\",\\\\\\\"%deref_r(\\\\\\\\\\\\\\\"member\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"fqdn\\\\\\\\\\\\\\\")\\\\\\\",\\\\\\\"%deref_r(\\\\\\\\\\\\\\\"memberHost\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"member\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"fqdn\\\\\\\\\\\\\\\")\\\\\\\")\\\")\",\"%ifeq(\\\"hostCategory\\\",\\\"all\\\",\\\"\\\",\\\"-\\\")\",\",\",\"%ifeq(\\\"userCategory\\\",\\\"all\\\",\\\"\\\",\\\"%collect(\\\\\\\"%deref(\\\\\\\\\\\\\\\"memberUser\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"uid\\\\\\\\\\\\\\\")\\\\\\\",\\\\\\\"%deref_r(\\\\\\\\\\\\\\\"member\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"uid\\\\\\\\\\\\\\\")\\\\\\\",\\\\\\\"%deref_r(\\\\\\\\\\\\\\\"memberUser\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"member\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"uid\\\\\\\\\\\\\\\")\\\\\\\")\\\")\",\"%ifeq(\\\"userCategory\\\",\\\"all\\\",\\\"\\\",\\\"-\\\")\"),%{nisDomainName:-})")

adding new entry "cn=NIS Server,cn=plugins,cn=config"

adding new entry "nis-domain=example.com+nis-map=groups,cn=NIS Server,cn=plugins,cn=config"

[2] Added test entries:

# ldapadd -D "cn=Directory Manager" -w Secret123
dn: cn=tuser,ou=People,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: organizationalPerson
objectClass: top
objectClass: person
cn: tuser
sn: tuser
uid: tuser
gidNumber: 2000
homeDirectory: /home/tuser
uidNumber: 2000

dn: cn=tgroup,ou=Groups,dc=example,dc=com
objectClass: groupofnames
objectClass: top
cn: tgroup
member:
member: cn=tuser,ou=People,dc=example,dc=com

adding new entry "cn=tuser,ou=People,dc=example,dc=com"

adding new entry "cn=tgroup,ou=Groups,dc=example,dc=com"


[3] Removed empty member attribute without server crashing:

# ldapmodify -D "cn=Directory Manager" -w Secret123
dn: cn=tgroup,ou=Groups,dc=example,dc=com
changetype: modify
replace: member
member: cn=tuser,ou=People,dc=example,dc=com

modifying entry "cn=tgroup,ou=Groups,dc=example,dc=com"

Marking as VERIFIED.

Comment 15 errata-xmlrpc 2018-04-10 14:16:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0811