Bug 1498773

Summary: Installation of ipa fails with crash in topology plugin
Product: Red Hat Enterprise Linux 7 Reporter: Lukas Slebodnik <lslebodn>
Component: 389-ds-baseAssignee: mreynolds
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.5CC: jpazdziora, nkinder, ppicka, pvoborni, rcritten, rmeggins, tbordaz, tscherf
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.7.5-5.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1511824 (view as bug list) Environment:
Last Closed: 2018-04-10 14:19:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1511824    

Description Lukas Slebodnik 2017-10-05 08:22:33 UTC
Description of problem:


Version-Release number of selected component (if applicable):
sh$ rpm -q ipa-server 389-ds-base
ipa-server-4.5.0-21.el7_4.1.2.x86_64
389-ds-base-1.3.7.5-4.el7.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
1. /usr/sbin/ipa-server-install --setup-dns --forwarder=$FORWARDER_IP --reverse-zone=$REVERSE.in-addr.arpa. --allow-zone-overlap --hostname=ipa.testrelm.test -r TESTRELM.TEST -n testrelm.test -p Password123 -a Password123 --ip-address=$IP_ADDRESS -U

Actual results:
//snip
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/45]: creating directory server instance
  [2/45]: enabling ldapi
  [3/45]: configure autobind for root
  [4/45]: stopping directory server
  [5/45]: updating configuration in dse.ldif
  [6/45]: starting directory server
  [7/45]: adding default schema
  [8/45]: enabling memberof plugin
  [9/45]: enabling winsync plugin
  [10/45]: configuring replication version plugin
  [11/45]: enabling IPA enrollment plugin
  [12/45]: configuring uniqueness plugin
  [13/45]: configuring uuid plugin
  [14/45]: configuring modrdn plugin
  [15/45]: configuring DNS plugin
  [16/45]: enabling entryUSN plugin
  [17/45]: configuring lockout plugin
  [18/45]: configuring topology plugin
  [19/45]: creating indices
  [20/45]: enabling referential integrity plugin
  [21/45]: configuring certmap.conf
  [22/45]: configure new location for managed entries
  [23/45]: configure dirsrv ccache
  [24/45]: enabling SASL mapping fallback
  [25/45]: restarting directory server
  [26/45]: adding sasl mappings to the directory
  [27/45]: adding default layout
  [28/45]: adding delegation layout
  [29/45]: creating container for managed entries
  [30/45]: configuring user private groups
  [31/45]: configuring netgroups from hostgroups
  [32/45]: creating default Sudo bind user
  [33/45]: creating default Auto Member layout
  [34/45]: adding range check plugin
  [35/45]: creating default HBAC rule allow_all
  [36/45]: adding entries for topology management
  [37/45]: initializing group membership
  [38/45]: adding master entry
  [39/45]: initializing domain level
  [40/45]: configuring Posix uid/gid generation
ipa         : CRITICAL Failed to load dna.ldif: Command '/usr/bin/ldapmodify -v -f /tmp/tmpSzZXI_ -H ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket -Y EXTERNAL' returned non-zero exit status 255
  [error] CalledProcessError: Command '/usr/bin/ldapmodify -v -f /tmp/tmpSzZXI_ -H ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket -Y EXTERNAL' returned non-zero exit status 255
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Command '/usr/bin/ldapmodify -v -f /tmp/tmpSzZXI_ -H ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket -Y EXTERNAL' returned non-zero exit status 255
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

Expected results:
installation passed without any problem

Additional info:
Crash in 389-ds
#0  0x00007f411e5ee939 in __strcasecmp_l_sse2 () at ../sysdeps/x86_64/strcmp.S:212
#1  0x00007f41151e5761 in ipa_topo_check_segment_updates (pb=pb@entry=0x556e1722e0a0) at topology_pre.c:393
#2  0x00007f41151e5c78 in ipa_topo_pre_mod (pb=0x556e1722e0a0) at topology_pre.c:573
#3  0x00007f41216c1f74 in plugin_call_func (list=0x556e171ee100, operation=operation@entry=451, pb=pb@entry=0x556e1722e0a0, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2028
#4  0x00007f41216c2203 in plugin_call_list (pb=0x556e1722e0a0, operation=451, list=<optimized out>) at ldap/servers/slapd/plugin.c:1972
#5  0x00007f41216c2203 in plugin_call_plugins (pb=pb@entry=0x556e1722e0a0, whichfunction=whichfunction@entry=451) at ldap/servers/slapd/plugin.c:442
#6  0x00007f412167c00a in dse_modify (pb=0x556e1722e0a0) at ldap/servers/slapd/dse.c:1751
#7  0x00007f41216ae456 in op_shared_modify (pb=pb@entry=0x556e1722e0a0, pw_change=pw_change@entry=0, old_pw=0x0) at ldap/servers/slapd/modify.c:1013
#8  0x00007f41216af8eb in do_modify (pb=pb@entry=0x556e1722e0a0) at ldap/servers/slapd/modify.c:383
#9  0x0000556e159e32b7 in connection_dispatch_operation (pb=0x556e1722e0a0, op=0x556e1732e000, conn=0x556e17d66050) at ldap/servers/slapd/connection.c:624
#10 0x0000556e159e32b7 in connection_threadmain () at ldap/servers/slapd/connection.c:1761
#11 0x00007f411f60dc1b in _pt_root (arg=0x556e17d22900) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#12 0x00007f411efade25 in start_thread (arg=0x7f40f7a4e700) at pthread_create.c:308
#13 0x00007f411e65c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Comment 3 Lukas Slebodnik 2017-10-05 08:25:05 UTC
Part of ipa

2017-10-04T20:45:31Z DEBUG   [40/45]: configuring Posix uid/gid generation
2017-10-04T20:45:31Z DEBUG Starting external process
2017-10-04T20:45:31Z DEBUG args=/usr/bin/ldapmodify -v -f /tmp/tmpSzZXI_ -H ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket -Y EXTERNAL
2017-10-04T20:45:31Z DEBUG Process finished, return code=255
2017-10-04T20:45:31Z DEBUG stdout=add objectclass:
        top
        extensibleObject
add cn:
        Posix IDs
add dnaType:
        uidNumber
        gidNumber
add dnaNextValue:
        1861400000
add dnaMaxValue:
        1861599999
add dnaMagicRegen:
        -1
add dnaFilter:
        (|(objectClass=posixAccount)(objectClass=posixGroup)(objectClass=ipaIDobject))
add dnaScope:
        dc=testrelm,dc=test
add dnaThreshold:
        500
add dnaSharedCfgDN:
        cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=test
add dnaExcludeScope:
        cn=provisioning,dc=testrelm,dc=test
adding new entry "cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config"
modify complete

replace nsslapd-pluginEnabled:
        on
modifying entry "cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config"


2017-10-04T20:45:31Z DEBUG stderr=ldap_initialize( ldapi://%2Fvar%2Frun%2Fslapd-TESTRELM-TEST.socket/??base )
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
ldap_result: Can't contact LDAP server (-1)
2017-10-04T20:45:31Z CRITICAL Failed to load dna.ldif: Command '/usr/bin/ldapmodify -v -f /tmp/tmpSzZXI_ -H ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket -Y EXTERNAL' returned non-zero exit status 255
2017-10-04T20:45:31Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 704, in __config_uidgid_gen
    self._ldap_mod("dna.ldif", self.sub_dict)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 308, in _ldap_mod
    ipautil.run(args, nolog=nologlist)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))
CalledProcessError: Command '/usr/bin/ldapmodify -v -f /tmp/tmpSzZXI_ -H ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket -Y EXTERNAL' returned non-zero exit status 255

2017-10-04T20:45:31Z DEBUG   [error] CalledProcessError: Command '/usr/bin/ldapmodify -v -f /tmp/tmpSzZXI_ -H ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket -Y EXTERNAL' returned non-zero exit status 255

Comment 5 thierry bordaz 2017-10-05 09:43:00 UTC
Debug details:


#
# the crash occurs while enabling DNA plugin, topology plugin catches the update
#
(gdb) where
#0  0x00007f411e5ee939 in __strcasecmp_l_sse2 () at ../sysdeps/x86_64/strcmp.S:212
#1  0x00007f41151e5761 in ipa_topo_check_segment_updates (pb=pb@entry=0x556e1722e0a0) at topology_pre.c:393
#2  0x00007f41151e5c78 in ipa_topo_pre_mod (pb=0x556e1722e0a0) at topology_pre.c:573
#3  0x00007f41216c1f74 in plugin_call_func (list=0x556e171ee100, operation=operation@entry=451, pb=pb@entry=0x556e1722e0a0, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2028
#4  0x00007f41216c2203 in plugin_call_list (pb=0x556e1722e0a0, operation=451, list=<optimized out>)
    at ldap/servers/slapd/plugin.c:1972
#5  0x00007f41216c2203 in plugin_call_plugins (pb=pb@entry=0x556e1722e0a0, whichfunction=whichfunction@entry=451)
    at ldap/servers/slapd/plugin.c:442
#6  0x00007f412167c00a in dse_modify (pb=0x556e1722e0a0) at ldap/servers/slapd/dse.c:1751
#7  0x00007f41216ae456 in op_shared_modify (pb=pb@entry=0x556e1722e0a0, pw_change=pw_change@entry=0, old_pw=0x0)
    at ldap/servers/slapd/modify.c:1013
#8  0x00007f41216af8eb in do_modify (pb=pb@entry=0x556e1722e0a0) at ldap/servers/slapd/modify.c:383
#9  0x0000556e159e32b7 in connection_dispatch_operation (pb=0x556e1722e0a0, op=0x556e1732e000, conn=0x556e17d66050)
    at ldap/servers/slapd/connection.c:624
#10 0x0000556e159e32b7 in connection_threadmain () at ldap/servers/slapd/connection.c:1761
#11 0x00007f411f60dc1b in _pt_root (arg=0x556e17d22900) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#12 0x00007f411efade25 in start_thread (arg=0x7f40f7a4e700) at pthread_create.c:308
#13 0x00007f411e65c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113


#
# Topology plugin retrieves the SLAPI_PLUGIN_IDENTITY to be sure not
# process its own updates
#
# it uses 'pi' that is not initialized to retrieve SLAPI_PLUGIN_IDENTITY value
#
(gdb) frame 1
#1  0x00007f41151e5761 in ipa_topo_check_segment_updates (pb=pb@entry=0x556e1722e0a0) at topology_pre.c:393
393	    if (pi && 0 == strcasecmp(pi, ipa_topo_get_plugin_id())) {
(gdb) list 382,393
382	int
383	ipa_topo_check_segment_updates(Slapi_PBlock *pb)
384	{
385	    int rc = 0;
386	    Slapi_Entry *mod_entry;
387	    char *pi;
388	
389	    /* we have to check if the operation is triggered by the
390	     * topology plugin itself - allow it
391	     */
392	    slapi_pblock_get(pb, SLAPI_PLUGIN_IDENTITY,&pi);
393	    if (pi && 0 == strcasecmp(pi, ipa_topo_get_plugin_id())) {

#
# https://pagure.io/389-ds-base/issue/49097 split pblocks into
# several sub-structs
#
#  pb_intplugin that contains SLAPI_PLUGIN_IDENTITY is allocated
# on demand only if one of its fields is set.
#
# Here the field was not slapi-pblock-set before slapi-pblock-get
#
(gdb) print *pb
$25 = {pb_backend = 0x556e16fd7dc0, pb_conn = 0x556e17d66050, pb_op = 0x556e1732e000, pb_plugin = 0x556e171ee100, pb_dse = 0x0, 
  pb_task = 0x0, pb_mr = 0x0, pb_misc = 0x556e1801bf00, pb_intop = 0x556e16fd4ea0, pb_intplugin = 0x0, pb_deprecated = 0x0}


#
# slapi_pblock_get leaves the variable unchanged if the
# sub-struct that contains the requested field is not allocate
#
# the consequence is that 'pi' is keeping the value it got from the stack
(gdb) list slapi_pblock_get
353	#define SLAPI_PBLOCK_GET_PLUGIN_RELATED_POINTER(pb, element) \
354	    ((pb)->pb_plugin == NULL ? NULL : (pb)->pb_plugin->element)
355	
356	int
357	slapi_pblock_get(Slapi_PBlock *pblock, int arg, void *value)
358	{
359	
360	#ifdef PBLOCK_ANALYTICS
361	    pblock_analytics_record(pblock, arg);
362	#endif

(gdb) list 703
698	        }
699	        break;
700	    case SLAPI_PLUGIN_DESCRIPTION:
701	        (*(Slapi_PluginDesc *)value) = pblock->pb_plugin->plg_desc;
702	        break;
703	    case SLAPI_PLUGIN_IDENTITY:
704	        if (pblock->pb_intplugin != NULL) {
705	            (*(void **)value) = pblock->pb_intplugin->pb_plugin_identity;
706	        }
707	        break;


Before, pblock was a calloc struct. Get of a unset field always set the variable &pi.
So the API behavior changed.

Others plugins may hit that issue (accessing a field that has not been previously set). This is a 389-ds bug. slapi-pblock-get should always set the variable, even if the sub-struct is not allocated.

Comment 6 thierry bordaz 2017-10-05 10:59:30 UTC
Upstream ticket https://pagure.io/389-ds-base/issue/49394

Comment 7 Rob Crittenden 2017-10-13 12:32:49 UTC
*** Bug 1499179 has been marked as a duplicate of this bug. ***

Comment 8 thierry bordaz 2017-10-16 15:56:31 UTC
Upstream ticket is pushed -> POST

Comment 13 Viktor Ashirov 2017-11-21 13:34:22 UTC
Build tested:
389-ds-base-1.3.7.5-10.el7.x86_64
ipa-server-4.5.4-4.el7.x86_64

I did ipa-server-install several times and didn't observe a crash.

Marking as VERIFIED, SanityOnly.

Comment 16 errata-xmlrpc 2018-04-10 14:19:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0811