Bug 1264224

Hi Noriko,

Is that a reliably reproducible crash? If so, what are the steps to reproduce?

Thanks!

(In reply to Viktor Ashirov from comment #1)
> Hi Noriko,
> 
> Is that a reliably reproducible crash? If so, what are the steps to
> reproduce?

Hi Viktor,

It requires freeipa and slapi-nis/compat plug-in.  More precisely, this function backend_bind_cb(Slapi_PBlock *pb) needs to be called with the following condition.
 * 2. If bind target DN exists in LDAP store, its map cache entry
 *    will have orginal entry DN recorded. Enforcing SLAPI_BIND_TARGET_SDN
 *    to it will force other plugins to handle authentication request against
 *    the original because slapi-nis' map cache entry doesn't have paswords
 *    recorded. To make it working, slapi-nis should be registered with higher
 *    plugin ordering priority than other plugins.
As described in this comment, SLAPI_BIND_TARGET_SDN is reset in the pblock which is the cause of the crash.

I think we need this setup/scenario to be translated by the slapi-nis expert...

Alexander, could you please help us?  Thanks!

Alexander, could you please help us on the reproducer described in Comment 2?
Thank you!!

Noriko,

I think this also is similar to what Simo was doing with support for non-DN binds (like AD does with sAMAccountName value), i.e. 389-ds needs indeed to refresh SDN value after the plugins' run.

I think your patch is roughly correct. I wonder, though, what causes this crash because to me any rebinding that slapi-nis does works fine. Perhaps, another thread did remove the user at the same time as somebody tried to bind to compat entry of that user? I.e. the entry did exist at the bind time in slapi-nis map cache but disappeared from the primary source in a separate thread?

(In reply to Alexander Bokovoy from comment #4)
> Noriko,
> 
> I think this also is similar to what Simo was doing with support for non-DN
> binds (like AD does with sAMAccountName value), i.e. 389-ds needs indeed to
> refresh SDN value after the plugins' run.
> 
> I think your patch is roughly correct. I wonder, though, what causes this
> crash because to me any rebinding that slapi-nis does works fine. Perhaps,
> another thread did remove the user at the same time as somebody tried to
> bind to compat entry of that user? I.e. the entry did exist at the bind time
> in slapi-nis map cache but disappeared from the primary source in a separate
> thread?

Thanks, Alexander.  I pushed the fix (enhanced some more by the comments by Rich) to upstream.
https://fedorahosted.org/389/ticket/48188#comment:26

I'm building the rhel-7.2 candidate with the patch from now.  The patches prevent the crash caused by the replaced sdn in the pre-op bind plug-in such as slapi-nis.  And if the entry is deleted, the following backend bind code (ldbm_back_bind) fails like this.
    /*
     * find the target entry.  find_entry() takes care of referrals
     *   and sending errors if the entry does not exist.
     */
    if (( e = find_entry( pb, be, addr, &txn )) == NULL ) {
        rc = SLAPI_BIND_FAIL;
        goto bail;
    }
Hopefully, it works (bind fails) as expected.

That said, the reproducer would be binding as a user and simultaneous (deleting and adding the user) in a loop?  Could you please tell me to enable slapi-nis, what option needs to be set to ipa-server-install?  (sorry about this too basic question...)  Thanks!!

Noriko,

please use 'ipa-compat-manage status|enable|disable' after you installed the IPA master to see status, enable (or disable) compat plugin. I think it should be enabled by default.

When binding from ldapsearch as compat user you would get something like this in the logs:

[19/Sep/2015:18:40:55 +0200] conn=1950 fd=111 slot=111 SSL connection from 192.168.7.1 to 192.168.7.1
[19/Sep/2015:18:40:56 +0200] conn=1950 TLS1.2 128-bit AES
[19/Sep/2015:18:40:56 +0200] conn=1950 op=0 BIND dn="uid=abokovoy,cn=users,cn=compat,dc=vda,dc=li" method=128 version=3
[19/Sep/2015:18:40:56 +0200] conn=1950 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=abokovoy,cn=users,cn=accounts,dc=vda,dc=li"
[19/Sep/2015:18:40:56 +0200] conn=1950 op=1 SRCH base="dc=vda,dc=li" scope=2 filter="(uid=abokovoy)" attrs=ALL
[19/Sep/2015:18:40:56 +0200] conn=1950 op=1 RESULT err=0 tag=101 nentries=1 etime=0
[19/Sep/2015:18:40:56 +0200] conn=1950 op=2 UNBIND
[19/Sep/2015:18:40:56 +0200] conn=1950 op=2 fd=111 closed - U1

e.g. result of bind should show a rewritten DN. I can't reproduce the crash issue unless I would try to delete the user while I have a high load on the machine so that deletion does not propagate fast to the slapi-nis callback.

If entry was deleted, bind has to fail, indeed.

Installed ipa-server, enabled compat plugin and ran ldapsearch as user test1 from compat entry. No crashes observed.

[root@cloud-qe-21 ~]# ldapsearch -x -p 389 -h localhost -D "uid=test1,cn=users,cn=compat,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=redhat,dc=com" -w Secret123 -b "dc=idmqe,dc=lab,dc=eng,dc=bos,dc=redhat,dc=com" 

[root@cloud-qe-21 ~]# tail -f /var/log/dirsrv/slapd-IDMQE-LAB-ENG-BOS-REDHAT-COM/errors 
[16/Oct/2015:09:13:33 -0400] NSACLPlugin - The ACL target cn=ad,cn=etc,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=redhat,dc=com does not exist
[16/Oct/2015:09:13:33 -0400] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=redhat,dc=com does not exist
[16/Oct/2015:09:13:33 -0400] NSACLPlugin - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=redhat,dc=com does not exist
[16/Oct/2015:09:13:33 -0400] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[16/Oct/2015:09:13:33 -0400] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[16/Oct/2015:09:13:33 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[16/Oct/2015:09:13:33 -0400] - Listening on All Interfaces port 636 for LDAPS requests
[16/Oct/2015:09:13:33 -0400] - Listening on /var/run/slapd-IDMQE-LAB-ENG-BOS-REDHAT-COM.socket for LDAPI requests
[16/Oct/2015:09:13:42 -0400] memberof-plugin - Memberof task starts (arg: (objectclass=*)) ...
[16/Oct/2015:09:13:44 -0400] memberof-plugin - Memberof task finished (arg: (objectclass=*)) ...


Packages tested:
[root@cloud-qe-21 ~]# rpm -qa |egrep 'ipa-|389-ds'
389-ds-base-1.3.4.0-19.el7.x86_64
389-ds-base-libs-1.3.4.0-19.el7.x86_64
ipa-server-dns-4.2.0-12.el7.x86_64
389-ds-base-debuginfo-1.3.4.0-19.el7.x86_64
ipa-client-4.2.0-12.el7.x86_64
redhat-access-plugin-ipa-0.9.1-2.el7.noarch
ipa-admintools-4.2.0-12.el7.x86_64
ipa-python-4.2.0-12.el7.x86_64
sssd-ipa-1.13.0-39.el7.x86_64
ipa-server-4.2.0-12.el7.x86_64


Is this enough to mark the bug as Verified?

(In reply to Sankar Ramalingam from comment #8)
> Is this enough to mark the bug as Verified?

The DN of the bind user was replaced in slapi-nis, which triggered the crash.  I'm not sure what data/set-up is needed to make slapi-nis modify the bind DN.

Scott, could you please enlighten us how to verify this bug?  Thanks!!

I'm not sure we can test this with anything but sanity only, right?

From the looks of it from the ticket, it was never reproduced outside of a production environment.  The description from the ticket though also sounded like the crashes occurred during multiple simultaneous lookups.  So, do we need a script to fork multiple ssh for different users at the same time?  or enough maybe to just run multiple searches in parallel?

(In reply to Scott Poore from comment #10)
> I'm not sure we can test this with anything but sanity only, right?
> 
> From the looks of it from the ticket, it was never reproduced outside of a
> production environment.  The description from the ticket though also sounded
> like the crashes occurred during multiple simultaneous lookups.  So, do we
> need a script to fork multiple ssh for different users at the same time?  or
> enough maybe to just run multiple searches in parallel?

Thank you, Scott.

It is quite clear in the code backend_bind_cb (back-sch.c)... :)

/* If user comes from NSSWITCH, it will get authentication handled by PAM. */
This check "if (data->source == backend_entry_source_nsswitch)" is false and it goes to:
  /* Otherwise force rewrite of the SLAPI_BIND_TARGET_SDN
   * and let other plugins to handle it.
   * slapi-nis should have plugin ordering set below standard 50 to succeed */
in this "else code", it replace the Slapi_DN in pblock and it used to crash 389-ds-base.  

Is this else clause usually executed?  If so, this bug is verified.  (but I doubt it is ...  If so, the crash should have observed more often...):

Thanks Noriko and Scott. As per Scott's suggestion in comment #10, I executed multiple searches in parallel on the IPA server setup and I found no crash. Hence, marking the bug as Verified.

[root@cloud-qe-21 ~]# find /var/log/dirsrv/slapd-*/* -name core*
[root@cloud-qe-21 ~]# 

Packages tested in comment #8.

(In reply to Sankar Ramalingam from comment #12)
> Thanks Noriko and Scott. As per Scott's suggestion in comment #10, I
> executed multiple searches in parallel on the IPA server setup and I found

On the ipa server side, I enabled slapi-nis plugin as well and restarted the Directory server. This is to make sure both ipa compat and nis plugin were enabled when running the ldap searches.
> no crash. Hence, marking the bug as Verified.
> 
> [root@cloud-qe-21 ~]# find /var/log/dirsrv/slapd-*/* -name core*
> [root@cloud-qe-21 ~]# 
> 
> Packages tested in comment #8.

Hi,

when will this package be released, any timeline?


Thanks

Sven

(In reply to Sven Kieske from comment #14)
> Hi,
> 
> when will this package be released, any timeline?
> 
> 
> Thanks
> 
> Sven

It will be included in the next release.  We will let you know when it is ready.
Thanks for your patience.
--noriko

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2351.html