Bug 471138 - LDCLT: add abandon to ldclt
Summary: LDCLT: add abandon to ldclt
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: 389
Classification: Retired
Component: Command Line Utilities
Version: 1.1.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Noriko Hosoi
QA Contact: Chandrasekar Kannan
URL:
Whiteboard:
Depends On:
Blocks: 249650 FDS1.2.0
TreeView+ depends on / blocked
 
Reported: 2008-11-11 22:48 UTC by Noriko Hosoi
Modified: 2015-01-04 23:34 UTC (History)
4 users (show)

Fixed In Version: 8.1
Clone Of:
Environment:
Last Closed: 2009-04-29 23:07:52 UTC
Embargoed:


Attachments (Terms of Use)
cvs diff ldclt files (16.00 KB, patch)
2008-11-11 23:37 UTC, Noriko Hosoi
no flags Details | Diff
vcs diff ldclt files (16.28 KB, patch)
2008-11-17 23:09 UTC, Noriko Hosoi
no flags Details | Diff
cvs commit messages (1.11 KB, text/plain)
2008-11-17 23:14 UTC, Noriko Hosoi
no flags Details

Description Noriko Hosoi 2008-11-11 22:48:04 UTC
Description of problem:
To debug the bug 459440 "rhds71sp3 - ns-slapd crash on 'too many' ABANDON targetop=NOTFOUND", we need a tool to send abandon requests to the server

Comment 2 Noriko Hosoi 2008-11-11 23:37:55 UTC
Created attachment 323274 [details]
cvs diff ldclt files

Files:
  ldap/servers/slapd/tools/ldapfct.c
                           ldclt.c
                           ldclt.h
                           ldcltU.c
                           threadMain.c

Description: added "-e abandon"

Sample usage:
  ROOTDN="cn=Directory Manager"
  ROOTDNPW="password"
  BASEDN="dc=example,dc=com"
  HOST="host.example.com"
  PORT=389
  DAT=/export/data
  ldclt -D "$ROOTDN" -w "$ROOTDNPW" -h "$HOST" -p "$PORT" \
      -e abandon -e random -b "$BASEDN" -f "uid=emailperson_XXXXX" \
      -q -n 2 -r 10000 -R 99999 -I 32 -e inetOrgPerson -e imagesdir=$DAT

Sample access log:
[..] conn=68 op=16297 SRCH base="dc=example,dc=com" scope=2 filter="(uid=emailperson_78683)" attrs=ALL
[..] conn=68 op=16297 RESULT err=0 tag=101 nentries=1 etime=6
[..] conn=68 op=16298 ABANDON targetop=NOTFOUND msgid=16298
[..] conn=68 op=16299 SRCH base="dc=example,dc=com" scope=2 filter="(uid=emailperson_15947)" attrs=ALL
[..] conn=68 op=16299 RESULT err=0 tag=101 nentries=1 etime=11
[..] conn=68 op=16300 ABANDON targetop=NOTFOUND msgid=16300

Comment 4 Nathan Kinder 2008-11-12 18:16:00 UTC
The code looks fine.  Your sample access logs show the abandons happening after the operation has already completed.

Will we be able to have the abandons actually abandon active operations if we put enough load against the server with your changes?  It is important to be able to stress that code path in the server as well.  I suppose this will also depend if you pass in a complex enough search that it will take some time for the server to process it (an unindexed subtree search maybe).

Comment 5 Noriko Hosoi 2008-11-12 18:45:35 UTC
I'm issuing the abandon request between the async search and receiving the result.  Actually, pretty close to the search request in the code.  As you wrote, it's quite difficult for the exact match search to give enough time to abandon the operation.

The case I wanted to simulate was also the searches are fast and abandons come after the results:
[..] conn=22840 op=94993 SRCH base="ou=Autofs5, ou=Application Services, ou=engineering, o=example, c=US" scope=2 filter="(&(|(objectClass=automount))(|(automountKey=\2a)))" attrs="automountKey automountInformation automountKey"
[..] conn=22840 op=94993 RESULT err=0 tag=101 nentries=2 etime=0
[..] conn=22840 op=94994 ABANDON targetop=NOTFOUND msgid=94994
[..] conn=22840 op=94995 SRCH base="ou=Autofs5, ou=Application Services, ou=engineering, o=example, c=US" scope=2 filter="(&(|(objectClass=automount))(|(automountKey=.DS_Store)))" attrs="automountKey automountInformation automountKey"
[..] conn=22840 op=94995 RESULT err=0 tag=101 nentries=0 etime=0
[..] conn=22840 op=94996 ABANDON targetop=NOTFOUND msgid=94996

So, I think this is good for the test case.  I think I'm going to introduce another option(maybe?) to replace the search filter with the unindexed search.

Thanks for the useful suggestion, Nathan!

Comment 6 Nathan Kinder 2008-11-12 19:00:19 UTC
(In reply to comment #5)
> So, I think this is good for the test case.  I think I'm going to introduce
> another option(maybe?) to replace the search filter with the unindexed search.

I think we'd be able to just pass in a different filter in the "-f" option that is an unindexed attribute.  Even though it is an exact match, we should still see some abandons take place before the server is finished processing if the database is large enough and we have enough load.

With the current ldclt options, it is possible to do a search using a wildcard in the filter?

Comment 7 Rich Megginson 2008-11-12 22:25:31 UTC
Looks good.

Do we have a way to send ABANDON requests for bogus msgids?

Comment 8 Noriko Hosoi 2008-11-13 00:35:26 UTC
Thank you, Nathan and Rich, for reviewing the code

I'm trying to run these cases which somehow "hangs" either in ber_flush or poll. :(
  ldclt -D "$ROOTDN" -w "$ROOTDNPW" -h "$HOST" -p "$PORT" \
      -e abandon -e random -b "$BASEDN" -f "uid=emailperson_*" \
      -q -n 2 -r 10000 -R 99999 -I 32 -e inetOrgPerson -e imagesdir=$DAT

  ldclt -D "$ROOTDN" -w "$ROOTDNPW" -h "$HOST" -p "$PORT" \
      -e abandon -e random -b "$BASEDN" -f "jpegphoto=*" \
      -q -n 2 -r 10000 -R 99999 -I 32 -e inetOrgPerson -e imagesdir=$DAT

I'm hoping it could be solved by tuning the timeout parameter and/or ldap_result options...

> Do we have a way to send ABANDON requests for bogus msgids?
Not now.  I'm going to add it.

Comment 9 Noriko Hosoi 2008-11-15 03:11:00 UTC
(In reply to comment #8)
> > Do we have a way to send ABANDON requests for bogus msgids?
> Not now.  I'm going to add it.

I tried to add the feature, but unfortunately, LDAP C SDK is so smart that it does not send bogus requests.  The library checks the request history in the LDAP handle and get the msgids in it.  If the msgid to abandon is not in the msgids from the request history, the request won't be sent.
125	static int
126	do_abandon( LDAP *ld, int origid, int msgid, LDAPControl **serverctrls,
127	    LDAPControl **clientctrls )
128	{
           [...]
143        /*
144         * Find the request that we are abandoning.  Don't send an
145         * abandon message unless there is something to abandon.
146         */

Comment 10 Rich Megginson 2008-11-15 03:47:31 UTC
Ok.  If we need to test sending bogus abandons, we can do that with the ldapbertest.pl thing.

Comment 11 Noriko Hosoi 2008-11-17 23:09:23 UTC
Created attachment 323816 [details]
vcs diff ldclt files

Made some minor changes to doAbandon inldapfct.c.  I'm attaching the new diffs.

Comment 12 Noriko Hosoi 2008-11-17 23:14:03 UTC
Created attachment 323817 [details]
cvs commit messages

Thanks to Rich and Nathan for their reviews and comments.

Checked in into CVS HEAD.

Comment 13 Michael Gregg 2009-04-10 00:21:47 UTC
There was no duration listed here, but I ran the following script for about 1 hour without errors.

  ROOTDN="cn=Directory Manager"
  ROOTDNPW="pw"
  BASEDN="dc=example,dc=com"
  HOST="epsilon.dsdev.sjc.redhat.com"
  PORT=389
  DAT=/export/data
  ldclt -D "$ROOTDN" -w "$ROOTDNPW" -h "$HOST" -p "$PORT" \
      -v -e abandon -e random -b "$BASEDN" -f "uid=emailperson_XXXXX" \
      -q -n 8 -r 10000 -R 99999 -I 32 -e inetOrgPerson -e imagesdir=$DAT

verified against 8.1.0-0.6.el5dsrv

Comment 14 Chandrasekar Kannan 2009-04-29 23:07:52 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-0455.html


Note You need to log in before you can comment on or make changes to this bug.