Bug 137220

Created attachment 105809 [details]
Sample LDIF creating an auto.master map, and an auto.misc with 7 entries.

Created attachment 105810 [details]
suggested patch which returns LKP_NOTSUP for certain LDAP errors, and always retries lookups with a specific query if a first lookup attempt fails

This may be related to bug 133063 or bug 128103, depending on the size
of the maps which are being used by the reporters of those bugs.

Tested patch at NCSU.  Our LDAP server has over 5,000 entries that are
automounted which doesn't include the 60,000 some odd user accounts
that will have home directories automounted as well.  Needless to say
I was hitting the problem where openldap would return
LDAP_SIZELIMIT_EXCEEDED and blow up.
                                                                     
          
This patch is working great.  I plan on using this feature in RHEL 4.

nalin, shouldn't we attempt to read the entire map in lookup_ghost? 
Isn't there a way to iterate over the map in smaller chunks?  Is there
some reason this is not advisable?  The specific chunk I'm talking
about is here:

@@ -289,13 +307,19 @@
 {
 	struct lookup_context *ctxt = (struct lookup_context *) context;
 	struct mapent_cache *me;
-	int status = 1;
+	int status = 1, rv = LDAP_SUCCESS;
 	char *mapname;
 
 	chdir("/");
 
-	if (!read_map(root, ctxt))
-		return LKP_FAIL;
+	if (!read_map(root, ctxt, NULL, 0, &rv))
+		switch (rv) {
+		case LDAP_SIZELIMIT_EXCEEDED:
+		case LDAP_UNWILLING_TO_PERFORM:
+			return LKP_NOTSUP;
+		default:
+			return LKP_FAIL;
+		}

I think the change I describe is necessary for ghosting to work properly.

Comments?

The lookup_ghost function is trying to read the entire map (via
read_map), but it's failing because the server is configured to cut
short the number of answers which it will send to a client for any
single search request.  When automount gets this error (along with a
partial set of results), it currently exits.

Caching just the results which it gets back would allow the
automounter to ghost some random subset of the map, but because it
currently expects the entire map to be in the cache, it would never
mount anything which wasn't in the subset of results which the server
returned.

Breaking up the search, by attempting to walk the tree under the map's
base DN using multiple searches with scope="one", might work in some
cases, but a sufficiently-large single map would still trigger a
size-limit-exceeded error, and it would be very time-consuming.

I don't know of any way to circumvent a server-imposed size limit.

See RFC 2696 - LDAP Control Extension for Simple Paged Results Manipulation.  
Surely we can do an ldap_get_option for the LDAP_OPT_SIZELIMIT and base our page
size on that.  This only really makes sense if we can determine how many results
there are for a query.  If we can get both of these values, then we can make an
informed decision on whether to parse the entire map, or produce a partial
cache.  (i.e. if the result set size was large and the page size was low, we
would not want to parse the full map.)

According to RFC 2696:

   Each time the server returns a set of results to the client when
   processing a search request containing the pagedResultsControl, the
   server includes the pagedResultsControl control in the
   searchResultDone message. In the control returned to the client, the
   size MAY be set to the server's estimate of the total number of
   entries in the entire result set. Servers that cannot provide such an
   estimate MAY set this size to zero (0).  The cookie MUST be set to an
   empty value if there are no more entries to return (i.e., the page of
   search results returned was the last), or, if there are more entries
   to return, to an octet string of the server's choosing,used to resume
   the search.

So, we may or may not get the estimate depending on the implementation.

In the case where we don't parse the entire map, should we ghost any of the
entries?  This could result in user confusion, to be sure.  Whatever method we
choose, this will require corresponding documentation.

I have tried out the patch and it works as advertised.  If the the
search limit error is recieved the only difference seems to be that
the map is not cached.

If you attempt to cd into an autofs directory using ldap (and the
results are greater than the search limit), the automounter will just
query the ldap server for that specific directory.

It may be a good idea to document this, as well as print an error/info
message notifying the user that the map will not be cached.

The fix is in autofs-4.1.3-33 & autofs-4.1.3-35.

Jeffrey,

Would the page size option only be used when ghosting is turned on, or
when the client /needs/ to have the entire cache?  

With LDAP maps for some 60,000 accounts I really don't want my linux
clients to attempt to get more than they need from the LDAP servers.

I just spoke with Nalin, and he cleared up a misconception I had about rfc2696.
 It is not a way to iterate through an entire map.  Rather, it is a way for the
client to rate limit the results it receives from the server (not to exceed the
defined maximum result set size imposed by the server).

As such, I think nalin's patch is okay as-is.  Thanks, Nalin!

Can someone comfirm this is fixed with the latest automount packages available?
 Thanks.

The patch as currenlty applied causes a problem if the user is trying
to use ghost and the LDAP_SIZELIMIT_EXCEEDED error is recieved.

(In reply to comment #13)
> The patch as currently applied causes a problem if the user is trying
> to use ghost and the LDAP_SIZELIMIT_EXCEEDED error is received.

FWIW, this is wrt 4.1.3-44.

Confirmed fixed in 4.1.3-67.

Throwing in ON_QA since we're just waiting on the package to land in a tree.

autofs-4.1.3-67 included in re1212.0 tree.  Closing out.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2005-177.html