Bug 845125 - Slow response when binding with GSSAPI to 389 Directory Server
Slow response when binding with GSSAPI to 389 Directory Server
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: krb5 (Show other bugs)
6.3
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Nalin Dahyabhai
Zbysek MRAZ
: ZStream
: 842564 (view as bug list)
Depends On:
Blocks: 852455
  Show dependency treegraph
 
Reported: 2012-08-01 17:19 EDT by Loris Santamaria
Modified: 2013-07-03 09:16 EDT (History)
17 users (show)

See Also:
Fixed In Version: krb5-1.9-35.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-21 03:37:21 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Access logs (3.96 KB, application/octet-stream)
2012-08-01 17:19 EDT, Loris Santamaria
no flags Details
proposed updated patch (31.42 KB, patch)
2012-09-10 21:18 EDT, Nalin Dahyabhai
no flags Details | Diff

  None (edit)
Description Loris Santamaria 2012-08-01 17:19:09 EDT
Created attachment 601825 [details]
Access logs

Description of problem:

I have this test server with 8.000 entries running IPA 2.2.0 and 389-ds-base-1.2.10.2-20.el6_3.x86_64

ldapsearch with "-Y GSSAPI" is much slower than
using plain autentication:

# time ldapsearch -x uid=bdteg01662 dn
# extended LDIF
#
# LDAPv3
# base <dc=xxx,dc=gob,dc=ve> (default) with scope subtree
# filter: uid=bdteg01662
# requesting: dn 
#

# bdteg01662, users, accounts, xxx.gob.ve
dn: uid=bdteg01662,cn=users,cn=accounts,dc=xxx,dc=gob,dc=ve

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

real    0m0.006s
user    0m0.001s
sys     0m0.003s

# time ldapsearch -Y GSSAPI uid=bdteg01662 dn
SASL/GSSAPI authentication started
SASL username: admin@XXX.GOB.VE
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <dc=xxx,dc=gob,dc=ve> (default) with scope subtree
# filter: uid=bdteg01662
# requesting: dn 
#

# bdteg01662, users, accounts, xxx.gob.ve
dn: uid=bdteg01662,cn=users,cn=accounts,dc=xxx,dc=gob,dc=ve

# search result
search: 4
result: 0 Success

# numResponses: 2
# numEntries: 1

real    0m2.344s
user    0m0.007s
sys     0m0.005s

Version-Release number of selected component (if applicable):

389-ds-base-1.2.10.2-20.el6_3.x86_64

How reproducible:

always

Steps to Reproduce:
1.do a ldapsearch authenticating using GSSAPI
  
Actual results:

The command return succesfully after two seconds

Expected results:

The command should return succesfully almost immediately
Comment 2 Rich Megginson 2012-08-01 17:50:49 EDT
I can sort of reproduce the issue:

1) setup RHEL 6.3 x86_64 VM
2) ipa-server-install ... -N --selfsign --setup-dns --forwarder=hostipaddress
3) kinit admin
4) time ldapsearch -Y GSSAPI -s base -b "" objectclass=* dn

I consistently see times greater than 1.3 seconds - this is with no users except for the admin user.  I think 1.3 seconds is too high.
Comment 3 Rich Megginson 2012-08-02 10:08:14 EDT
Upstream ticket:
https://fedorahosted.org/389/ticket/424
Comment 4 Simo Sorce 2012-08-02 16:00:39 EDT
Ok after some investigation it seem that the issue is entirely confied into libkrb5 and its selinux integration.
Disabling selinux makes the delays disappear entirely.

Moving to krb5 as the fix needs to be done there.
Comment 5 Karel Srot 2012-08-09 08:10:00 EDT
Hi Nalin,
could you please share some details where is the problem and how is it going to be fixed?
Comment 6 Nalin Dahyabhai 2012-08-09 11:21:52 EDT
The speed hit comes from reading in the configuration for file contexts, which we have to do so that we can correctly label files that we're about to create.

The version of the package in Fedora 17 contains a fix to skip attempting to set the file creation context when opening a file using fopen() with mode "rb".  This is in addition to "r", which it already recognizes as a mode that doesn't require that it be set.  The file contexts aren't read in until they're needed, so this removes work when it's truly unnecessary.  So part of this is backporting that.

In addition, the F18 and later branches have started to cache the contents of the contexts file, only re-reading it when the configuration file's modification date changes.  While this means that we hang on to the memory that's required to hold that data, it makes the largest portion of the work involved something that should only happen occasionally (specifically, when the file is edited or the SELinux policy is upgraded).  The rest of it is backporting that.
Comment 8 Karel Srot 2012-08-09 12:07:31 EDT
*** Bug 842564 has been marked as a duplicate of this bug. ***
Comment 9 Jeff Tickle 2012-08-15 16:09:14 EDT
"Me too."  RHEL 6.3.  10 users, 30 clients, and some information associated with sudo and autofs.  I'm afraid I'm not good enough at LDAP to give more than that, but suffice it to say, it's a pretty small installation, and we're not talking about thousands of users here.

389-ds-base-1.2.10.2-20.el6_3.i686

With SELinux Enabled or Permissive, ldapsearch -s base -b "" takes:
real    0m6.088s
user    0m0.009s
sys     0m0.007s

After disabling SELinux on the IPA Server:
real    0m0.078s
user    0m0.011s
sys     0m0.029s

I'm leaving SELinux disabled for now, but would like to turn it back on if at all possible!
Comment 14 Karel Srot 2012-09-06 11:01:31 EDT
Hi Nalin, 
do you think you can provide more simple reproducer than the one mentioned in #c2?
Thank you in advance.
Comment 15 Nalin Dahyabhai 2012-09-06 19:08:35 EDT
(In reply to comment #14)
> do you think you can provide more simple reproducer than the one mentioned
> in #c2?

Certainly not anything less labor-intensive.

The general test method is to find a process that's going to create multiple files which need to be labeled, and to have it create those files while being instrumented somehow so that we can check how many times it reads in the SELinux file context configuration.

You can approximate what the directory server is doing with GSSAPI using the gss-client and gss-server sample programs.  Set up a keytab and obtain credentials using kinit.  Then, assuming you have a key for the "host" service on the local system, instrument:
  gss-server -port 1234 host@`hostname`
while you periodically connect to it by running
  gss-client  -port 1234 `hostname` host "Test message text."

Assuming the timestamp on that configuration doesn't change while we're performing the test, it should only read that configuration once, rather than multiple times as it might have before.
Comment 17 Nalin Dahyabhai 2012-09-10 21:18:05 EDT
Created attachment 611611 [details]
proposed updated patch
Comment 25 errata-xmlrpc 2013-02-21 03:37:21 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0319.html
Comment 26 Jan-Frode Myklebust 2013-02-21 06:09:33 EST
So his is fixed in RHEL6.4 ? Will it be safe to upgrade only the krb5-packages on our RHEL 6.3 servers to these, or will we have to do a full 6.4 upgrade (including IPA 3.0 upgrade) ?
Comment 27 Nalin Dahyabhai 2013-02-21 10:43:36 EST
Yes, the update in 6.4 includes this fix.  But due to differences in the kdb module ABI between krb5 versions 1.9 and 1.10, I don't think you can drop the updated packages onto a 6.3 system that's acting as an IPA server.

Note You need to log in before you can comment on or make changes to this bug.