845125 – Slow response when binding with GSSAPI to 389 Directory Server

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 845125 - Slow response when binding with GSSAPI to 389 Directory Server

Summary: Slow response when binding with GSSAPI to 389 Directory Server

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	krb5
Sub Component:
Version:	6.3
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Nalin Dahyabhai
QA Contact:	Zbysek MRAZ
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	842564 (view as bug list)
Depends On:
Blocks:	852455
TreeView+	depends on / blocked

Reported:	2012-08-01 21:19 UTC by Loris Santamaria
Modified:	2020-09-13 20:15 UTC (History)
CC List:	17 users (show)
Fixed In Version:	krb5-1.9-35.el6
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2013-02-21 08:37:21 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Access logs (3.96 KB, application/octet-stream) 2012-08-01 21:19 UTC, Loris Santamaria	no flags	Details
proposed updated patch (31.42 KB, patch) 2012-09-11 01:18 UTC, Nalin Dahyabhai	no flags	Details \| Diff
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	389ds 389-ds-base issues 424	0	None	None	None	2020-09-13 20:15:35 UTC
Red Hat Product Errata	RHBA-2013:0319	0	normal	SHIPPED_LIVE	krb5 bug fix update	2013-02-20 20:54:56 UTC

Description Loris Santamaria 2012-08-01 21:19:09 UTC

Created attachment 601825 [details]
Access logs

Description of problem:

I have this test server with 8.000 entries running IPA 2.2.0 and 389-ds-base-1.2.10.2-20.el6_3.x86_64

ldapsearch with "-Y GSSAPI" is much slower than
using plain autentication:

# time ldapsearch -x uid=bdteg01662 dn
# extended LDIF
#
# LDAPv3
# base <dc=xxx,dc=gob,dc=ve> (default) with scope subtree
# filter: uid=bdteg01662
# requesting: dn 
#

# bdteg01662, users, accounts, xxx.gob.ve
dn: uid=bdteg01662,cn=users,cn=accounts,dc=xxx,dc=gob,dc=ve

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

real    0m0.006s
user    0m0.001s
sys     0m0.003s

# time ldapsearch -Y GSSAPI uid=bdteg01662 dn
SASL/GSSAPI authentication started
SASL username: admin.VE
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <dc=xxx,dc=gob,dc=ve> (default) with scope subtree
# filter: uid=bdteg01662
# requesting: dn 
#

# bdteg01662, users, accounts, xxx.gob.ve
dn: uid=bdteg01662,cn=users,cn=accounts,dc=xxx,dc=gob,dc=ve

# search result
search: 4
result: 0 Success

# numResponses: 2
# numEntries: 1

real    0m2.344s
user    0m0.007s
sys     0m0.005s

Version-Release number of selected component (if applicable):

389-ds-base-1.2.10.2-20.el6_3.x86_64

How reproducible:

always

Steps to Reproduce:
1.do a ldapsearch authenticating using GSSAPI
  
Actual results:

The command return succesfully after two seconds

Expected results:

The command should return succesfully almost immediately

Comment 2 Rich Megginson 2012-08-01 21:50:49 UTC

I can sort of reproduce the issue:

1) setup RHEL 6.3 x86_64 VM
2) ipa-server-install ... -N --selfsign --setup-dns --forwarder=hostipaddress
3) kinit admin
4) time ldapsearch -Y GSSAPI -s base -b "" objectclass=* dn

I consistently see times greater than 1.3 seconds - this is with no users except for the admin user.  I think 1.3 seconds is too high.

Comment 3 Rich Megginson 2012-08-02 14:08:14 UTC

Upstream ticket:
https://fedorahosted.org/389/ticket/424

Comment 4 Simo Sorce 2012-08-02 20:00:39 UTC

Ok after some investigation it seem that the issue is entirely confied into libkrb5 and its selinux integration.
Disabling selinux makes the delays disappear entirely.

Moving to krb5 as the fix needs to be done there.

Comment 5 Karel Srot 2012-08-09 12:10:00 UTC

Hi Nalin,
could you please share some details where is the problem and how is it going to be fixed?

Comment 6 Nalin Dahyabhai 2012-08-09 15:21:52 UTC

The speed hit comes from reading in the configuration for file contexts, which we have to do so that we can correctly label files that we're about to create.

The version of the package in Fedora 17 contains a fix to skip attempting to set the file creation context when opening a file using fopen() with mode "rb".  This is in addition to "r", which it already recognizes as a mode that doesn't require that it be set.  The file contexts aren't read in until they're needed, so this removes work when it's truly unnecessary.  So part of this is backporting that.

In addition, the F18 and later branches have started to cache the contents of the contexts file, only re-reading it when the configuration file's modification date changes.  While this means that we hang on to the memory that's required to hold that data, it makes the largest portion of the work involved something that should only happen occasionally (specifically, when the file is edited or the SELinux policy is upgraded).  The rest of it is backporting that.

Comment 8 Karel Srot 2012-08-09 16:07:31 UTC

*** Bug 842564 has been marked as a duplicate of this bug. ***

Comment 9 Jeff Tickle 2012-08-15 20:09:14 UTC

"Me too."  RHEL 6.3.  10 users, 30 clients, and some information associated with sudo and autofs.  I'm afraid I'm not good enough at LDAP to give more than that, but suffice it to say, it's a pretty small installation, and we're not talking about thousands of users here.

389-ds-base-1.2.10.2-20.el6_3.i686

With SELinux Enabled or Permissive, ldapsearch -s base -b "" takes:
real    0m6.088s
user    0m0.009s
sys     0m0.007s

After disabling SELinux on the IPA Server:
real    0m0.078s
user    0m0.011s
sys     0m0.029s

I'm leaving SELinux disabled for now, but would like to turn it back on if at all possible!

Comment 14 Karel Srot 2012-09-06 15:01:31 UTC

Hi Nalin, 
do you think you can provide more simple reproducer than the one mentioned in #c2?
Thank you in advance.

Comment 15 Nalin Dahyabhai 2012-09-06 23:08:35 UTC

(In reply to comment #14)
> do you think you can provide more simple reproducer than the one mentioned
> in #c2?

Certainly not anything less labor-intensive.

The general test method is to find a process that's going to create multiple files which need to be labeled, and to have it create those files while being instrumented somehow so that we can check how many times it reads in the SELinux file context configuration.

You can approximate what the directory server is doing with GSSAPI using the gss-client and gss-server sample programs.  Set up a keytab and obtain credentials using kinit.  Then, assuming you have a key for the "host" service on the local system, instrument:
  gss-server -port 1234 host@`hostname`
while you periodically connect to it by running
  gss-client  -port 1234 `hostname` host "Test message text."

Assuming the timestamp on that configuration doesn't change while we're performing the test, it should only read that configuration once, rather than multiple times as it might have before.

Comment 17 Nalin Dahyabhai 2012-09-11 01:18:05 UTC

Created attachment 611611 [details]
proposed updated patch

Comment 25 errata-xmlrpc 2013-02-21 08:37:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0319.html

Comment 26 Jan-Frode Myklebust 2013-02-21 11:09:33 UTC

So his is fixed in RHEL6.4 ? Will it be safe to upgrade only the krb5-packages on our RHEL 6.3 servers to these, or will we have to do a full 6.4 upgrade (including IPA 3.0 upgrade) ?

Comment 27 Nalin Dahyabhai 2013-02-21 15:43:36 UTC

Yes, the update in 6.4 includes this fix.  But due to differences in the kdb module ABI between krb5 versions 1.9 and 1.10, I don't think you can drop the updated packages onto a 6.3 system that's acting as an IPA server.

Note You need to log in before you can comment on or make changes to this bug.