Bug 799141

Summary:	SSSD with kerberos auth is very slow without nscd running. nscd is not supposed to run alongside of sssd!
Product:	Red Hat Enterprise Linux 6	Reporter:	jgibson
Component:	sssd	Assignee:	Stephen Gallagher <sgallagh>
Status:	CLOSED DUPLICATE	QA Contact:	IDM QE LIST <seceng-idm-qe-list>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	6.1	CC:	grajaiya, jgalipea, jhrozek, prc
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-03-02 10:33:32 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description jgibson 2012-03-01 21:02:30 UTC

I've noticed that ssh logins to a server using krb5 in sssd take much longer than they should (5-40 seconds).  I started sssd with debugging enabled (sssd --debug-level=9 --debug-timestamps=1 -c /etc/sssd/sssd.conf) and noticed the following errors on the console output:

(Thu Mar  1 12:31:05 2012) [sssd] [check_file] (1): lstat for [/var/run/nscd/socket] failed: [2][No such file or directory].
(Thu Mar  1 12:31:05 2012) [sssd] [ldb] (6): server_sort:Unable to register control with rootdse!
(Thu Mar  1 12:31:05 2012) [sssd] [confdb_get_domain_internal] (1): No enumeration for [default]!
(Thu Mar  1 12:31:05 2012) [sssd] [server_setup] (3): Becoming a daemon.

Running ssh -v on the client had the following output:
$ ssh -v server
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /home/user/.ssh/config
debug1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to server [1.2.3.4] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 24:4b:96:a5:07:71:d3:e5:63:e3:f6:b1:e9:aa:ba:2f
debug1: Host 'server' is known and matches the RSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:503
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

LONG PAUSE HERE if NOT using GSSAPI


debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic


LONG PAUSE HERE if using GSSAPI


debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to server ([1.2.3.4]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Sending environment.
debug1: Sending env LC_MESSAGES = en_US.UTF-8
debug1: Sending env LC_COLLATE = en_US.UTF-8
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_CTYPE = en_US.UTF-8
Last login: Thu Mar  1 11:34:02 2012 from 1.2.3.5





Since I didn't have nscd installed, I tried just creating the /var/run/nscd directory & restarted sssd but there was no improvement.

I then installed nscd and started it - the /var/run/nscd/socket was created.  Afterwards I restarted sssd again.  Now logins take less than a second.

Before starting nscd:
$ time ssh server :

real	0m21.829s
user	0m0.016s
sys	0m0.016s

After starting nscd:
$ time ssh server :

real	0m0.851s
user	0m0.016s
sys	0m0.012s

I would be happy with this as a solution, but the redhat sssd documentation clearly recommends that it should NOT be run with nscd side-by-side.

"SSSD is not designed to be used with the NSCD daemon. Even though SSSD does not directly conflict with NSCD, using both services can result in unexpected behavior, especially with how long entries are cached."
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/usingnscd-sssd.html


Versions:
Server-
RHEL 6.1
glibc-2.12-1.25.el6_1.3.x86_64
sssd-1.5.1-34.el6_1.3.x86_64
nscd-2.12-1.25.el6_1.3.x86_64
krb5-workstation-1.9-9.el6_1.2.x86_64
krb5-libs-1.9-9.el6_1.2.x86_64
pam_krb5-2.3.11-6.el6.x86_64

Client- 
Ubuntu 11.04 x64
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011

How reproducible:
Always.  With or without client gssapi authentication

Steps to Reproduce:
1. Configure sssd with auth_provider = krb5.  We are using Windows 2008R2 for LDAP/kerberos.
2. Attempt to log in or run an ssh command.  The time to authenticate will very wildly - 5 to over 40 seconds
3. Install nscd and start it.
4. Restart sssd
5. Attempt to log in - logins are now much quicker - generally under a second.
  
Actual results:
$ time ssh server :

real	0m21.829s
user	0m0.016s
sys	0m0.016s

Expected results:
$ time ssh server :

real	0m0.851s
user	0m0.016s
sys	0m0.012s

Additional info:
We are using LDAPS for directory lookups.  Client is configured to use gssapi but problem happens with gssapi disabled too.

sssd.conf:
[sssd]
config_file_version = 2
reconnection_retries = 3
services = nss, pam
domains = default
[nss]
filter_groups = root
filter_users = root
reconnection_retries = 3
[pam]
reconnection_retries = 3
[domain/default]
auth_provider = krb5
cache_credentials = True
ldap_group_object_class = group
ldap_id_use_start_tls = False
ldap_user_uid_number = UidNumber
chpass_provider = krb5
enumerate = False
ldap_schema = rfc2307bis
ldap_force_upper_case_realm = True
ldap_user_principal = userPrincipalName
ldap_user_object_class = person
ldap_user_gid_number = GidNumber
ldap_group_name = msSFU30Name
ldap_user_name = uid
ldap_group_gid_number = GidNumber
ldap_default_authtok_type = password
ldap_search_base = dc=mydomain,dc=com
id_provider = ldap
ldap_default_bind_dn = CN=linuxuser,OU=accounts,DC=mydomain,DC=com
ldap_user_shell = LoginShell
krb5_server = server1.mydomain.com,server2.mydomain.com,server3.mydomain.com,server4.mydomain.com
min_id = 1000
ldap_uri = ldaps://server1.mydomain.com,ldaps://server2.mydomain.com,ldaps://server3.mydomain.com,ldaps://server4.mydomain.com
krb5_realm = MYDOMAIN.COM
krb5_kdcip = server1.mydomain.com,server2.mydomain.com,server3.mydomain.com,server4.mydomain.com
ldap_user_home_directory = unixHomeDirectory
ldap_default_authtok = ********
ldap_tls_cacertdir = /etc/openldap/cacerts
dns_discovery_domain = mydomain.com

Comment 2 Jakub Hrozek 2012-03-02 10:33:32 UTC

Sorry, but you need to include more information. Primarily the SSSD version you are running and the SSSD logs.

> (Thu Mar  1 12:31:05 2012) [sssd] [check_file] (1): lstat for
> [/var/run/nscd/socket] failed: [2][No such file or directory].
> (Thu Mar  1 12:31:05 2012) [sssd] [ldb] (6): server_sort:Unable to register
> control with rootdse!
> (Thu Mar  1 12:31:05 2012) [sssd] [confdb_get_domain_internal] (1): No
> enumeration for [default]!
> (Thu Mar  1 12:31:05 2012) [sssd] [server_setup] (3): Becoming a daemon.

These are just warnings. Also, the format of the debug logs shows that you are running sssd older than 1.7.

Because nscd gives you a much better login time, I assume the problem is slow initgroups operation. There have been many performance improvements in both SSSD 1.7 and 1.8 related to faster initgroups. 

RHEL-6.3 is going to rebase the SSSD to 1.8 which might be a good place to try the performance improvements on your RHEL server.

> I would be happy with this as a solution, but the redhat sssd documentation
> clearly recommends that it should NOT be run with nscd side-by-side.
> 
> "SSSD is not designed to be used with the NSCD daemon. Even though SSSD does
> not directly conflict with NSCD, using both services can result in unexpected
> behavior, especially with how long entries are cached."
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/usingnscd-sssd.html

nscd and SSSD both include their own caching mechanism and it's better not to mix the two.

I'm going to close this bug here with the suggestion that you try out a newer version to see if your performance problem persists even with the fixes we did recently.

*** This bug has been marked as a duplicate of bug 743133 ***