Bug 726475

Summary: sssd_pam leaks file descriptors.
Product: Red Hat Enterprise Linux 6 Reporter: Stephen Gallagher <sgallagh>
Component: sssdAssignee: Stephen Gallagher <sgallagh>
Status: CLOSED ERRATA QA Contact: Chandrasekar Kannan <ckannan>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.1CC: benl, dpal, franta, grajaiya, jeffschroeder, jgalipea, jhrozek, jwest, kbanerje, mfuruta, msvoboda, prc
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sssd-1.5.1-46.el6 Doc Type: Bug Fix
Doc Text:
Previously, SSSD did not properly close its PAM sockets after an authentication attempt, which eventually resulted in process resource exhaustion and a denial of service situation. The code has been modified to fix this issue, and file descriptors are now properly released when they are no longer in use.
Story Points: ---
Clone Of:
: 748855 (view as bug list) Environment:
Last Closed: 2011-12-06 16:39:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 727041, 748855    

Description Stephen Gallagher 2011-07-28 18:30:51 UTC
Description of problem:
(Received on sssd-devel.org from Jan Engelhardt)

I am noticing an fd leak with the /usr/lib/sssd/sssd_pam helper program 
(affected versions: 1.4.1--1.5.11). As is usual with weird bugs, this 
happens on one machine while it does not on another, but it at least it
is reproducible on the one :)
What I observe is that for every authentication attempt (may either
fail or success), an fd is left behind::

Version-Release number of selected component (if applicable):
sssd-1.5.1-43.el6

How reproducible:
Every time

Steps to Reproduce:
1. Perform an authentication
2. Examine /proc/<PID of sssd_pam>/fd
  
Actual results:
File descriptors are left behind.

Expected results:
File descriptors should be closed when no longer in use.

Additional info:

17:25 mailsrv:/proc/31491/fd # ls -l
[...]
lrwx------ 1 root root 64 Jul 28 17:25 42 -> socket:[228511]
lrwx------ 1 root root 64 Jul 28 17:25 43 -> socket:[227514]
lrwx------ 1 root root 64 Jul 28 17:25 44 -> socket:[228537]
lrwx------ 1 root root 64 Jul 28 17:21 5 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jul 28 17:21 6 -> anon_inode:[eventpoll]
lr-x------ 1 root root 64 Jul 28 17:21 7 -> pipe:[213086]
lrwx------ 1 root root 64 Jul 28 17:21 76 -> socket:[220069]
l-wx------ 1 root root 64 Jul 28 17:21 8 -> pipe:[213086]
lrwx------ 1 root root 64 Jul 28 17:21 88 -> socket:[221781]
lrwx------ 1 root root 64 Jul 28 17:21 9 -> anon_inode:[eventpoll]
17:25 mailsrv:/proc/31491/fd # su - jengelh
17:25 mailsrv:~ > su - jengelh
Passord: 
17:25 mailsrv:~ > logout
17:25 mailsrv:~ > logout
17:25 mailsrv:/proc/31491/fd # ls -l
[...]
lrwx------ 1 root root 64 Jul 28 17:25 42 -> socket:[228511]
lrwx------ 1 root root 64 Jul 28 17:25 43 -> socket:[227514]
lrwx------ 1 root root 64 Jul 28 17:25 44 -> socket:[228537]
lrwx------ 1 root root 64 Jul 28 17:25 46 -> socket:[227688]
[...]


Furthermore, this new socket has no complementary process to have
the other end of socket:[227688] open (anymore):


17:30 mailsrv:/proc # ls -l */fd/* | grep socket:.227688
ls: cannot access 7175/fd/3: No such file or directory
ls: cannot access self/fd/255: No such file or directory
lrwx------ 1 root     root  64 Jul 28 17:25 31491/fd/46 -> socket:[227688]


lsof(8) tells us the name part of the socket for reference:


sssd_pam 31491 root   46u  unix 0xffff88001083a840      0t0 227688 /var/lib/sss/pipes/private/pam

Comment 1 Jeff Schroeder 2011-07-29 20:17:29 UTC
Can we get a Z-Stream patch for this?

This bug affects our RHEL6 postfix mailservers. It causes sssd to not allow logins which is effectively a denial of service. To us, this is a blocker bug.

Comment 3 Stephen Gallagher 2011-07-29 20:23:42 UTC
Note to QA: the reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=725281 can also be used to verify this issue.

Comment 7 Miroslav Svoboda 2011-08-10 09:12:05 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, SSSD did not properly close its PAM sockets after an authentication attempt, which eventually resulted in process resource exhaustion and a denial of service situation. The code has been modified to fix this issue, and file descriptors are now properly released when they are no longer in use.

Comment 8 Frantisek Hanzlik 2011-08-13 01:25:47 UTC
same problems at Fedora 14 i686 (sssd-1.5.11-2.fc14.i686). Here it block saslauthd, so no. of fd increases at both processes up to maximum number of open file descriptors (1024 in that case) and then things hungs at "too many open files" errors.

Comment 9 Jeff Schroeder 2011-08-13 12:02:08 UTC
@Frantisek: Yup, thats the exact same bug I saw that the fixed package takes care of. You just need to update your sssd version via yum (if it has hit the repositories yet).

http://koji.fedoraproject.org/koji/buildinfo?buildID=257319

Comment 10 Frantisek Hanzlik 2011-08-15 09:39:35 UTC
sssd-1.5.12-1.fc14.i686 (from F14 testing repo) after nearly two days and ~ 15000 requests seems working OK.

Comment 11 Jakub Hrozek 2011-08-15 10:23:34 UTC
(In reply to comment #10)
> sssd-1.5.12-1.fc14.i686 (from F14 testing repo) after nearly two days and ~
> 15000 requests seems working OK.

Good to hear that. Can you also add a positive karma to the update? 

https://admin.fedoraproject.org/updates/sssd-1.5.12-1.fc14

Comment 12 Kaushik Banerjee 2011-09-21 20:29:56 UTC
Re-opening the bug since the reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=725281 fails to complete:

# ./check_user -n 2000 kau20
Password:
Error in "pam_authenticate": Authentication failure
Authentication failure for user "kau20" in loop 177


Tested on build sssd-1.5.1-52.el6.x86_64

Comment 16 Kaushik Banerjee 2011-10-03 10:07:00 UTC
Works fine on a pristine machine. Will investigate the issue on my test machine and open a separate bug.

# ./check_user -n 2000 kau20
Password: 
User "kau20" authenticated successfully 2000 times

Verified on version:
# rpm -qi sssd | head
Name        : sssd                         Relocations: (not relocatable)
Version     : 1.5.1                             Vendor: Red Hat, Inc.
Release     : 53.el6                        Build Date: Fri 30 Sep 2011 10:10:28 AM EDT
Install Date: Mon 03 Oct 2011 02:57:07 AM EDT      Build Host: hs20-bc2-3.build.redhat.com
Group       : Applications/System           Source RPM: sssd-1.5.1-53.el6.src.rpm
Size        : 3551489                          License: GPLv3+
Signature   : (none)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://fedorahosted.org/sssd/
Summary     : System Security Services Daemon

Comment 17 Masaki Furuta ( RH ) 2011-10-04 11:19:40 UTC
(In reply to comment #16)
Dear Kaushik Banerjee,

Thank you so much for working on this bz, please let me know whtn test packages would be available to have Dell Japan's customer verified. (or we could do it with 52, or 53 as test packages to verify before releasing?

Comment 18 Stephen Gallagher 2011-10-04 12:30:31 UTC
(In reply to comment #17)
> (In reply to comment #16)
> Dear Kaushik Banerjee,
> 
> Thank you so much for working on this bz, please let me know whtn test packages
> would be available to have Dell Japan's customer verified. (or we could do it
> with 52, or 53 as test packages to verify before releasing?

This fix is present in the forthcoming RHEL 6.2 beta. Please install from there and test it out when the beta is released.

Comment 19 Stephen Gallagher 2011-10-04 12:32:12 UTC
Sorry, I forgot to note, this issue is also fixed in an errata for RHEL 6.1: http://rhn.redhat.com/errata/RHBA-2011-1143.html

Comment 28 errata-xmlrpc 2011-12-06 16:39:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1529.html