Bug 1368467

Summary: sssd runs out of available child slots and starts queuing requests in proxy mode
Product: Red Hat Enterprise Linux 7 Reporter: Thorsten Scherf <tscherf>
Component: sssdAssignee: Petr Čech <pcech>
Status: CLOSED ERRATA QA Contact: Amith <apeetham>
Severity: low Docs Contact:
Priority: low    
Version: 7.2CC: grajaiya, jhrozek, lslebodn, mkosek, mzidek, pbrezina, pcech, sssd-qe
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: sssd-1.15.0-2.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1369079 (view as bug list) Environment:
Last Closed: 2017-08-01 08:58:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1369079    

Description Thorsten Scherf 2016-08-19 13:32:15 UTC
Description of problem:

Authentication in proxy mode fails under heavy load 

(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [be_pam_handler] (4): Got request with the following data
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): command: PAM_AUTHENTICATE
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): domain: indis
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): user: APPUSER
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): service: vsftpd_nas
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): tty: ftp
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): ruser: APPUSER
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): rhost: nastf2.testfactory.copergmps
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): authtok type: 1
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): authtok size: 7
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): newauthtok type: 0
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): newauthtok size: 0
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): priv: 0
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [pam_print_data] (4): cli_pid: 27026
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [proxy_child_send] (8): Queueing request [19]
(Thu Aug 18 11:09:25 2016) [sssd[be[indis]]] [proxy_child_send] (8): All available child slots are full, queuing request

The request is turn the hardcoded number of preforked childs into a config option.

$ grep -B1  max_children src/providers/proxy/proxy_init.c
    /* Set up request hash table */
    /* FIXME: get max_children from configuration file */
    auth_ctx->max_children = 10;

Version-Release number of selected component (if applicable):

verified on RHEL-5.11, but some code also exists in master.


How reproducible:

for I in $(seq 1 350); do 
	lftp -e 'ls; quit' ftp://FTP-SERVER -u LDAP-USER,LDAP-USER-KRB5-PASS 1>/dev/null & 
done

while pidof lftp >/dev/null; do
	echo -n '.'
	sleep 1
done
echo 


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Jakub Hrozek 2016-08-22 14:40:32 UTC
Upstream ticket:
https://fedorahosted.org/sssd/ticket/3153

Comment 2 Lukas Slebodnik 2016-09-13 14:30:28 UTC
master:
* aef0171e0bdc9a683958d69c7ee984fb10cd5de7

Comment 4 Amith 2017-05-31 12:51:32 UTC
Verified bug on SSSD Version: sssd-1.15.2-37.el7.x86_64

This bug was automated after verifying BZ1369079. We successfully executed the code on RHEL7.4 without any errors.
The automated code is included within task /sssd/rhel69/client/proxy_provider/misc/apeetham and titled as "
sssd-runs-out-of-available-child-slots-and-starts-queuing-requests-in-proxy-mode-bz1369079"

This code will be merged into RHEL7.4 master branch shortly.
 
See the results in task /sssd/rhel69/client/proxy_provider/misc/apeetham in beaker job: https://beaker.engineering.redhat.com/jobs/1880650

Comment 5 errata-xmlrpc 2017-08-01 08:58:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2294