Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1491082 Details for
Bug 1626782
keystone ldap connection pool never filling up
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Documentation of Keystone LDAP pool behavior
ldap_pool.html (text/html), 19.70 KB, created by
John Dennis
on 2018-10-06 13:18:03 UTC
(
hide
)
Description:
Documentation of Keystone LDAP pool behavior
Filename:
MIME Type:
Creator:
John Dennis
Created:
2018-10-06 13:18:03 UTC
Size:
19.70 KB
patch
obsolete
><?xml version="1.0" encoding="utf-8" ?> ><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> ><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> ><head> ><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> ><meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" /> ><title>Keystone LDAP Pool Overview</title> ><meta name="author" content="John Dennis <jdennis@redhat.com>" /> ><style type="text/css"> > >/* >:Author: David Goodger (goodger@python.org) >:Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $ >:Copyright: This stylesheet has been placed in the public domain. > >Default cascading style sheet for the HTML output of Docutils. > >See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to >customize this style sheet. >*/ > >/* used to remove borders from tables and images */ >.borderless, table.borderless td, table.borderless th { > border: 0 } > >table.borderless td, table.borderless th { > /* Override padding for "table.docutils td" with "! important". > The right padding separates the table cells. */ > padding: 0 0.5em 0 0 ! important } > >.first { > /* Override more specific margin styles with "! important". */ > margin-top: 0 ! important } > >.last, .with-subtitle { > margin-bottom: 0 ! important } > >.hidden { > display: none } > >.subscript { > vertical-align: sub; > font-size: smaller } > >.superscript { > vertical-align: super; > font-size: smaller } > >a.toc-backref { > text-decoration: none ; > color: black } > >blockquote.epigraph { > margin: 2em 5em ; } > >dl.docutils dd { > margin-bottom: 0.5em } > >object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] { > overflow: hidden; >} > >/* Uncomment (and remove this text!) to get bold-faced definition list terms >dl.docutils dt { > font-weight: bold } >*/ > >div.abstract { > margin: 2em 5em } > >div.abstract p.topic-title { > font-weight: bold ; > text-align: center } > >div.admonition, div.attention, div.caution, div.danger, div.error, >div.hint, div.important, div.note, div.tip, div.warning { > margin: 2em ; > border: medium outset ; > padding: 1em } > >div.admonition p.admonition-title, div.hint p.admonition-title, >div.important p.admonition-title, div.note p.admonition-title, >div.tip p.admonition-title { > font-weight: bold ; > font-family: sans-serif } > >div.attention p.admonition-title, div.caution p.admonition-title, >div.danger p.admonition-title, div.error p.admonition-title, >div.warning p.admonition-title, .code .error { > color: red ; > font-weight: bold ; > font-family: sans-serif } > >/* Uncomment (and remove this text!) to get reduced vertical space in > compound paragraphs. >div.compound .compound-first, div.compound .compound-middle { > margin-bottom: 0.5em } > >div.compound .compound-last, div.compound .compound-middle { > margin-top: 0.5em } >*/ > >div.dedication { > margin: 2em 5em ; > text-align: center ; > font-style: italic } > >div.dedication p.topic-title { > font-weight: bold ; > font-style: normal } > >div.figure { > margin-left: 2em ; > margin-right: 2em } > >div.footer, div.header { > clear: both; > font-size: smaller } > >div.line-block { > display: block ; > margin-top: 1em ; > margin-bottom: 1em } > >div.line-block div.line-block { > margin-top: 0 ; > margin-bottom: 0 ; > margin-left: 1.5em } > >div.sidebar { > margin: 0 0 0.5em 1em ; > border: medium outset ; > padding: 1em ; > background-color: #ffffee ; > width: 40% ; > float: right ; > clear: right } > >div.sidebar p.rubric { > font-family: sans-serif ; > font-size: medium } > >div.system-messages { > margin: 5em } > >div.system-messages h1 { > color: red } > >div.system-message { > border: medium outset ; > padding: 1em } > >div.system-message p.system-message-title { > color: red ; > font-weight: bold } > >div.topic { > margin: 2em } > >h1.section-subtitle, h2.section-subtitle, h3.section-subtitle, >h4.section-subtitle, h5.section-subtitle, h6.section-subtitle { > margin-top: 0.4em } > >h1.title { > text-align: center } > >h2.subtitle { > text-align: center } > >hr.docutils { > width: 75% } > >img.align-left, .figure.align-left, object.align-left, table.align-left { > clear: left ; > float: left ; > margin-right: 1em } > >img.align-right, .figure.align-right, object.align-right, table.align-right { > clear: right ; > float: right ; > margin-left: 1em } > >img.align-center, .figure.align-center, object.align-center { > display: block; > margin-left: auto; > margin-right: auto; >} > >table.align-center { > margin-left: auto; > margin-right: auto; >} > >.align-left { > text-align: left } > >.align-center { > clear: both ; > text-align: center } > >.align-right { > text-align: right } > >/* reset inner alignment in figures */ >div.align-right { > text-align: inherit } > >/* div.align-center * { */ >/* text-align: left } */ > >.align-top { > vertical-align: top } > >.align-middle { > vertical-align: middle } > >.align-bottom { > vertical-align: bottom } > >ol.simple, ul.simple { > margin-bottom: 1em } > >ol.arabic { > list-style: decimal } > >ol.loweralpha { > list-style: lower-alpha } > >ol.upperalpha { > list-style: upper-alpha } > >ol.lowerroman { > list-style: lower-roman } > >ol.upperroman { > list-style: upper-roman } > >p.attribution { > text-align: right ; > margin-left: 50% } > >p.caption { > font-style: italic } > >p.credits { > font-style: italic ; > font-size: smaller } > >p.label { > white-space: nowrap } > >p.rubric { > font-weight: bold ; > font-size: larger ; > color: maroon ; > text-align: center } > >p.sidebar-title { > font-family: sans-serif ; > font-weight: bold ; > font-size: larger } > >p.sidebar-subtitle { > font-family: sans-serif ; > font-weight: bold } > >p.topic-title { > font-weight: bold } > >pre.address { > margin-bottom: 0 ; > margin-top: 0 ; > font: inherit } > >pre.literal-block, pre.doctest-block, pre.math, pre.code { > margin-left: 2em ; > margin-right: 2em } > >pre.code .ln { color: grey; } /* line numbers */ >pre.code, code { background-color: #eeeeee } >pre.code .comment, code .comment { color: #5C6576 } >pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold } >pre.code .literal.string, code .literal.string { color: #0C5404 } >pre.code .name.builtin, code .name.builtin { color: #352B84 } >pre.code .deleted, code .deleted { background-color: #DEB0A1} >pre.code .inserted, code .inserted { background-color: #A3D289} > >span.classifier { > font-family: sans-serif ; > font-style: oblique } > >span.classifier-delimiter { > font-family: sans-serif ; > font-weight: bold } > >span.interpreted { > font-family: sans-serif } > >span.option { > white-space: nowrap } > >span.pre { > white-space: pre } > >span.problematic { > color: red } > >span.section-subtitle { > /* font-size relative to parent (h1..h6 element) */ > font-size: 80% } > >table.citation { > border-left: solid 1px gray; > margin-left: 1px } > >table.docinfo { > margin: 2em 4em } > >table.docutils { > margin-top: 0.5em ; > margin-bottom: 0.5em } > >table.footnote { > border-left: solid 1px black; > margin-left: 1px } > >table.docutils td, table.docutils th, >table.docinfo td, table.docinfo th { > padding-left: 0.5em ; > padding-right: 0.5em ; > vertical-align: top } > >table.docutils th.field-name, table.docinfo th.docinfo-name { > font-weight: bold ; > text-align: left ; > white-space: nowrap ; > padding-left: 0 } > >/* "booktabs" style (no vertical lines) */ >table.docutils.booktabs { > border: 0px; > border-top: 2px solid; > border-bottom: 2px solid; > border-collapse: collapse; >} >table.docutils.booktabs * { > border: 0px; >} >table.docutils.booktabs th { > border-bottom: thin solid; > text-align: left; >} > >h1 tt.docutils, h2 tt.docutils, h3 tt.docutils, >h4 tt.docutils, h5 tt.docutils, h6 tt.docutils { > font-size: 100% } > >ul.auto-toc { > list-style-type: none } > ></style> ></head> ><body> ><div class="document" id="keystone-ldap-pool-overview"> ><h1 class="title">Keystone LDAP Pool Overview</h1> ><table class="docinfo" frame="void" rules="none"> ><col class="docinfo-name" /> ><col class="docinfo-content" /> ><tbody valign="top"> ><tr><th class="docinfo-name">Author:</th> ><td>John Dennis <<a class="reference external" href="mailto:jdennis@redhat.com">jdennis@redhat.com</a>></td></tr> ><tr><th class="docinfo-name">Version:</th> ><td>1.0</td></tr> ></tbody> ></table> ><div class="section" id="executive-summary"> ><h1>Executive Summary</h1> ><p>There are many misconceptions concerning the LDAP pool functionality >in Keystone and what advantages it offers. In general the LDAP pool >does <em>not</em> increase performance via parallelism. Instead performance >gains via the LDAP pool derive from the fact connections to the LDAP >server are "kept alive" across multiple LDAP requests thus eliminating >the overhead of setting up and tearing down the connection to the LDAP >server. It is unlikely the LDAP pool size will grow beyond 1 (see ><a class="reference internal" href="#why-pool-size-of-1">why pool size of 1</a>)</p> ></div> ><div class="section" id="what-is-an-ldap-pool"> ><span id="ldap-pool-definition"></span><h1>What is an LDAP pool?</h1> ><p>An LDAP pool is a collection of long lived <a class="footnote-reference" href="#id5" id="id1">[1]</a> LDAP connections >available to the threads executing in a single Python interpreter >process. Each LDAP connection in the pool is uniquely identified by >the (user,password) 2-tuple used to bind the LDAP connection. All >members of the LDAP pool connect to the same LDAP server identified by >the Keystone LDAP <cite>url</cite> configuration parameter. <a class="footnote-reference" href="#id6" id="id2">[2]</a></p> ><div class="section" id="how-is-the-ldap-pool-managed"> ><span id="pool-management"></span><h2>How is the LDAP pool managed?</h2> ><p>When a Python thread needs an LDAP connection the LDAP pool is locked >and searched for the first <em>inactive</em> connection whose (user,password) >2-tuple matches. Inactive pool members whose ><cite>pool_connection_lifetime</cite> have expired when the search is performed >are ejected from the pool and hence from further consideration. If no ><em>inactive</em> connections with the requisite (user,password) 2-tuple is >found then a new pool member is added. Independent of whether a >candidate pool member was found in the pool or a new pool member was >created the selected member is immediately marked as <em>active</em> and >returned for use. If no candidate pool members can be located and the >pool is at it's <cite>pool_size</cite> maximum then a <cite>MaxConnectionReachedError</cite> >is raised.</p> ><p>After the connection pool member is returned it is used to perform a ><strong>synchronous</strong> LDAP request. During this time the Python thread ><strong>blocks</strong> until the result is returned. The LDAP pool member used to >perform the synchronous LDAP request is then marked as <em>inactive</em> at >the completion of the LDAP request thus becoming available to other >threads in the same Python interpreter process. At this point any >thread in the Python interpreter process can then use the already open >connection belonging the newly returned <em>inactive</em> pool member to >perform the next LDAP request in it's thread without having to perform >a costly connect operation (e.g. keep-alive).</p> ></div> ></div> ><div class="section" id="httpd-servers-wsgi-daemons-python-interpreter-processes-and-threads"> ><h1>HTTPD Servers, WSGI Daemons, Python Interpreter processes and threads</h1> ><p>In the discussion on <a class="reference internal" href="#pool-management">Pool Management</a> we learned that the LDAP pool >is shared by threads in one single Python interpreter process. In a >typical Keystone deployment there are many Python interpreter >processes available to service Keystone requests hence there are many >LDAP pools. This is contrast to the often erroneous assumption members >of the LDAP pool are shared between processes thus forming one large >aggregate.</p> ><p>Keystone is run inside a HTTPD server (e.g. Apache). Because Keystone >is a Python Application it is serviced by HTTPD via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> >module which implements <a class="reference external" href="https://wsgi.readthedocs.io/en/latest/">WSGI</a> (Web Server Gateway Interface).</p> ><p>Each HTTPD server may fork multiple copies of itself to simultaneously >service incoming HTTP requests. Within an HTTPD server there may be >one or more Python interpreter processes running due to the >configuration of <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a>. Typically <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> is run in <cite>daemon >mode</cite> via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html">WSGIDaemonProcess</a> configuration >parameter. The parent HTTPD server will fork a number of Python >interpreter processes each of which may run one or more threads. This >is controlled by this HTTPD configuration parameter:</p> ><pre class="literal-block"> >WSGIDaemonProcess processes=num threads=num ></pre> ><p>Exactly how many Python interpreter processes are spawned depends on a >somewhat complicated set of relationships between HTTPD configuration >directives and <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> configuration parameters. This is explained in >detail in the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html">WSGI Process & Thread Documentation</a>.</p> ><p>In addition to the multiple processes and threads servicing Keystone >requests in a single HTTPD instance there may be multiple HTTPD >instances if a load balancer is present in an High Availability (HA) >deployment.</p> ><p>The important point to remember is in most Keystone deployments there >will be <strong>many Python interpreter processes, each of which will have >it's own LDAP pool.</strong> Requests to the Keystone server will be >serviced in round-robin fashion across each of these Python interpreter >processes.</p> ><p>Recalling that the LDAP pool is available for <strong>threads</strong> within a >single Python interpreter process we note the following consequences:</p> ><ul class="simple" id="when-wsgidaemonprocess-threads-1"> ><li>If the number of threads in a Python interpreter process is 1 >(e.g. <cite>WSGIDaemonProcess threads=1</cite>) the LDAP pool size will never >exceed 1. This is because only one thread shares the pool. The >pool size will be zero if it's never been utilized or if the ><cite>pool_connection_lifetime</cite> <a class="footnote-reference" href="#id5" id="id3">[1]</a> of the single pool member >expires.</li> ></ul> ><ul class="simple" id="why-pool-size-of-1"> ><li>For the size of the LDAP pool to grow to greater than one the >following must occur: The number of threads must be greater than >one. The number of available Python interpreter processes servicing >Keystone requests must be saturated or some other condition holds >such that a second thread within a Python interpreter process is run >to service the request <strong>and</strong> each running thread in the Python >interpreter process is <strong>simultaneously</strong> attempting to perform a >LDAP request. <a class="footnote-reference" href="#id7" id="id4">[3]</a> The likelihood of satisfying all the conditions >necessary to grow the pool is low. It would be exacerbated either by >slow Keystone execution, heavy Keystone load, or slow responses from >the LDAP server.</li> ></ul> ><p>A single Keystone request typically induces multiple LDAP requests to >satisfy the Keystone request. The Keystone request will execute within >only one Python thread. Thus the multiple LDAP operations need to >satisfy the Keystone request will execute <em>sequentially</em> within the >thread. This is further enforced by the fact the LDAP pool demands >each LDAP request be <strong>synchronous</strong>. <em>Hence there is no parallel >overlap when executing LDAP operations for a given Keystone request.</em></p> ><div class="admonition note"> ><p class="first admonition-title">Note</p> ><p class="last">Most Keystone deployments set <cite>WSGIDaemonProcess threads=1</cite> >enforcing a one-to-one relationship between Python >interpreter processes and threads with the consequence the >LDAP pool size will never exceed 1. (see <a class="reference internal" href="#when-wsgidaemonprocess-threads-1">When >WSGIDaemonProcess threads=1</a>)</p> ></div> ></div> ><div class="section" id="what-is-the-advantage-of-using-ldap-pools"> ><h1>What is the advantage of using LDAP pools?</h1> ><p>The primary advantage of using the Keystone LDAP pool is the pool's >ability to keep a continuous LDAP connection open for use by the >threads in a single Python interpreter process. Because a Keystone >request is serviced by one thread and because during the servicing of >that request multiple LDAP operations will be executed in a sequential >synchronous fashion access to the LDAP server will be much faster >because the same connection will be left open and utilized thus >avoiding the overhead of connection setup and tear down.</p> ></div> ><div class="section" id="footnotes"> ><h1>Footnotes</h1> ><table class="docutils footnote" frame="void" id="id5" rules="none"> ><colgroup><col class="label" /><col /></colgroup> ><tbody valign="top"> ><tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#id1">1</a>, <a class="fn-backref" href="#id3">2</a>)</em> <p>The duration of a connection to a LDAP server in the LDAP pool >is controlled by the <cite>pool_connection_lifetime</cite> configuration >variable.</p> ><blockquote class="last"> ><cite>pool_connection_lifetime</cite>: The maximum connection lifetime >to the LDAP server in seconds. When this lifetime is >exceeded, the connection will be unbound and removed from >the connection pool. This option has no effect unless ><cite>[ldap] use_pool</cite> is also enabled.</blockquote> ></td></tr> ></tbody> ></table> ><table class="docutils footnote" frame="void" id="id6" rules="none"> ><colgroup><col class="label" /><col /></colgroup> ><tbody valign="top"> ><tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><p class="first">The LDAP <cite>url</cite> configuration parameter permits you to specify a >list of LDAP server URL's. This feature has a confusing >interaction with pools and automatic reconnect >behavior. Internally Keystone treats the LDAP <cite>url</cite> parameter >as a simple string, it does <em>not</em> break it apart into independent >LDAP URL's where each LDAP URL is managed independently. The >LDAP URL comma separated list is passed directly into the >OpenLDAP client C library which then breaks the list into >independent LDAP URL's. When a connection to an LDAP server is >attempted the OpenLDAP client C library iterates over the >members of the list attempting to connect to each in turn. Thus >the Keystone code does not know which LDAP server you're >actually connected to if you specify multiple servers in the ><cite>url</cite> configuration parameter.</p> ><blockquote class="last"> ><cite>url</cite>: URL(s) for connecting to the LDAP server. Multiple >LDAP URLs may be specified as a comma separated string. The >first URL to successfully bind is used for the connection.</blockquote> ></td></tr> ></tbody> ></table> ><table class="docutils footnote" frame="void" id="id7" rules="none"> ><colgroup><col class="label" /><col /></colgroup> ><tbody valign="top"> ><tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>The pool size would also grow if distinct (user,password) >2-tuples are used to bind the connection because the pool keeps >connections segregated by their bind parameters. However the >current Keystone LDAP configuration prevents multiple distinct >bind credentials hence the pool can only grow when threads in a >single Python interpreter process <strong>simultaneously</strong> compete >for a connection.</td></tr> ></tbody> ></table> ></div> ></div> ></body> ></html>
<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" /> <title>Keystone LDAP Pool Overview</title> <meta name="author" content="John Dennis <jdennis@redhat.com>" /> <style type="text/css"> /* :Author: David Goodger (goodger@python.org) :Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $ :Copyright: This stylesheet has been placed in the public domain. Default cascading style sheet for the HTML output of Docutils. See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to customize this style sheet. */ /* used to remove borders from tables and images */ .borderless, table.borderless td, table.borderless th { border: 0 } table.borderless td, table.borderless th { /* Override padding for "table.docutils td" with "! important". The right padding separates the table cells. */ padding: 0 0.5em 0 0 ! important } .first { /* Override more specific margin styles with "! important". */ margin-top: 0 ! important } .last, .with-subtitle { margin-bottom: 0 ! important } .hidden { display: none } .subscript { vertical-align: sub; font-size: smaller } .superscript { vertical-align: super; font-size: smaller } a.toc-backref { text-decoration: none ; color: black } blockquote.epigraph { margin: 2em 5em ; } dl.docutils dd { margin-bottom: 0.5em } object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] { overflow: hidden; } /* Uncomment (and remove this text!) to get bold-faced definition list terms dl.docutils dt { font-weight: bold } */ div.abstract { margin: 2em 5em } div.abstract p.topic-title { font-weight: bold ; text-align: center } div.admonition, div.attention, div.caution, div.danger, div.error, div.hint, div.important, div.note, div.tip, div.warning { margin: 2em ; border: medium outset ; padding: 1em } div.admonition p.admonition-title, div.hint p.admonition-title, div.important p.admonition-title, div.note p.admonition-title, div.tip p.admonition-title { font-weight: bold ; font-family: sans-serif } div.attention p.admonition-title, div.caution p.admonition-title, div.danger p.admonition-title, div.error p.admonition-title, div.warning p.admonition-title, .code .error { color: red ; font-weight: bold ; font-family: sans-serif } /* Uncomment (and remove this text!) to get reduced vertical space in compound paragraphs. div.compound .compound-first, div.compound .compound-middle { margin-bottom: 0.5em } div.compound .compound-last, div.compound .compound-middle { margin-top: 0.5em } */ div.dedication { margin: 2em 5em ; text-align: center ; font-style: italic } div.dedication p.topic-title { font-weight: bold ; font-style: normal } div.figure { margin-left: 2em ; margin-right: 2em } div.footer, div.header { clear: both; font-size: smaller } div.line-block { display: block ; margin-top: 1em ; margin-bottom: 1em } div.line-block div.line-block { margin-top: 0 ; margin-bottom: 0 ; margin-left: 1.5em } div.sidebar { margin: 0 0 0.5em 1em ; border: medium outset ; padding: 1em ; background-color: #ffffee ; width: 40% ; float: right ; clear: right } div.sidebar p.rubric { font-family: sans-serif ; font-size: medium } div.system-messages { margin: 5em } div.system-messages h1 { color: red } div.system-message { border: medium outset ; padding: 1em } div.system-message p.system-message-title { color: red ; font-weight: bold } div.topic { margin: 2em } h1.section-subtitle, h2.section-subtitle, h3.section-subtitle, h4.section-subtitle, h5.section-subtitle, h6.section-subtitle { margin-top: 0.4em } h1.title { text-align: center } h2.subtitle { text-align: center } hr.docutils { width: 75% } img.align-left, .figure.align-left, object.align-left, table.align-left { clear: left ; float: left ; margin-right: 1em } img.align-right, .figure.align-right, object.align-right, table.align-right { clear: right ; float: right ; margin-left: 1em } img.align-center, .figure.align-center, object.align-center { display: block; margin-left: auto; margin-right: auto; } table.align-center { margin-left: auto; margin-right: auto; } .align-left { text-align: left } .align-center { clear: both ; text-align: center } .align-right { text-align: right } /* reset inner alignment in figures */ div.align-right { text-align: inherit } /* div.align-center * { */ /* text-align: left } */ .align-top { vertical-align: top } .align-middle { vertical-align: middle } .align-bottom { vertical-align: bottom } ol.simple, ul.simple { margin-bottom: 1em } ol.arabic { list-style: decimal } ol.loweralpha { list-style: lower-alpha } ol.upperalpha { list-style: upper-alpha } ol.lowerroman { list-style: lower-roman } ol.upperroman { list-style: upper-roman } p.attribution { text-align: right ; margin-left: 50% } p.caption { font-style: italic } p.credits { font-style: italic ; font-size: smaller } p.label { white-space: nowrap } p.rubric { font-weight: bold ; font-size: larger ; color: maroon ; text-align: center } p.sidebar-title { font-family: sans-serif ; font-weight: bold ; font-size: larger } p.sidebar-subtitle { font-family: sans-serif ; font-weight: bold } p.topic-title { font-weight: bold } pre.address { margin-bottom: 0 ; margin-top: 0 ; font: inherit } pre.literal-block, pre.doctest-block, pre.math, pre.code { margin-left: 2em ; margin-right: 2em } pre.code .ln { color: grey; } /* line numbers */ pre.code, code { background-color: #eeeeee } pre.code .comment, code .comment { color: #5C6576 } pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold } pre.code .literal.string, code .literal.string { color: #0C5404 } pre.code .name.builtin, code .name.builtin { color: #352B84 } pre.code .deleted, code .deleted { background-color: #DEB0A1} pre.code .inserted, code .inserted { background-color: #A3D289} span.classifier { font-family: sans-serif ; font-style: oblique } span.classifier-delimiter { font-family: sans-serif ; font-weight: bold } span.interpreted { font-family: sans-serif } span.option { white-space: nowrap } span.pre { white-space: pre } span.problematic { color: red } span.section-subtitle { /* font-size relative to parent (h1..h6 element) */ font-size: 80% } table.citation { border-left: solid 1px gray; margin-left: 1px } table.docinfo { margin: 2em 4em } table.docutils { margin-top: 0.5em ; margin-bottom: 0.5em } table.footnote { border-left: solid 1px black; margin-left: 1px } table.docutils td, table.docutils th, table.docinfo td, table.docinfo th { padding-left: 0.5em ; padding-right: 0.5em ; vertical-align: top } table.docutils th.field-name, table.docinfo th.docinfo-name { font-weight: bold ; text-align: left ; white-space: nowrap ; padding-left: 0 } /* "booktabs" style (no vertical lines) */ table.docutils.booktabs { border: 0px; border-top: 2px solid; border-bottom: 2px solid; border-collapse: collapse; } table.docutils.booktabs * { border: 0px; } table.docutils.booktabs th { border-bottom: thin solid; text-align: left; } h1 tt.docutils, h2 tt.docutils, h3 tt.docutils, h4 tt.docutils, h5 tt.docutils, h6 tt.docutils { font-size: 100% } ul.auto-toc { list-style-type: none } </style> </head> <body> <div class="document" id="keystone-ldap-pool-overview"> <h1 class="title">Keystone LDAP Pool Overview</h1> <table class="docinfo" frame="void" rules="none"> <col class="docinfo-name" /> <col class="docinfo-content" /> <tbody valign="top"> <tr><th class="docinfo-name">Author:</th> <td>John Dennis <<a class="reference external" href="mailto:jdennis@redhat.com">jdennis@redhat.com</a>></td></tr> <tr><th class="docinfo-name">Version:</th> <td>1.0</td></tr> </tbody> </table> <div class="section" id="executive-summary"> <h1>Executive Summary</h1> <p>There are many misconceptions concerning the LDAP pool functionality in Keystone and what advantages it offers. In general the LDAP pool does <em>not</em> increase performance via parallelism. Instead performance gains via the LDAP pool derive from the fact connections to the LDAP server are "kept alive" across multiple LDAP requests thus eliminating the overhead of setting up and tearing down the connection to the LDAP server. It is unlikely the LDAP pool size will grow beyond 1 (see <a class="reference internal" href="#why-pool-size-of-1">why pool size of 1</a>)</p> </div> <div class="section" id="what-is-an-ldap-pool"> <span id="ldap-pool-definition"></span><h1>What is an LDAP pool?</h1> <p>An LDAP pool is a collection of long lived <a class="footnote-reference" href="#id5" id="id1">[1]</a> LDAP connections available to the threads executing in a single Python interpreter process. Each LDAP connection in the pool is uniquely identified by the (user,password) 2-tuple used to bind the LDAP connection. All members of the LDAP pool connect to the same LDAP server identified by the Keystone LDAP <cite>url</cite> configuration parameter. <a class="footnote-reference" href="#id6" id="id2">[2]</a></p> <div class="section" id="how-is-the-ldap-pool-managed"> <span id="pool-management"></span><h2>How is the LDAP pool managed?</h2> <p>When a Python thread needs an LDAP connection the LDAP pool is locked and searched for the first <em>inactive</em> connection whose (user,password) 2-tuple matches. Inactive pool members whose <cite>pool_connection_lifetime</cite> have expired when the search is performed are ejected from the pool and hence from further consideration. If no <em>inactive</em> connections with the requisite (user,password) 2-tuple is found then a new pool member is added. Independent of whether a candidate pool member was found in the pool or a new pool member was created the selected member is immediately marked as <em>active</em> and returned for use. If no candidate pool members can be located and the pool is at it's <cite>pool_size</cite> maximum then a <cite>MaxConnectionReachedError</cite> is raised.</p> <p>After the connection pool member is returned it is used to perform a <strong>synchronous</strong> LDAP request. During this time the Python thread <strong>blocks</strong> until the result is returned. The LDAP pool member used to perform the synchronous LDAP request is then marked as <em>inactive</em> at the completion of the LDAP request thus becoming available to other threads in the same Python interpreter process. At this point any thread in the Python interpreter process can then use the already open connection belonging the newly returned <em>inactive</em> pool member to perform the next LDAP request in it's thread without having to perform a costly connect operation (e.g. keep-alive).</p> </div> </div> <div class="section" id="httpd-servers-wsgi-daemons-python-interpreter-processes-and-threads"> <h1>HTTPD Servers, WSGI Daemons, Python Interpreter processes and threads</h1> <p>In the discussion on <a class="reference internal" href="#pool-management">Pool Management</a> we learned that the LDAP pool is shared by threads in one single Python interpreter process. In a typical Keystone deployment there are many Python interpreter processes available to service Keystone requests hence there are many LDAP pools. This is contrast to the often erroneous assumption members of the LDAP pool are shared between processes thus forming one large aggregate.</p> <p>Keystone is run inside a HTTPD server (e.g. Apache). Because Keystone is a Python Application it is serviced by HTTPD via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> module which implements <a class="reference external" href="https://wsgi.readthedocs.io/en/latest/">WSGI</a> (Web Server Gateway Interface).</p> <p>Each HTTPD server may fork multiple copies of itself to simultaneously service incoming HTTP requests. Within an HTTPD server there may be one or more Python interpreter processes running due to the configuration of <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a>. Typically <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> is run in <cite>daemon mode</cite> via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html">WSGIDaemonProcess</a> configuration parameter. The parent HTTPD server will fork a number of Python interpreter processes each of which may run one or more threads. This is controlled by this HTTPD configuration parameter:</p> <pre class="literal-block"> WSGIDaemonProcess processes=num threads=num </pre> <p>Exactly how many Python interpreter processes are spawned depends on a somewhat complicated set of relationships between HTTPD configuration directives and <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> configuration parameters. This is explained in detail in the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html">WSGI Process & Thread Documentation</a>.</p> <p>In addition to the multiple processes and threads servicing Keystone requests in a single HTTPD instance there may be multiple HTTPD instances if a load balancer is present in an High Availability (HA) deployment.</p> <p>The important point to remember is in most Keystone deployments there will be <strong>many Python interpreter processes, each of which will have it's own LDAP pool.</strong> Requests to the Keystone server will be serviced in round-robin fashion across each of these Python interpreter processes.</p> <p>Recalling that the LDAP pool is available for <strong>threads</strong> within a single Python interpreter process we note the following consequences:</p> <ul class="simple" id="when-wsgidaemonprocess-threads-1"> <li>If the number of threads in a Python interpreter process is 1 (e.g. <cite>WSGIDaemonProcess threads=1</cite>) the LDAP pool size will never exceed 1. This is because only one thread shares the pool. The pool size will be zero if it's never been utilized or if the <cite>pool_connection_lifetime</cite> <a class="footnote-reference" href="#id5" id="id3">[1]</a> of the single pool member expires.</li> </ul> <ul class="simple" id="why-pool-size-of-1"> <li>For the size of the LDAP pool to grow to greater than one the following must occur: The number of threads must be greater than one. The number of available Python interpreter processes servicing Keystone requests must be saturated or some other condition holds such that a second thread within a Python interpreter process is run to service the request <strong>and</strong> each running thread in the Python interpreter process is <strong>simultaneously</strong> attempting to perform a LDAP request. <a class="footnote-reference" href="#id7" id="id4">[3]</a> The likelihood of satisfying all the conditions necessary to grow the pool is low. It would be exacerbated either by slow Keystone execution, heavy Keystone load, or slow responses from the LDAP server.</li> </ul> <p>A single Keystone request typically induces multiple LDAP requests to satisfy the Keystone request. The Keystone request will execute within only one Python thread. Thus the multiple LDAP operations need to satisfy the Keystone request will execute <em>sequentially</em> within the thread. This is further enforced by the fact the LDAP pool demands each LDAP request be <strong>synchronous</strong>. <em>Hence there is no parallel overlap when executing LDAP operations for a given Keystone request.</em></p> <div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">Most Keystone deployments set <cite>WSGIDaemonProcess threads=1</cite> enforcing a one-to-one relationship between Python interpreter processes and threads with the consequence the LDAP pool size will never exceed 1. (see <a class="reference internal" href="#when-wsgidaemonprocess-threads-1">When WSGIDaemonProcess threads=1</a>)</p> </div> </div> <div class="section" id="what-is-the-advantage-of-using-ldap-pools"> <h1>What is the advantage of using LDAP pools?</h1> <p>The primary advantage of using the Keystone LDAP pool is the pool's ability to keep a continuous LDAP connection open for use by the threads in a single Python interpreter process. Because a Keystone request is serviced by one thread and because during the servicing of that request multiple LDAP operations will be executed in a sequential synchronous fashion access to the LDAP server will be much faster because the same connection will be left open and utilized thus avoiding the overhead of connection setup and tear down.</p> </div> <div class="section" id="footnotes"> <h1>Footnotes</h1> <table class="docutils footnote" frame="void" id="id5" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#id1">1</a>, <a class="fn-backref" href="#id3">2</a>)</em> <p>The duration of a connection to a LDAP server in the LDAP pool is controlled by the <cite>pool_connection_lifetime</cite> configuration variable.</p> <blockquote class="last"> <cite>pool_connection_lifetime</cite>: The maximum connection lifetime to the LDAP server in seconds. When this lifetime is exceeded, the connection will be unbound and removed from the connection pool. This option has no effect unless <cite>[ldap] use_pool</cite> is also enabled.</blockquote> </td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id6" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><p class="first">The LDAP <cite>url</cite> configuration parameter permits you to specify a list of LDAP server URL's. This feature has a confusing interaction with pools and automatic reconnect behavior. Internally Keystone treats the LDAP <cite>url</cite> parameter as a simple string, it does <em>not</em> break it apart into independent LDAP URL's where each LDAP URL is managed independently. The LDAP URL comma separated list is passed directly into the OpenLDAP client C library which then breaks the list into independent LDAP URL's. When a connection to an LDAP server is attempted the OpenLDAP client C library iterates over the members of the list attempting to connect to each in turn. Thus the Keystone code does not know which LDAP server you're actually connected to if you specify multiple servers in the <cite>url</cite> configuration parameter.</p> <blockquote class="last"> <cite>url</cite>: URL(s) for connecting to the LDAP server. Multiple LDAP URLs may be specified as a comma separated string. The first URL to successfully bind is used for the connection.</blockquote> </td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id7" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>The pool size would also grow if distinct (user,password) 2-tuples are used to bind the connection because the pool keeps connections segregated by their bind parameters. However the current Keystone LDAP configuration prevents multiple distinct bind credentials hence the pool can only grow when threads in a single Python interpreter process <strong>simultaneously</strong> compete for a connection.</td></tr> </tbody> </table> </div> </div> </body> </html>
View Attachment As Raw
Actions:
View
Attachments on
bug 1626782
: 1491082