Attachment 1491082 Details for Bug 1626782 – Documentation of Keystone LDAP pool behavior

Documentation of Keystone LDAP pool behavior

ldap_pool.html (text/html), 19.70 KB, created by John Dennis on 2018-10-06 13:18:03 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: John Dennis

Created: 2018-10-06 13:18:03 UTC

Size: 19.70 KB

patch

obsolete

><?xml version="1.0" encoding="utf-8" ?>
><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
><head>
><meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
><meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" />
><title>Keystone LDAP Pool Overview</title>
><meta name="author" content="John Dennis &lt;jdennis&#64;redhat.com&gt;" />
><style type="text/css">
>
>/*
>:Author: David Goodger (goodger@python.org)
>:Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $
>:Copyright: This stylesheet has been placed in the public domain.
>
>Default cascading style sheet for the HTML output of Docutils.
>
>See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
>customize this style sheet.
>*/
>
>/* used to remove borders from tables and images */
>.borderless, table.borderless td, table.borderless th {
>  border: 0 }
>
>table.borderless td, table.borderless th {
>  /* Override padding for "table.docutils td" with "! important".
>     The right padding separates the table cells. */
>  padding: 0 0.5em 0 0 ! important }
>
>.first {
>  /* Override more specific margin styles with "! important". */
>  margin-top: 0 ! important }
>
>.last, .with-subtitle {
>  margin-bottom: 0 ! important }
>
>.hidden {
>  display: none }
>
>.subscript {
>  vertical-align: sub;
>  font-size: smaller }
>
>.superscript {
>  vertical-align: super;
>  font-size: smaller }
>
>a.toc-backref {
>  text-decoration: none ;
>  color: black }
>
>blockquote.epigraph {
>  margin: 2em 5em ; }
>
>dl.docutils dd {
>  margin-bottom: 0.5em }
>
>object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
>  overflow: hidden;
>}
>
>/* Uncomment (and remove this text!) to get bold-faced definition list terms
>dl.docutils dt {
>  font-weight: bold }
>*/
>
>div.abstract {
>  margin: 2em 5em }
>
>div.abstract p.topic-title {
>  font-weight: bold ;
>  text-align: center }
>
>div.admonition, div.attention, div.caution, div.danger, div.error,
>div.hint, div.important, div.note, div.tip, div.warning {
>  margin: 2em ;
>  border: medium outset ;
>  padding: 1em }
>
>div.admonition p.admonition-title, div.hint p.admonition-title,
>div.important p.admonition-title, div.note p.admonition-title,
>div.tip p.admonition-title {
>  font-weight: bold ;
>  font-family: sans-serif }
>
>div.attention p.admonition-title, div.caution p.admonition-title,
>div.danger p.admonition-title, div.error p.admonition-title,
>div.warning p.admonition-title, .code .error {
>  color: red ;
>  font-weight: bold ;
>  font-family: sans-serif }
>
>/* Uncomment (and remove this text!) to get reduced vertical space in
>   compound paragraphs.
>div.compound .compound-first, div.compound .compound-middle {
>  margin-bottom: 0.5em }
>
>div.compound .compound-last, div.compound .compound-middle {
>  margin-top: 0.5em }
>*/
>
>div.dedication {
>  margin: 2em 5em ;
>  text-align: center ;
>  font-style: italic }
>
>div.dedication p.topic-title {
>  font-weight: bold ;
>  font-style: normal }
>
>div.figure {
>  margin-left: 2em ;
>  margin-right: 2em }
>
>div.footer, div.header {
>  clear: both;
>  font-size: smaller }
>
>div.line-block {
>  display: block ;
>  margin-top: 1em ;
>  margin-bottom: 1em }
>
>div.line-block div.line-block {
>  margin-top: 0 ;
>  margin-bottom: 0 ;
>  margin-left: 1.5em }
>
>div.sidebar {
>  margin: 0 0 0.5em 1em ;
>  border: medium outset ;
>  padding: 1em ;
>  background-color: #ffffee ;
>  width: 40% ;
>  float: right ;
>  clear: right }
>
>div.sidebar p.rubric {
>  font-family: sans-serif ;
>  font-size: medium }
>
>div.system-messages {
>  margin: 5em }
>
>div.system-messages h1 {
>  color: red }
>
>div.system-message {
>  border: medium outset ;
>  padding: 1em }
>
>div.system-message p.system-message-title {
>  color: red ;
>  font-weight: bold }
>
>div.topic {
>  margin: 2em }
>
>h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
>h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
>  margin-top: 0.4em }
>
>h1.title {
>  text-align: center }
>
>h2.subtitle {
>  text-align: center }
>
>hr.docutils {
>  width: 75% }
>
>img.align-left, .figure.align-left, object.align-left, table.align-left {
>  clear: left ;
>  float: left ;
>  margin-right: 1em }
>
>img.align-right, .figure.align-right, object.align-right, table.align-right {
>  clear: right ;
>  float: right ;
>  margin-left: 1em }
>
>img.align-center, .figure.align-center, object.align-center {
>  display: block;
>  margin-left: auto;
>  margin-right: auto;
>}
>
>table.align-center {
>  margin-left: auto;
>  margin-right: auto;
>}
>
>.align-left {
>  text-align: left }
>
>.align-center {
>  clear: both ;
>  text-align: center }
>
>.align-right {
>  text-align: right }
>
>/* reset inner alignment in figures */
>div.align-right {
>  text-align: inherit }
>
>/* div.align-center * { */
>/*   text-align: left } */
>
>.align-top    {
>  vertical-align: top }
>
>.align-middle {
>  vertical-align: middle }
>
>.align-bottom {
>  vertical-align: bottom }
>
>ol.simple, ul.simple {
>  margin-bottom: 1em }
>
>ol.arabic {
>  list-style: decimal }
>
>ol.loweralpha {
>  list-style: lower-alpha }
>
>ol.upperalpha {
>  list-style: upper-alpha }
>
>ol.lowerroman {
>  list-style: lower-roman }
>
>ol.upperroman {
>  list-style: upper-roman }
>
>p.attribution {
>  text-align: right ;
>  margin-left: 50% }
>
>p.caption {
>  font-style: italic }
>
>p.credits {
>  font-style: italic ;
>  font-size: smaller }
>
>p.label {
>  white-space: nowrap }
>
>p.rubric {
>  font-weight: bold ;
>  font-size: larger ;
>  color: maroon ;
>  text-align: center }
>
>p.sidebar-title {
>  font-family: sans-serif ;
>  font-weight: bold ;
>  font-size: larger }
>
>p.sidebar-subtitle {
>  font-family: sans-serif ;
>  font-weight: bold }
>
>p.topic-title {
>  font-weight: bold }
>
>pre.address {
>  margin-bottom: 0 ;
>  margin-top: 0 ;
>  font: inherit }
>
>pre.literal-block, pre.doctest-block, pre.math, pre.code {
>  margin-left: 2em ;
>  margin-right: 2em }
>
>pre.code .ln { color: grey; } /* line numbers */
>pre.code, code { background-color: #eeeeee }
>pre.code .comment, code .comment { color: #5C6576 }
>pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
>pre.code .literal.string, code .literal.string { color: #0C5404 }
>pre.code .name.builtin, code .name.builtin { color: #352B84 }
>pre.code .deleted, code .deleted { background-color: #DEB0A1}
>pre.code .inserted, code .inserted { background-color: #A3D289}
>
>span.classifier {
>  font-family: sans-serif ;
>  font-style: oblique }
>
>span.classifier-delimiter {
>  font-family: sans-serif ;
>  font-weight: bold }
>
>span.interpreted {
>  font-family: sans-serif }
>
>span.option {
>  white-space: nowrap }
>
>span.pre {
>  white-space: pre }
>
>span.problematic {
>  color: red }
>
>span.section-subtitle {
>  /* font-size relative to parent (h1..h6 element) */
>  font-size: 80% }
>
>table.citation {
>  border-left: solid 1px gray;
>  margin-left: 1px }
>
>table.docinfo {
>  margin: 2em 4em }
>
>table.docutils {
>  margin-top: 0.5em ;
>  margin-bottom: 0.5em }
>
>table.footnote {
>  border-left: solid 1px black;
>  margin-left: 1px }
>
>table.docutils td, table.docutils th,
>table.docinfo td, table.docinfo th {
>  padding-left: 0.5em ;
>  padding-right: 0.5em ;
>  vertical-align: top }
>
>table.docutils th.field-name, table.docinfo th.docinfo-name {
>  font-weight: bold ;
>  text-align: left ;
>  white-space: nowrap ;
>  padding-left: 0 }
>
>/* "booktabs" style (no vertical lines) */
>table.docutils.booktabs {
>  border: 0px;
>  border-top: 2px solid;
>  border-bottom: 2px solid;
>  border-collapse: collapse;
>}
>table.docutils.booktabs * {
>  border: 0px;
>}
>table.docutils.booktabs th {
>  border-bottom: thin solid;
>  text-align: left;
>}
>
>h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
>h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
>  font-size: 100% }
>
>ul.auto-toc {
>  list-style-type: none }
>
></style>
></head>
><body>
><div class="document" id="keystone-ldap-pool-overview">
><h1 class="title">Keystone LDAP Pool Overview</h1>
><table class="docinfo" frame="void" rules="none">
><col class="docinfo-name" />
><col class="docinfo-content" />
><tbody valign="top">
><tr><th class="docinfo-name">Author:</th>
><td>John Dennis &lt;<a class="reference external" href="mailto:jdennis&#64;redhat.com">jdennis&#64;redhat.com</a>&gt;</td></tr>
><tr><th class="docinfo-name">Version:</th>
><td>1.0</td></tr>
></tbody>
></table>
><div class="section" id="executive-summary">
><h1>Executive Summary</h1>
><p>There are many misconceptions concerning the LDAP pool functionality
>in Keystone and what advantages it offers. In general the LDAP pool
>does <em>not</em> increase performance via parallelism. Instead performance
>gains via the LDAP pool derive from the fact connections to the LDAP
>server are &quot;kept alive&quot; across multiple LDAP requests thus eliminating
>the overhead of setting up and tearing down the connection to the LDAP
>server. It is unlikely the LDAP pool size will grow beyond 1 (see
><a class="reference internal" href="#why-pool-size-of-1">why pool size of 1</a>)</p>
></div>
><div class="section" id="what-is-an-ldap-pool">
><span id="ldap-pool-definition"></span><h1>What is an LDAP pool?</h1>
><p>An LDAP pool is a collection of long lived <a class="footnote-reference" href="#id5" id="id1">[1]</a> LDAP connections
>available to the threads executing in a single Python interpreter
>process. Each LDAP connection in the pool is uniquely identified by
>the (user,password) 2-tuple used to bind the LDAP connection. All
>members of the LDAP pool connect to the same LDAP server identified by
>the Keystone LDAP <cite>url</cite> configuration parameter. <a class="footnote-reference" href="#id6" id="id2">[2]</a></p>
><div class="section" id="how-is-the-ldap-pool-managed">
><span id="pool-management"></span><h2>How is the LDAP pool managed?</h2>
><p>When a Python thread needs an LDAP connection the LDAP pool is locked
>and searched for the first <em>inactive</em> connection whose (user,password)
>2-tuple matches. Inactive pool members whose
><cite>pool_connection_lifetime</cite> have expired when the search is performed
>are ejected from the pool and hence from further consideration. If no
><em>inactive</em> connections with the requisite (user,password) 2-tuple is
>found then a new pool member is added. Independent of whether a
>candidate pool member was found in the pool or a new pool member was
>created the selected member is immediately marked as <em>active</em> and
>returned for use. If no candidate pool members can be located and the
>pool is at it's <cite>pool_size</cite> maximum then a <cite>MaxConnectionReachedError</cite>
>is raised.</p>
><p>After the connection pool member is returned it is used to perform a
><strong>synchronous</strong> LDAP request. During this time the Python thread
><strong>blocks</strong> until the result is returned. The LDAP pool member used to
>perform the synchronous LDAP request is then marked as <em>inactive</em> at
>the completion of the LDAP request thus becoming available to other
>threads in the same Python interpreter process. At this point any
>thread in the Python interpreter process can then use the already open
>connection belonging the newly returned <em>inactive</em> pool member to
>perform the next LDAP request in it's thread without having to perform
>a costly connect operation (e.g. keep-alive).</p>
></div>
></div>
><div class="section" id="httpd-servers-wsgi-daemons-python-interpreter-processes-and-threads">
><h1>HTTPD Servers, WSGI Daemons, Python Interpreter processes and threads</h1>
><p>In the discussion on <a class="reference internal" href="#pool-management">Pool Management</a> we learned that the LDAP pool
>is shared by threads in one single Python interpreter process. In a
>typical Keystone deployment there are many Python interpreter
>processes available to service Keystone requests hence there are many
>LDAP pools. This is contrast to the often erroneous assumption members
>of the LDAP pool are shared between processes thus forming one large
>aggregate.</p>
><p>Keystone is run inside a HTTPD server (e.g. Apache). Because Keystone
>is a Python Application it is serviced by HTTPD via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a>
>module which implements <a class="reference external" href="https://wsgi.readthedocs.io/en/latest/">WSGI</a> (Web Server Gateway Interface).</p>
><p>Each HTTPD server may fork multiple copies of itself to simultaneously
>service incoming HTTP requests. Within an HTTPD server there may be
>one or more Python interpreter processes running due to the
>configuration of <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a>. Typically <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> is run in <cite>daemon
>mode</cite> via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html">WSGIDaemonProcess</a> configuration
>parameter. The parent HTTPD server will fork a number of Python
>interpreter processes each of which may run one or more threads. This
>is controlled by this HTTPD configuration parameter:</p>
><pre class="literal-block">
>WSGIDaemonProcess processes=num threads=num
></pre>
><p>Exactly how many Python interpreter processes are spawned depends on a
>somewhat complicated set of relationships between HTTPD configuration
>directives and <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> configuration parameters. This is explained in
>detail in the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html">WSGI Process &amp; Thread Documentation</a>.</p>
><p>In addition to the multiple processes and threads servicing Keystone
>requests in a single HTTPD instance there may be multiple HTTPD
>instances if a load balancer is present in an High Availability (HA)
>deployment.</p>
><p>The important point to remember is in most Keystone deployments there
>will be <strong>many Python interpreter processes, each of which will have
>it's own LDAP pool.</strong> Requests to the Keystone server will be
>serviced in round-robin fashion across each of these Python interpreter
>processes.</p>
><p>Recalling that the LDAP pool is available for <strong>threads</strong> within a
>single Python interpreter process we note the following consequences:</p>
><ul class="simple" id="when-wsgidaemonprocess-threads-1">
><li>If the number of threads in a Python interpreter process is 1
>(e.g. <cite>WSGIDaemonProcess threads=1</cite>) the LDAP pool size will never
>exceed 1. This is because only one thread shares the pool. The
>pool size will be zero if it's never been utilized or if the
><cite>pool_connection_lifetime</cite> <a class="footnote-reference" href="#id5" id="id3">[1]</a> of the single pool member
>expires.</li>
></ul>
><ul class="simple" id="why-pool-size-of-1">
><li>For the size of the LDAP pool to grow to greater than one the
>following must occur: The number of threads must be greater than
>one. The number of available Python interpreter processes servicing
>Keystone requests must be saturated or some other condition holds
>such that a second thread within a Python interpreter process is run
>to service the request <strong>and</strong> each running thread in the Python
>interpreter process is <strong>simultaneously</strong> attempting to perform a
>LDAP request. <a class="footnote-reference" href="#id7" id="id4">[3]</a> The likelihood of satisfying all the conditions
>necessary to grow the pool is low. It would be exacerbated either by
>slow Keystone execution, heavy Keystone load, or slow responses from
>the LDAP server.</li>
></ul>
><p>A single Keystone request typically induces multiple LDAP requests to
>satisfy the Keystone request. The Keystone request will execute within
>only one Python thread. Thus the multiple LDAP operations need to
>satisfy the Keystone request will execute <em>sequentially</em> within the
>thread. This is further enforced by the fact the LDAP pool demands
>each LDAP request be <strong>synchronous</strong>. <em>Hence there is no parallel
>overlap when executing LDAP operations for a given Keystone request.</em></p>
><div class="admonition note">
><p class="first admonition-title">Note</p>
><p class="last">Most Keystone deployments set <cite>WSGIDaemonProcess threads=1</cite>
>enforcing a one-to-one relationship between Python
>interpreter processes and threads with the consequence the
>LDAP pool size will never exceed 1. (see <a class="reference internal" href="#when-wsgidaemonprocess-threads-1">When
>WSGIDaemonProcess threads=1</a>)</p>
></div>
></div>
><div class="section" id="what-is-the-advantage-of-using-ldap-pools">
><h1>What is the advantage of using LDAP pools?</h1>
><p>The primary advantage of using the Keystone LDAP pool is the pool's
>ability to keep a continuous LDAP connection open for use by the
>threads in a single Python interpreter process. Because a Keystone
>request is serviced by one thread and because during the servicing of
>that request multiple LDAP operations will be executed in a sequential
>synchronous fashion access to the LDAP server will be much faster
>because the same connection will be left open and utilized thus
>avoiding the overhead of connection setup and tear down.</p>
></div>
><div class="section" id="footnotes">
><h1>Footnotes</h1>
><table class="docutils footnote" frame="void" id="id5" rules="none">
><colgroup><col class="label" /><col /></colgroup>
><tbody valign="top">
><tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#id1">1</a>, <a class="fn-backref" href="#id3">2</a>)</em> <p>The duration of a connection to a LDAP server in the LDAP pool
>is controlled by the <cite>pool_connection_lifetime</cite> configuration
>variable.</p>
><blockquote class="last">
><cite>pool_connection_lifetime</cite>: The maximum connection lifetime
>to the LDAP server in seconds. When this lifetime is
>exceeded, the connection will be unbound and removed from
>the connection pool. This option has no effect unless
><cite>[ldap] use_pool</cite> is also enabled.</blockquote>
></td></tr>
></tbody>
></table>
><table class="docutils footnote" frame="void" id="id6" rules="none">
><colgroup><col class="label" /><col /></colgroup>
><tbody valign="top">
><tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><p class="first">The LDAP <cite>url</cite> configuration parameter permits you to specify a
>list of LDAP server URL's. This feature has a confusing
>interaction with pools and automatic reconnect
>behavior. Internally Keystone treats the LDAP <cite>url</cite> parameter
>as a simple string, it does <em>not</em> break it apart into independent
>LDAP URL's where each LDAP URL is managed independently. The
>LDAP URL comma separated list is passed directly into the
>OpenLDAP client C library which then breaks the list into
>independent LDAP URL's. When a connection to an LDAP server is
>attempted the OpenLDAP client C library iterates over the
>members of the list attempting to connect to each in turn. Thus
>the Keystone code does not know which LDAP server you're
>actually connected to if you specify multiple servers in the
><cite>url</cite> configuration parameter.</p>
><blockquote class="last">
><cite>url</cite>: URL(s) for connecting to the LDAP server. Multiple
>LDAP URLs may be specified as a comma separated string. The
>first URL to successfully bind is used for the connection.</blockquote>
></td></tr>
></tbody>
></table>
><table class="docutils footnote" frame="void" id="id7" rules="none">
><colgroup><col class="label" /><col /></colgroup>
><tbody valign="top">
><tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>The pool size would also grow if distinct (user,password)
>2-tuples are used to bind the connection because the pool keeps
>connections segregated by their bind parameters. However the
>current Keystone LDAP configuration prevents multiple distinct
>bind credentials hence the pool can only grow when threads in a
>single Python interpreter process <strong>simultaneously</strong> compete
>for a connection.</td></tr>
></tbody>
></table>
></div>
></div>
></body>
></html>

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" />
<title>Keystone LDAP Pool Overview</title>
<meta name="author" content="John Dennis &lt;jdennis&#64;redhat.com&gt;" />
<style type="text/css">

/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $
:Copyright: This stylesheet has been placed in the public domain.

Default cascading style sheet for the HTML output of Docutils.

See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/

/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
  border: 0 }

table.borderless td, table.borderless th {
  /* Override padding for "table.docutils td" with "! important".
     The right padding separates the table cells. */
  padding: 0 0.5em 0 0 ! important }

.first {
  /* Override more specific margin styles with "! important". */
  margin-top: 0 ! important }

.last, .with-subtitle {
  margin-bottom: 0 ! important }

.hidden {
  display: none }

.subscript {
  vertical-align: sub;
  font-size: smaller }

.superscript {
  vertical-align: super;
  font-size: smaller }

a.toc-backref {
  text-decoration: none ;
  color: black }

blockquote.epigraph {
  margin: 2em 5em ; }

dl.docutils dd {
  margin-bottom: 0.5em }

object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
  overflow: hidden;
}

/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
  font-weight: bold }
*/

div.abstract {
  margin: 2em 5em }

div.abstract p.topic-title {
  font-weight: bold ;
  text-align: center }

div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
  margin: 2em ;
  border: medium outset ;
  padding: 1em }

div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
  font-weight: bold ;
  font-family: sans-serif }

div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
  color: red ;
  font-weight: bold ;
  font-family: sans-serif }

/* Uncomment (and remove this text!) to get reduced vertical space in
   compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
  margin-bottom: 0.5em }

div.compound .compound-last, div.compound .compound-middle {
  margin-top: 0.5em }
*/

div.dedication {
  margin: 2em 5em ;
  text-align: center ;
  font-style: italic }

div.dedication p.topic-title {
  font-weight: bold ;
  font-style: normal }

div.figure {
  margin-left: 2em ;
  margin-right: 2em }

div.footer, div.header {
  clear: both;
  font-size: smaller }

div.line-block {
  display: block ;
  margin-top: 1em ;
  margin-bottom: 1em }

div.line-block div.line-block {
  margin-top: 0 ;
  margin-bottom: 0 ;
  margin-left: 1.5em }

div.sidebar {
  margin: 0 0 0.5em 1em ;
  border: medium outset ;
  padding: 1em ;
  background-color: #ffffee ;
  width: 40% ;
  float: right ;
  clear: right }

div.sidebar p.rubric {
  font-family: sans-serif ;
  font-size: medium }

div.system-messages {
  margin: 5em }

div.system-messages h1 {
  color: red }

div.system-message {
  border: medium outset ;
  padding: 1em }

div.system-message p.system-message-title {
  color: red ;
  font-weight: bold }

div.topic {
  margin: 2em }

h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
  margin-top: 0.4em }

h1.title {
  text-align: center }

h2.subtitle {
  text-align: center }

hr.docutils {
  width: 75% }

img.align-left, .figure.align-left, object.align-left, table.align-left {
  clear: left ;
  float: left ;
  margin-right: 1em }

img.align-right, .figure.align-right, object.align-right, table.align-right {
  clear: right ;
  float: right ;
  margin-left: 1em }

img.align-center, .figure.align-center, object.align-center {
  display: block;
  margin-left: auto;
  margin-right: auto;
}

table.align-center {
  margin-left: auto;
  margin-right: auto;
}

.align-left {
  text-align: left }

.align-center {
  clear: both ;
  text-align: center }

.align-right {
  text-align: right }

/* reset inner alignment in figures */
div.align-right {
  text-align: inherit }

/* div.align-center * { */
/*   text-align: left } */

.align-top    {
  vertical-align: top }

.align-middle {
  vertical-align: middle }

.align-bottom {
  vertical-align: bottom }

ol.simple, ul.simple {
  margin-bottom: 1em }

ol.arabic {
  list-style: decimal }

ol.loweralpha {
  list-style: lower-alpha }

ol.upperalpha {
  list-style: upper-alpha }

ol.lowerroman {
  list-style: lower-roman }

ol.upperroman {
  list-style: upper-roman }

p.attribution {
  text-align: right ;
  margin-left: 50% }

p.caption {
  font-style: italic }

p.credits {
  font-style: italic ;
  font-size: smaller }

p.label {
  white-space: nowrap }

p.rubric {
  font-weight: bold ;
  font-size: larger ;
  color: maroon ;
  text-align: center }

p.sidebar-title {
  font-family: sans-serif ;
  font-weight: bold ;
  font-size: larger }

p.sidebar-subtitle {
  font-family: sans-serif ;
  font-weight: bold }

p.topic-title {
  font-weight: bold }

pre.address {
  margin-bottom: 0 ;
  margin-top: 0 ;
  font: inherit }

pre.literal-block, pre.doctest-block, pre.math, pre.code {
  margin-left: 2em ;
  margin-right: 2em }

pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}

span.classifier {
  font-family: sans-serif ;
  font-style: oblique }

span.classifier-delimiter {
  font-family: sans-serif ;
  font-weight: bold }

span.interpreted {
  font-family: sans-serif }

span.option {
  white-space: nowrap }

span.pre {
  white-space: pre }

span.problematic {
  color: red }

span.section-subtitle {
  /* font-size relative to parent (h1..h6 element) */
  font-size: 80% }

table.citation {
  border-left: solid 1px gray;
  margin-left: 1px }

table.docinfo {
  margin: 2em 4em }

table.docutils {
  margin-top: 0.5em ;
  margin-bottom: 0.5em }

table.footnote {
  border-left: solid 1px black;
  margin-left: 1px }

table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
  padding-left: 0.5em ;
  padding-right: 0.5em ;
  vertical-align: top }

table.docutils th.field-name, table.docinfo th.docinfo-name {
  font-weight: bold ;
  text-align: left ;
  white-space: nowrap ;
  padding-left: 0 }

/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
  border: 0px;
  border-top: 2px solid;
  border-bottom: 2px solid;
  border-collapse: collapse;
}
table.docutils.booktabs * {
  border: 0px;
}
table.docutils.booktabs th {
  border-bottom: thin solid;
  text-align: left;
}

h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
  font-size: 100% }

ul.auto-toc {
  list-style-type: none }

</style>
</head>
<body>
<div class="document" id="keystone-ldap-pool-overview">
<h1 class="title">Keystone LDAP Pool Overview</h1>
<table class="docinfo" frame="void" rules="none">
<col class="docinfo-name" />
<col class="docinfo-content" />
<tbody valign="top">
<tr><th class="docinfo-name">Author:</th>
<td>John Dennis &lt;<a class="reference external" href="mailto:jdennis&#64;redhat.com">jdennis&#64;redhat.com</a>&gt;</td></tr>
<tr><th class="docinfo-name">Version:</th>
<td>1.0</td></tr>
</tbody>
</table>
<div class="section" id="executive-summary">
<h1>Executive Summary</h1>
<p>There are many misconceptions concerning the LDAP pool functionality
in Keystone and what advantages it offers. In general the LDAP pool
does <em>not</em> increase performance via parallelism. Instead performance
gains via the LDAP pool derive from the fact connections to the LDAP
server are &quot;kept alive&quot; across multiple LDAP requests thus eliminating
the overhead of setting up and tearing down the connection to the LDAP
server. It is unlikely the LDAP pool size will grow beyond 1 (see
<a class="reference internal" href="#why-pool-size-of-1">why pool size of 1</a>)</p>
</div>
<div class="section" id="what-is-an-ldap-pool">
<span id="ldap-pool-definition"></span><h1>What is an LDAP pool?</h1>
<p>An LDAP pool is a collection of long lived <a class="footnote-reference" href="#id5" id="id1">[1]</a> LDAP connections
available to the threads executing in a single Python interpreter
process. Each LDAP connection in the pool is uniquely identified by
the (user,password) 2-tuple used to bind the LDAP connection. All
members of the LDAP pool connect to the same LDAP server identified by
the Keystone LDAP <cite>url</cite> configuration parameter. <a class="footnote-reference" href="#id6" id="id2">[2]</a></p>
<div class="section" id="how-is-the-ldap-pool-managed">
<span id="pool-management"></span><h2>How is the LDAP pool managed?</h2>
<p>When a Python thread needs an LDAP connection the LDAP pool is locked
and searched for the first <em>inactive</em> connection whose (user,password)
2-tuple matches. Inactive pool members whose
<cite>pool_connection_lifetime</cite> have expired when the search is performed
are ejected from the pool and hence from further consideration. If no
<em>inactive</em> connections with the requisite (user,password) 2-tuple is
found then a new pool member is added. Independent of whether a
candidate pool member was found in the pool or a new pool member was
created the selected member is immediately marked as <em>active</em> and
returned for use. If no candidate pool members can be located and the
pool is at it's <cite>pool_size</cite> maximum then a <cite>MaxConnectionReachedError</cite>
is raised.</p>
<p>After the connection pool member is returned it is used to perform a
<strong>synchronous</strong> LDAP request. During this time the Python thread
<strong>blocks</strong> until the result is returned. The LDAP pool member used to
perform the synchronous LDAP request is then marked as <em>inactive</em> at
the completion of the LDAP request thus becoming available to other
threads in the same Python interpreter process. At this point any
thread in the Python interpreter process can then use the already open
connection belonging the newly returned <em>inactive</em> pool member to
perform the next LDAP request in it's thread without having to perform
a costly connect operation (e.g. keep-alive).</p>
</div>
</div>
<div class="section" id="httpd-servers-wsgi-daemons-python-interpreter-processes-and-threads">
<h1>HTTPD Servers, WSGI Daemons, Python Interpreter processes and threads</h1>
<p>In the discussion on <a class="reference internal" href="#pool-management">Pool Management</a> we learned that the LDAP pool
is shared by threads in one single Python interpreter process. In a
typical Keystone deployment there are many Python interpreter
processes available to service Keystone requests hence there are many
LDAP pools. This is contrast to the often erroneous assumption members
of the LDAP pool are shared between processes thus forming one large
aggregate.</p>
<p>Keystone is run inside a HTTPD server (e.g. Apache). Because Keystone
is a Python Application it is serviced by HTTPD via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a>
module which implements <a class="reference external" href="https://wsgi.readthedocs.io/en/latest/">WSGI</a> (Web Server Gateway Interface).</p>
<p>Each HTTPD server may fork multiple copies of itself to simultaneously
service incoming HTTP requests. Within an HTTPD server there may be
one or more Python interpreter processes running due to the
configuration of <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a>. Typically <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> is run in <cite>daemon
mode</cite> via the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html">WSGIDaemonProcess</a> configuration
parameter. The parent HTTPD server will fork a number of Python
interpreter processes each of which may run one or more threads. This
is controlled by this HTTPD configuration parameter:</p>
<pre class="literal-block">
WSGIDaemonProcess processes=num threads=num
</pre>
<p>Exactly how many Python interpreter processes are spawned depends on a
somewhat complicated set of relationships between HTTPD configuration
directives and <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/index.html">mod_wsgi</a> configuration parameters. This is explained in
detail in the <a class="reference external" href="https://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html">WSGI Process &amp; Thread Documentation</a>.</p>
<p>In addition to the multiple processes and threads servicing Keystone
requests in a single HTTPD instance there may be multiple HTTPD
instances if a load balancer is present in an High Availability (HA)
deployment.</p>
<p>The important point to remember is in most Keystone deployments there
will be <strong>many Python interpreter processes, each of which will have
it's own LDAP pool.</strong> Requests to the Keystone server will be
serviced in round-robin fashion across each of these Python interpreter
processes.</p>
<p>Recalling that the LDAP pool is available for <strong>threads</strong> within a
single Python interpreter process we note the following consequences:</p>
<ul class="simple" id="when-wsgidaemonprocess-threads-1">
<li>If the number of threads in a Python interpreter process is 1
(e.g. <cite>WSGIDaemonProcess threads=1</cite>) the LDAP pool size will never
exceed 1. This is because only one thread shares the pool. The
pool size will be zero if it's never been utilized or if the
<cite>pool_connection_lifetime</cite> <a class="footnote-reference" href="#id5" id="id3">[1]</a> of the single pool member
expires.</li>
</ul>
<ul class="simple" id="why-pool-size-of-1">
<li>For the size of the LDAP pool to grow to greater than one the
following must occur: The number of threads must be greater than
one. The number of available Python interpreter processes servicing
Keystone requests must be saturated or some other condition holds
such that a second thread within a Python interpreter process is run
to service the request <strong>and</strong> each running thread in the Python
interpreter process is <strong>simultaneously</strong> attempting to perform a
LDAP request. <a class="footnote-reference" href="#id7" id="id4">[3]</a> The likelihood of satisfying all the conditions
necessary to grow the pool is low. It would be exacerbated either by
slow Keystone execution, heavy Keystone load, or slow responses from
the LDAP server.</li>
</ul>
<p>A single Keystone request typically induces multiple LDAP requests to
satisfy the Keystone request. The Keystone request will execute within
only one Python thread. Thus the multiple LDAP operations need to
satisfy the Keystone request will execute <em>sequentially</em> within the
thread. This is further enforced by the fact the LDAP pool demands
each LDAP request be <strong>synchronous</strong>. <em>Hence there is no parallel
overlap when executing LDAP operations for a given Keystone request.</em></p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Most Keystone deployments set <cite>WSGIDaemonProcess threads=1</cite>
enforcing a one-to-one relationship between Python
interpreter processes and threads with the consequence the
LDAP pool size will never exceed 1. (see <a class="reference internal" href="#when-wsgidaemonprocess-threads-1">When
WSGIDaemonProcess threads=1</a>)</p>
</div>
</div>
<div class="section" id="what-is-the-advantage-of-using-ldap-pools">
<h1>What is the advantage of using LDAP pools?</h1>
<p>The primary advantage of using the Keystone LDAP pool is the pool's
ability to keep a continuous LDAP connection open for use by the
threads in a single Python interpreter process. Because a Keystone
request is serviced by one thread and because during the servicing of
that request multiple LDAP operations will be executed in a sequential
synchronous fashion access to the LDAP server will be much faster
because the same connection will be left open and utilized thus
avoiding the overhead of connection setup and tear down.</p>
</div>
<div class="section" id="footnotes">
<h1>Footnotes</h1>
<table class="docutils footnote" frame="void" id="id5" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#id1">1</a>, <a class="fn-backref" href="#id3">2</a>)</em> <p>The duration of a connection to a LDAP server in the LDAP pool
is controlled by the <cite>pool_connection_lifetime</cite> configuration
variable.</p>
<blockquote class="last">
<cite>pool_connection_lifetime</cite>: The maximum connection lifetime
to the LDAP server in seconds. When this lifetime is
exceeded, the connection will be unbound and removed from
the connection pool. This option has no effect unless
<cite>[ldap] use_pool</cite> is also enabled.</blockquote>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id6" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><p class="first">The LDAP <cite>url</cite> configuration parameter permits you to specify a
list of LDAP server URL's. This feature has a confusing
interaction with pools and automatic reconnect
behavior. Internally Keystone treats the LDAP <cite>url</cite> parameter
as a simple string, it does <em>not</em> break it apart into independent
LDAP URL's where each LDAP URL is managed independently. The
LDAP URL comma separated list is passed directly into the
OpenLDAP client C library which then breaks the list into
independent LDAP URL's. When a connection to an LDAP server is
attempted the OpenLDAP client C library iterates over the
members of the list attempting to connect to each in turn. Thus
the Keystone code does not know which LDAP server you're
actually connected to if you specify multiple servers in the
<cite>url</cite> configuration parameter.</p>
<blockquote class="last">
<cite>url</cite>: URL(s) for connecting to the LDAP server. Multiple
LDAP URLs may be specified as a comma separated string. The
first URL to successfully bind is used for the connection.</blockquote>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id7" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>The pool size would also grow if distinct (user,password)
2-tuples are used to bind the connection because the pool keeps
connections segregated by their bind parameters. However the
current Keystone LDAP configuration prevents multiple distinct
bind credentials hence the pool can only grow when threads in a
single Python interpreter process <strong>simultaneously</strong> compete
for a connection.</td></tr>
</tbody>
</table>
</div>
</div>
</body>
</html>

Actions: View

Attachments on bug 1626782: 1491082