Bug 1330980 - Undercloud deployed with 1 keystone worker and cpu_count for threads
Summary: Undercloud deployed with 1 keystone worker and cpu_count for threads
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-keystone
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
Target Milestone: Upstream M3
: 11.0 (Ocata)
Assignee: RHOS Maint
QA Contact: nlevinki
Depends On:
Blocks: 1435472
TreeView+ depends on / blocked
Reported: 2016-04-27 11:49 UTC by Alex Krzos
Modified: 2017-05-17 19:29 UTC (History)
16 users (show)

Fixed In Version: puppet-keystone-9.4.0-0.20161102012005.dfd9690.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1435472 (view as bug list)
Last Closed: 2017-05-17 19:29:20 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
OpenStack gerrit 382857 0 None MERGED Change worker defaults to ::os_workers 2020-11-10 16:02:10 UTC
Red Hat Product Errata RHEA-2017:1245 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Alex Krzos 2016-04-27 11:49:51 UTC
Description of problem:
OSP8 undercloud deploys Keystone in Apache and tunes the number of workers (processes) to 1 and threads to cpu_count.  During scaling up of a deployment we witness timeouts from keystone:
 <class 'keystoneclient.exceptions.RequestTimeout'> (HTTP 500)

Previous benchmarking results from rally have shown that a single keystone process becomes bound to a single cpu and scaling the number of threads does not improve the response timing.

In order to scale up we had to manually tune the number of keystone workers to allow keystone more hardware resources to due the job.  In our case we tuned (main&admin) it to:

WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=16 threads=1 user=keystone

Version-Release number of selected component (if applicable):

How reproducible:
Always (100%)

Steps to Reproduce:

Actual results:

Expected results:

Additional info:
I have attempted to patch this issue in the keystone puppet modules by setting workers == logical cpu core count that is cap-ed at 32 workers to prevent keystone from consuming all db connections.


Comment 2 Emilien Macchi 2016-05-03 15:33:26 UTC
puppet-keystone already provides the interface to change threads/workers parameters, so this is a bug for undercloud.

Comment 3 Adam Young 2016-09-22 17:37:35 UTC
The values for keystone start with the yaml file here


This ends up on the undercloud reified as


Quickstart adds a follow on file that overrides values


Please put any overrides you need into a comparable file.

Perhaps try using 4-8 workers with 1 thread per worker, as Python/WSGI does not make effective use of threading. 

The parameters should be something like 


Does this work for you?

Comment 4 Adam Young 2016-09-23 20:07:50 UTC
This logic here is a safer approach:


And is what the puppet-keystone and other WSGI modules are going to use.

Comment 5 Nathan Kinder 2016-09-26 21:05:36 UTC
(In reply to Adam Young from comment #4)
> This logic here is a safer approach:
> https://review.openstack.org/#/c/375146/
> And is what the puppet-keystone and other WSGI modules are going to use.

Are there changes proposed already for puppet-keystone to make use of this?  If so, what versions are we going to pull those changes into?

Does this bug need to be filed against a different component (like o-p-m)?

Comment 6 Adam Young 2016-09-30 18:54:39 UTC
https://review.openstack.org/#/c/297342/  Has been updated to use the new fact.

Comment 8 Nathan Kinder 2016-11-07 15:40:03 UTC
This has merged upstream in the master branch of puppet-keystone.

Comment 12 Rodrigo Duarte 2017-02-13 21:52:40 UTC
verified for openstack-keystone-11.0.0-0.20170127043446.cefbc3c.el7ost.noarch

- conf files in httpd now have a max between '(<# processors> / 4)' and '2' with a cap of 8. In the testing environment, each controller has 4 processors:

# cat 10-keystone_wsgi_main.conf | grep WSGIDaemonProcess
  WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=2 threads=1 user=keystone

# cat 10-keystone_wsgi_admin.conf | grep WSGIDaemonProcess
  WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=2 threads=1 user=keystone

Comment 13 errata-xmlrpc 2017-05-17 19:29:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.