Red Hat Bugzilla – Bug 1330980
Undercloud deployed with 1 keystone worker and cpu_count for threads
Last modified: 2017-05-17 15:29:20 EDT
Description of problem:
OSP8 undercloud deploys Keystone in Apache and tunes the number of workers (processes) to 1 and threads to cpu_count. During scaling up of a deployment we witness timeouts from keystone:
<class 'keystoneclient.exceptions.RequestTimeout'> (HTTP 500)
Previous benchmarking results from rally have shown that a single keystone process becomes bound to a single cpu and scaling the number of threads does not improve the response timing.
In order to scale up we had to manually tune the number of keystone workers to allow keystone more hardware resources to due the job. In our case we tuned (main&admin) it to:
WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=16 threads=1 user=keystone
Version-Release number of selected component (if applicable):
Steps to Reproduce:
I have attempted to patch this issue in the keystone puppet modules by setting workers == logical cpu core count that is cap-ed at 32 workers to prevent keystone from consuming all db connections.
puppet-keystone already provides the interface to change threads/workers parameters, so this is a bug for undercloud.
The values for keystone start with the yaml file here
This ends up on the undercloud reified as
Quickstart adds a follow on file that overrides values
Please put any overrides you need into a comparable file.
Perhaps try using 4-8 workers with 1 thread per worker, as Python/WSGI does not make effective use of threading.
The parameters should be something like
Does this work for you?
This logic here is a safer approach:
And is what the puppet-keystone and other WSGI modules are going to use.
(In reply to Adam Young from comment #4)
> This logic here is a safer approach:
> And is what the puppet-keystone and other WSGI modules are going to use.
Are there changes proposed already for puppet-keystone to make use of this? If so, what versions are we going to pull those changes into?
Does this bug need to be filed against a different component (like o-p-m)?
https://review.openstack.org/#/c/297342/ Has been updated to use the new fact.
This has merged upstream in the master branch of puppet-keystone.
verified for openstack-keystone-11.0.0-0.20170127043446.cefbc3c.el7ost.noarch
- conf files in httpd now have a max between '(<# processors> / 4)' and '2' with a cap of 8. In the testing environment, each controller has 4 processors:
# cat 10-keystone_wsgi_main.conf | grep WSGIDaemonProcess
WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=2 threads=1 user=keystone
# cat 10-keystone_wsgi_admin.conf | grep WSGIDaemonProcess
WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=2 threads=1 user=keystone
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.