Description of problem: OSP8 undercloud deploys Keystone in Apache and tunes the number of workers (processes) to 1 and threads to cpu_count. During scaling up of a deployment we witness timeouts from keystone: <class 'keystoneclient.exceptions.RequestTimeout'> (HTTP 500) Previous benchmarking results from rally have shown that a single keystone process becomes bound to a single cpu and scaling the number of threads does not improve the response timing. In order to scale up we had to manually tune the number of keystone workers to allow keystone more hardware resources to due the job. In our case we tuned (main&admin) it to: WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=16 threads=1 user=keystone Version-Release number of selected component (if applicable): python-tripleoclient-0.3.1-1.el7ost.noarch How reproducible: Always (100%) Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: I have attempted to patch this issue in the keystone puppet modules by setting workers == logical cpu core count that is cap-ed at 32 workers to prevent keystone from consuming all db connections. https://review.openstack.org/#/c/297342/
puppet-keystone already provides the interface to change threads/workers parameters, so this is a bug for undercloud.
The values for keystone start with the yaml file here http://git.openstack.org/cgit/openstack/instack-undercloud/tree/elements/puppet-stack-config/puppet-stack-config.yaml.template#n173 This ends up on the undercloud reified as /etc/puppet/hieradata/puppet-stack-config.yaml Quickstart adds a follow on file that overrides values /etc/puppet/hieradata/quickstart-hieradata-overrides.yaml Please put any overrides you need into a comparable file. Perhaps try using 4-8 workers with 1 thread per worker, as Python/WSGI does not make effective use of threading. The parameters should be something like keystone::wsgi::apache::workers Does this work for you?
This logic here is a safer approach: https://review.openstack.org/#/c/375146/ And is what the puppet-keystone and other WSGI modules are going to use.
(In reply to Adam Young from comment #4) > This logic here is a safer approach: > > https://review.openstack.org/#/c/375146/ > > And is what the puppet-keystone and other WSGI modules are going to use. Are there changes proposed already for puppet-keystone to make use of this? If so, what versions are we going to pull those changes into? Does this bug need to be filed against a different component (like o-p-m)?
https://review.openstack.org/#/c/297342/ Has been updated to use the new fact.
This has merged upstream in the master branch of puppet-keystone.
verified for openstack-keystone-11.0.0-0.20170127043446.cefbc3c.el7ost.noarch - conf files in httpd now have a max between '(<# processors> / 4)' and '2' with a cap of 8. In the testing environment, each controller has 4 processors: # cat 10-keystone_wsgi_main.conf | grep WSGIDaemonProcess WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=2 threads=1 user=keystone # cat 10-keystone_wsgi_admin.conf | grep WSGIDaemonProcess WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=2 threads=1 user=keystone
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245