Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1330980 - Undercloud deployed with 1 keystone worker and cpu_count for threads
Undercloud deployed with 1 keystone worker and cpu_count for threads
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-keystone (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
medium Severity medium
: Upstream M3
: 11.0 (Ocata)
Assigned To: RHOS Maint
nlevinki
: Triaged
Depends On:
Blocks: 1435472
  Show dependency treegraph
 
Reported: 2016-04-27 07:49 EDT by Alex Krzos
Modified: 2017-05-17 15:29 EDT (History)
16 users (show)

See Also:
Fixed In Version: puppet-keystone-9.4.0-0.20161102012005.dfd9690.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1435472 (view as bug list)
Environment:
Last Closed: 2017-05-17 15:29:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 382857 None None None 2016-11-07 10:40 EST
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 19:01:50 EDT

  None (edit)
Description Alex Krzos 2016-04-27 07:49:51 EDT
Description of problem:
OSP8 undercloud deploys Keystone in Apache and tunes the number of workers (processes) to 1 and threads to cpu_count.  During scaling up of a deployment we witness timeouts from keystone:
 <class 'keystoneclient.exceptions.RequestTimeout'> (HTTP 500)

Previous benchmarking results from rally have shown that a single keystone process becomes bound to a single cpu and scaling the number of threads does not improve the response timing.

In order to scale up we had to manually tune the number of keystone workers to allow keystone more hardware resources to due the job.  In our case we tuned (main&admin) it to:

WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=16 threads=1 user=keystone

Version-Release number of selected component (if applicable):
python-tripleoclient-0.3.1-1.el7ost.noarch

How reproducible:
Always (100%)

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
I have attempted to patch this issue in the keystone puppet modules by setting workers == logical cpu core count that is cap-ed at 32 workers to prevent keystone from consuming all db connections.

https://review.openstack.org/#/c/297342/
Comment 2 Emilien Macchi 2016-05-03 11:33:26 EDT
puppet-keystone already provides the interface to change threads/workers parameters, so this is a bug for undercloud.
Comment 3 Adam Young 2016-09-22 13:37:35 EDT
The values for keystone start with the yaml file here

http://git.openstack.org/cgit/openstack/instack-undercloud/tree/elements/puppet-stack-config/puppet-stack-config.yaml.template#n173

This ends up on the undercloud reified as

/etc/puppet/hieradata/puppet-stack-config.yaml

Quickstart adds a follow on file that overrides values

/etc/puppet/hieradata/quickstart-hieradata-overrides.yaml


Please put any overrides you need into a comparable file.

Perhaps try using 4-8 workers with 1 thread per worker, as Python/WSGI does not make effective use of threading. 

The parameters should be something like 

keystone::wsgi::apache::workers

Does this work for you?
Comment 4 Adam Young 2016-09-23 16:07:50 EDT
This logic here is a safer approach:

https://review.openstack.org/#/c/375146/

And is what the puppet-keystone and other WSGI modules are going to use.
Comment 5 Nathan Kinder 2016-09-26 17:05:36 EDT
(In reply to Adam Young from comment #4)
> This logic here is a safer approach:
> 
> https://review.openstack.org/#/c/375146/
> 
> And is what the puppet-keystone and other WSGI modules are going to use.

Are there changes proposed already for puppet-keystone to make use of this?  If so, what versions are we going to pull those changes into?

Does this bug need to be filed against a different component (like o-p-m)?
Comment 6 Adam Young 2016-09-30 14:54:39 EDT
https://review.openstack.org/#/c/297342/  Has been updated to use the new fact.
Comment 8 Nathan Kinder 2016-11-07 10:40:03 EST
This has merged upstream in the master branch of puppet-keystone.
Comment 12 Rodrigo Duarte 2017-02-13 16:52:40 EST
verified for openstack-keystone-11.0.0-0.20170127043446.cefbc3c.el7ost.noarch

- conf files in httpd now have a max between '(<# processors> / 4)' and '2' with a cap of 8. In the testing environment, each controller has 4 processors:

# cat 10-keystone_wsgi_main.conf | grep WSGIDaemonProcess
  WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=2 threads=1 user=keystone

# cat 10-keystone_wsgi_admin.conf | grep WSGIDaemonProcess
  WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=2 threads=1 user=keystone
Comment 13 errata-xmlrpc 2017-05-17 15:29:20 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245

Note You need to log in before you can comment on or make changes to this bug.