Bug 1330980

Summary: Undercloud deployed with 1 keystone worker and cpu_count for threads
Product: Red Hat OpenStack Reporter: Alex Krzos <akrzos>
Component: puppet-keystoneAssignee: RHOS Maint <rhos-maint>
Status: CLOSED ERRATA QA Contact: nlevinki <nlevinki>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.0 (Liberty)CC: akrzos, ayoung, hbrock, jcoufal, jjoyce, jschluet, jslagle, kbasil, mburns, nkinder, rduartes, rhel-osp-director-maint, sclewis, slinaber, srevivo, tvignaud
Target Milestone: Upstream M3Keywords: Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-keystone-9.4.0-0.20161102012005.dfd9690.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1435472 (view as bug list) Environment:
Last Closed: 2017-05-17 19:29:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1435472    

Description Alex Krzos 2016-04-27 11:49:51 UTC
Description of problem:
OSP8 undercloud deploys Keystone in Apache and tunes the number of workers (processes) to 1 and threads to cpu_count.  During scaling up of a deployment we witness timeouts from keystone:
 <class 'keystoneclient.exceptions.RequestTimeout'> (HTTP 500)

Previous benchmarking results from rally have shown that a single keystone process becomes bound to a single cpu and scaling the number of threads does not improve the response timing.

In order to scale up we had to manually tune the number of keystone workers to allow keystone more hardware resources to due the job.  In our case we tuned (main&admin) it to:

WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=16 threads=1 user=keystone

Version-Release number of selected component (if applicable):
python-tripleoclient-0.3.1-1.el7ost.noarch

How reproducible:
Always (100%)

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
I have attempted to patch this issue in the keystone puppet modules by setting workers == logical cpu core count that is cap-ed at 32 workers to prevent keystone from consuming all db connections.

https://review.openstack.org/#/c/297342/

Comment 2 Emilien Macchi 2016-05-03 15:33:26 UTC
puppet-keystone already provides the interface to change threads/workers parameters, so this is a bug for undercloud.

Comment 3 Adam Young 2016-09-22 17:37:35 UTC
The values for keystone start with the yaml file here

http://git.openstack.org/cgit/openstack/instack-undercloud/tree/elements/puppet-stack-config/puppet-stack-config.yaml.template#n173

This ends up on the undercloud reified as

/etc/puppet/hieradata/puppet-stack-config.yaml

Quickstart adds a follow on file that overrides values

/etc/puppet/hieradata/quickstart-hieradata-overrides.yaml


Please put any overrides you need into a comparable file.

Perhaps try using 4-8 workers with 1 thread per worker, as Python/WSGI does not make effective use of threading. 

The parameters should be something like 

keystone::wsgi::apache::workers

Does this work for you?

Comment 4 Adam Young 2016-09-23 20:07:50 UTC
This logic here is a safer approach:

https://review.openstack.org/#/c/375146/

And is what the puppet-keystone and other WSGI modules are going to use.

Comment 5 Nathan Kinder 2016-09-26 21:05:36 UTC
(In reply to Adam Young from comment #4)
> This logic here is a safer approach:
> 
> https://review.openstack.org/#/c/375146/
> 
> And is what the puppet-keystone and other WSGI modules are going to use.

Are there changes proposed already for puppet-keystone to make use of this?  If so, what versions are we going to pull those changes into?

Does this bug need to be filed against a different component (like o-p-m)?

Comment 6 Adam Young 2016-09-30 18:54:39 UTC
https://review.openstack.org/#/c/297342/  Has been updated to use the new fact.

Comment 8 Nathan Kinder 2016-11-07 15:40:03 UTC
This has merged upstream in the master branch of puppet-keystone.

Comment 12 Rodrigo Duarte 2017-02-13 21:52:40 UTC
verified for openstack-keystone-11.0.0-0.20170127043446.cefbc3c.el7ost.noarch

- conf files in httpd now have a max between '(<# processors> / 4)' and '2' with a cap of 8. In the testing environment, each controller has 4 processors:

# cat 10-keystone_wsgi_main.conf | grep WSGIDaemonProcess
  WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=2 threads=1 user=keystone

# cat 10-keystone_wsgi_admin.conf | grep WSGIDaemonProcess
  WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=2 threads=1 user=keystone

Comment 13 errata-xmlrpc 2017-05-17 19:29:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245