Bug 1246138 - Keystone runs only single-threaded in OSP5
Summary: Keystone runs only single-threaded in OSP5
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-keystone
Version: 5.0 (RHEL 6)
Hardware: All
OS: Linux
high
high
Target Milestone: z6
: 5.0 (RHEL 6)
Assignee: Jamie Lennox
QA Contact: Mike Abrams
URL:
Whiteboard:
: 1261543 (view as bug list)
Depends On:
Blocks: 1261543
TreeView+ depends on / blocked
 
Reported: 2015-07-23 14:14 UTC by Christian Schwede (cschwede)
Modified: 2023-02-22 23:02 UTC (History)
13 users (show)

Fixed In Version: openstack-keystone-2014.1.5-2.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, there was an issue with Keystone in that it sometimes ran into time outs as it couldn't keep up with multiple requests. This was because there was only a single Keystone process running although there were multiple requests. Since the majority of the work Keystone does is cryptographic calculations and filtering database records, Keystone is CPU-bound. As a result, given that a keystone-all process had only one thread (i.e., eventlet's thread), keystone-all's throughput was limited to the throughput of a single CPU core. To increase keystone-all's throughput and its CPU parallelism, this patch added two configuration options, public_workers=N and admin_workers=N, that determine the number of keystone-all processes that handle requests for Keystone's public and admin WSGI applications respectively; and thereby the issue is now resolved.
Clone Of:
: 1261543 (view as bug list)
Environment:
Last Closed: 2015-10-15 14:04:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Backported patch for RHEL OSP5 (6.05 KB, patch)
2015-07-23 14:14 UTC, Christian Schwede (cschwede)
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 42967 0 None MERGED enable multiple keystone-all worker processes 2020-02-05 11:39:07 UTC

Description Christian Schwede (cschwede) 2015-07-23 14:14:50 UTC
Created attachment 1055393 [details]
Backported patch for RHEL OSP5

Description of problem:

One of our customers has a performance problem with Keystone. The problem is that they face a lot of Keystone requests per second (using Swift), and Keystone runs into timeouts sometimes because it can't keep up with the amount of requests.

The problem is that there is only a single Keystone process running. This was fixed shotly after the Icehouse release: 

https://review.openstack.org/#/c/42967/

I created a diff for RHEL6/OSP5 based on that commit, that can be applied directly on the system (patch attached).

Version-Release number of selected component (if applicable):

RHEL6, OSP5.

How reproducible:

Always.

Steps to Reproduce:
1. /etc/init.d/openstack-keystone start

Actual results:

Single keystone process.

Expected results:
Multiple keystone processes, for example:

$ pstree -c keystone
keystone-all─┬─keystone-all
             ├─keystone-all
             ├─keystone-all
             ├─keystone-all
             ├─keystone-all
             └─keystone-all

Additional info: my question is how to proceed in this case. The customer has a premium support level.

- is there a chance to backport this patch to RHEL OSP5 and publish updated RPMS?
- or shall we deliver the patch directly to the customer to fix this issue asap?

Comment 4 Jamie Lennox 2015-07-23 23:53:18 UTC
needinfo? alan, what is the policy on backporting this far?

Comment 14 Jon Schlueter 2015-09-09 13:37:42 UTC
I just verifed patch for this issue is part of the build openstack-keystone-2014.1.5-2.el6ost

Looking at commit message for patch that brings in this functionality it notes it adds 2 config options public_workers=N and admin_workers=N

This patch adds two configuration options, public_workers=N and
admin_workers=N, that determine the number of keystone-all processes
that handle requests for keystone's public and admin WSGI applications
respectively.

Comment 17 Scott Lewis 2015-09-09 15:33:01 UTC
*** Bug 1261543 has been marked as a duplicate of this bug. ***

Comment 19 Jamie Lennox 2015-09-16 02:16:17 UTC
This indeed requires setting the public_workers and admin_workers config options to something > 1 to see multiple processes.

Comment 24 Lon Hohberger 2015-10-15 14:04:30 UTC
This was resolved in the current release of openstack-keystone for RHEL OSP 5


Note You need to log in before you can comment on or make changes to this bug.