Bug 1573500

Summary: Keystone still configured for single process
Product: Red Hat OpenStack Reporter: Alex Krzos <akrzos>
Component: puppet-keystoneAssignee: Harry Rybacki <hrybacki>
Status: CLOSED ERRATA QA Contact: Pavan <pkesavar>
Severity: low Docs Contact:
Priority: low    
Version: 12.0 (Pike)CC: akrzos, aschultz, hrybacki, jagee, jeder, jjoyce, josorior, jschluet, kbasil, mburns, nkinder, slinaber, tvignaud
Target Milestone: zstreamKeywords: Triaged, ZStream
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: aos-scalability-310
Fixed In Version: puppet-keystone-11.4.0-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-05 18:52:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Krzos 2018-05-01 14:18:29 UTC
Description of problem:
Keystone is configured as a single process on OSP11 / Ocata z5

Version-Release number of selected component (if applicable):
OSP11z5

How reproducible:
Always

Steps to Reproduce:
1. Install Undercloud and see how many Keystone admin/main api processes exist
2.
3.

Actual results:
Single process despite multi-core system

Expected results:
Several keystone processes to handle greater load against keystone on undercloud

Additional info:

Anything > 10 nodes will suffer a slow overcloud install due to the single keystone process only able to use a ~1.2cores while building an overcloud.  For Example, we just recently built an 82 node cluster (3 controllers, 10 cephs, 69 computes) and the deploy actually exceeded the 4 hour timeout due to keystone spending most of its time stuck on 1.2 cores.  This easily can be tuned to more cores and greatly speeds up the deployment process.  OSP10 even has this fix.

This was supposedly fixed in OSP11 before according to this bz:
https://bugzilla.redhat.com/show_bug.cgi?id=1330980

[root@b04-h19-1029p ~]# cat /etc/version.json 
{
    "osp_series": "ocata",
    "osp_version": "11",
    "rhos_release": "11-director",
    "build": "z5",
    "uc_build_date": "20180501-140811"
}
[root@b04-h19-1029p ~]# rpm -qa | grep keystone
python-keystonemiddleware-4.14.0-1.el7ost.noarch
puppet-keystone-10.4.0-1.el7ost.noarch
python-keystone-11.0.3-5.el7ost.noarch
python-keystoneclient-3.10.0-1.el7ost.noarch
openstack-keystone-11.0.3-5.el7ost.noarch
python-keystoneauth1-2.18.0-1.el7ost.noarch
[root@b04-h19-1029p ~]# ps afx | grep keystone
 47316 pts/1    S+     0:00      \_ grep --color=auto keystone
 18547 ?        Sl     0:14  \_ keystone-admin  -DFOREGROUND
 18548 ?        Sl     0:09  \_ keystone-main   -DFOREGROUND
[root@b04-h19-1029p ~]# grep processes /etc/httpd/conf.d/10-keystone_wsgi_*
/etc/httpd/conf.d/10-keystone_wsgi_admin.conf:  WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=1 threads=12 user=keystone
/etc/httpd/conf.d/10-keystone_wsgi_main.conf:  WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=1 threads=12 user=keystone
[root@b04-h19-1029p ~]# lscpu | grep "^CPU(s):"
CPU(s):                64

I may have opened this against the incorrect component but I believe this is *fixed* in puppet-keystone but something is passing in the wrong parameters and I am not sure what component that is doing this.

Comment 8 Juan Antonio Osorio 2018-06-19 09:24:04 UTC
Alex, what's the undercloud.conf that you used to deploy?

Comment 9 Juan Antonio Osorio 2018-06-19 09:27:03 UTC
Alex, also, you are right, this is still an issue in OSP12/Pike. The fix landed for Queens though.

Comment 10 Alex Krzos 2018-06-21 12:58:57 UTC
(In reply to Juan Antonio Osorio from comment #8)
> Alex, what's the undercloud.conf that you used to deploy?

For OSP11 or OSP12?

The OSP11 undercloud.conf is available here:
https://gist.github.com/akrzos/c85bd1a281905f88e51c89efeedb84bf

Comment 11 Harry Rybacki 2018-06-25 19:09:37 UTC
Upstream review has merged. Moving RHBZ to POST.

Comment 12 Harry Rybacki 2018-11-08 16:57:54 UTC
Downstream build complete. Updatinf FIV and moving to MODIFIED.

Comment 21 errata-xmlrpc 2018-12-05 18:52:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3789