Bug 1267630

Summary: Out of memory on undercloud / overcloud controller due to puppet configuration
Product: Red Hat OpenStack Reporter: Cyril Lopez <cylopez>
Component: openstack-tripleo-heat-templatesAssignee: Jay Dobies <jason.dobies>
Status: CLOSED ERRATA QA Contact: Omri Hochman <ohochman>
Severity: medium Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: achernet, emacchi, hbrock, mburns, morazi, rhel-osp-director-maint, yguenane
Target Milestone: gaKeywords: TestOnly, Triaged
Target Release: 8.0 (Liberty)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.8-1.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1271329 (view as bug list) Environment:
Last Closed: 2016-04-20 13:03:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1271329    
Attachments:
Description Flags
OOM kill on overcloud-controller none

Description Cyril Lopez 2015-09-30 14:22:57 UTC 
Created attachment 1078671 [details]
OOM kill on overcloud-controller

Description of problem:
Puppet configuration is based on processorcount to setup each number of worker of service as nova, neutron...

But he never check amount of ram so in case of I got lot of CPU, I get lot of worker. And quickly I get OOM cause each thread increase his mem usage.

Version-Release number of selected component (if applicable): 
openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch
openstack-puppet-modules-2015.1.8-12.el7ost.noarch

How reproducible:
Install a undercloud or overcloud controller node with 16cpu 8G RAM

Steps to Reproduce:
1. Install undercloud
2. Use it a couple of day
3.

Actual results: OOM killer kill some of process...


Expected results: No more memory

Workaround:

Change value of :
/etc/keystone/keystone.conf:public_workers=16
/etc/keystone/keystone.conf:admin_workers=16
/etc/glance/glance-api.conf:workers=16
/etc/neutron/neutron.conf:api_workers = 16
/etc/nova/nova.conf:ec2_workers=16
/etc/nova/nova.conf:osapi_compute_workers=16

Additional info:

Example for nova:
https://github.com/openstack/puppet-nova/blob/0d57b76bf6c9a8989127cd16505ee62abc049087/manifests/api.pp
# [*ec2_workers*]
#   (optional) Number of workers for EC2 service
#   Defaults to $::processorcount
#
# [*metadata_workers*]
#   (optional) Number of workers for metadata service
#   Defaults to $::processorcount
Comment 2 Emilien Macchi 2015-09-30 15:11:19 UTC 
Patch sent upstream: https://review.openstack.org/229471

In the meantime, you can set your custom parameters to ExtraConfig parameter, using the example in the template doc.
Comment 11 Omri Hochman 2016-04-15 20:47:30 UTC 
verified with  openstack-tripleo-heat-templates-0.8.14-7.el7ost.noarch

the patch is merged:
https://review.openstack.org/#/c/229471/11/puppet/controller.yaml
Comment 13 errata-xmlrpc 2016-04-20 13:03:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0653.html