Bug 1267630 - Out of memory on undercloud / overcloud controller due to puppet configuration
Summary: Out of memory on undercloud / overcloud controller due to puppet configuration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 7.0 (Kilo)
Hardware: All
OS: Linux
high
medium
Target Milestone: ga
: 8.0 (Liberty)
Assignee: Jay Dobies
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks: 1271329
TreeView+ depends on / blocked
 
Reported: 2015-09-30 14:22 UTC by Cyril Lopez
Modified: 2016-04-20 13:03 UTC (History)
7 users (show)

Fixed In Version: openstack-tripleo-heat-templates-0.8.8-1.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1271329 (view as bug list)
Environment:
Last Closed: 2016-04-20 13:03:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
OOM kill on overcloud-controller (13.60 KB, image/png)
2015-09-30 14:22 UTC, Cyril Lopez
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 229471 0 None MERGED Implement Workers parameters 2020-07-16 12:30:15 UTC
Red Hat Product Errata RHBA-2016:0653 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OSP 8 director Bug Fix Advisory 2016-04-20 17:02:07 UTC

Description Cyril Lopez 2015-09-30 14:22:57 UTC
Created attachment 1078671 [details]
OOM kill on overcloud-controller

Description of problem:
Puppet configuration is based on processorcount to setup each number of worker of service as nova, neutron...

But he never check amount of ram so in case of I got lot of CPU, I get lot of worker. And quickly I get OOM cause each thread increase his mem usage.

Version-Release number of selected component (if applicable): 
openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch
openstack-puppet-modules-2015.1.8-12.el7ost.noarch

How reproducible:
Install a undercloud or overcloud controller node with 16cpu 8G RAM

Steps to Reproduce:
1. Install undercloud
2. Use it a couple of day
3.

Actual results: OOM killer kill some of process...


Expected results: No more memory

Workaround:

Change value of :
/etc/keystone/keystone.conf:public_workers=16
/etc/keystone/keystone.conf:admin_workers=16
/etc/glance/glance-api.conf:workers=16
/etc/neutron/neutron.conf:api_workers = 16
/etc/nova/nova.conf:ec2_workers=16
/etc/nova/nova.conf:osapi_compute_workers=16

Additional info:

Example for nova:
https://github.com/openstack/puppet-nova/blob/0d57b76bf6c9a8989127cd16505ee62abc049087/manifests/api.pp
# [*ec2_workers*]
#   (optional) Number of workers for EC2 service
#   Defaults to $::processorcount
#
# [*metadata_workers*]
#   (optional) Number of workers for metadata service
#   Defaults to $::processorcount

Comment 2 Emilien Macchi 2015-09-30 15:11:19 UTC
Patch sent upstream: https://review.openstack.org/229471

In the meantime, you can set your custom parameters to ExtraConfig parameter, using the example in the template doc.

Comment 11 Omri Hochman 2016-04-15 20:47:30 UTC
verified with  openstack-tripleo-heat-templates-0.8.14-7.el7ost.noarch

the patch is merged:
https://review.openstack.org/#/c/229471/11/puppet/controller.yaml

Comment 13 errata-xmlrpc 2016-04-20 13:03:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0653.html


Note You need to log in before you can comment on or make changes to this bug.