Bug 1267630 - Out of memory on undercloud / overcloud controller due to puppet configuration
Out of memory on undercloud / overcloud controller due to puppet configuration
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
7.0 (Kilo)
All Linux
high Severity medium
: ga
: 8.0 (Liberty)
Assigned To: Jay Dobies
Omri Hochman
: TestOnly, Triaged
Depends On:
Blocks: 1271329
  Show dependency treegraph
 
Reported: 2015-09-30 10:22 EDT by Cyril Lopez
Modified: 2016-04-20 09:03 EDT (History)
7 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-0.8.8-1.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1271329 (view as bug list)
Environment:
Last Closed: 2016-04-20 09:03:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
OOM kill on overcloud-controller (13.60 KB, image/png)
2015-09-30 10:22 EDT, Cyril Lopez
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 229471 None None None Never

  None (edit)
Description Cyril Lopez 2015-09-30 10:22:57 EDT
Created attachment 1078671 [details]
OOM kill on overcloud-controller

Description of problem:
Puppet configuration is based on processorcount to setup each number of worker of service as nova, neutron...

But he never check amount of ram so in case of I got lot of CPU, I get lot of worker. And quickly I get OOM cause each thread increase his mem usage.

Version-Release number of selected component (if applicable): 
openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch
openstack-puppet-modules-2015.1.8-12.el7ost.noarch

How reproducible:
Install a undercloud or overcloud controller node with 16cpu 8G RAM

Steps to Reproduce:
1. Install undercloud
2. Use it a couple of day
3.

Actual results: OOM killer kill some of process...


Expected results: No more memory

Workaround:

Change value of :
/etc/keystone/keystone.conf:public_workers=16
/etc/keystone/keystone.conf:admin_workers=16
/etc/glance/glance-api.conf:workers=16
/etc/neutron/neutron.conf:api_workers = 16
/etc/nova/nova.conf:ec2_workers=16
/etc/nova/nova.conf:osapi_compute_workers=16

Additional info:

Example for nova:
https://github.com/openstack/puppet-nova/blob/0d57b76bf6c9a8989127cd16505ee62abc049087/manifests/api.pp
# [*ec2_workers*]
#   (optional) Number of workers for EC2 service
#   Defaults to $::processorcount
#
# [*metadata_workers*]
#   (optional) Number of workers for metadata service
#   Defaults to $::processorcount
Comment 2 Emilien Macchi 2015-09-30 11:11:19 EDT
Patch sent upstream: https://review.openstack.org/229471

In the meantime, you can set your custom parameters to ExtraConfig parameter, using the example in the template doc.
Comment 11 Omri Hochman 2016-04-15 16:47:30 EDT
verified with  openstack-tripleo-heat-templates-0.8.14-7.el7ost.noarch

the patch is merged:
https://review.openstack.org/#/c/229471/11/puppet/controller.yaml
Comment 13 errata-xmlrpc 2016-04-20 09:03:47 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0653.html

Note You need to log in before you can comment on or make changes to this bug.