Bug 1525577 - ceph-ansible fork count should be a parameter
Summary: ceph-ansible fork count should be a parameter
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 12.0 (Pike)
Assignee: John Fulton
QA Contact: Yogev Rabl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-13 15:44 UTC by John Fulton
Modified: 2018-03-28 17:28 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-common-7.6.9-1.el7ost openstack-tripleo-heat-templates-7.0.9-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-28 17:27:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1738276 0 None None None 2017-12-14 23:29:49 UTC
OpenStack gerrit 531616 0 'None' MERGED Lowering the number of ansible forks to 25 2020-07-29 22:54:59 UTC
OpenStack gerrit 535291 0 'None' MERGED Parameterize Ansible environment vars in Mistral Workflow 2020-07-29 22:54:59 UTC
OpenStack gerrit 535292 0 'None' MERGED Parameterize ceph-ansible environment variables 2020-07-29 22:54:58 UTC
Red Hat Product Errata RHBA-2018:0607 0 None None None 2018-03-28 17:28:17 UTC

Description John Fulton 2017-12-13 15:44:07 UTC
By default OSP12 sets the ansible fork count to maximize the number of concurrent ceph-ansible processes [0] by setting it equal to the number of nodes needing to be configured up to 100. However, if the undercloud doesn't have enough memory, then this setting cause the undercloud to run out of RAM [1]. The user should be able to easily override this parameter and not use the formula. 

[0] https://github.com/openstack/tripleo-common/blob/fa0b9f52080580b7408dc6f5f2da6fc1dc07d500/workbooks/ceph-ansible.yaml#L25

[1] 
2017-12-13 04:05:53,887 p=7259 u=mistral |  ERROR! Unexpected Exception, this is probably a bug: [Errno 12] Cannot allocate memory                                                            │····················
2017-12-13 04:05:53,893 p=7259 u=mistral |  to see the full traceback, use -vvv                                                                                                               │····················
2017-12-13 04:05:58,541 p=7259 u=mistral |  the full traceback was:                                                                                                                           │····················
                                                                                                                                                                                              │····················
Traceback (most recent call last):                                                                                                                                                            │····················
  File "/bin/ansible-playbook", line 106, in <module>                                                                                                                                         │····················
    exit_code = cli.run()                                                                                                                                                                     │····················
  File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 130, in run                                                                                                           │····················
    results = pbex.run()                                                                                                                                                                      │····················
  File "/usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py", line 154, in run                                                                                             │····················
    result = self._tqm.run(play=play)                                                                                                                                                         │····················
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 302, in run                                                                                            │····················
    play_return = strategy.run(iterator, play_context)                                                                                                                                        │····················
  File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/linear.py", line 277, in run                                                                                                │····················
    self._queue_task(host, task, task_vars, play_context)                                                                                                                                     │····················
  File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/__init__.py", line 222, in _queue_task                                                                                      │····················
    worker_prc.start()                                                                                                                                                                        │····················
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start                                                                                                                  │····················
    self._popen = Popen(self)                                                                                                                                                                 │····················
  File "/usr/lib64/python2.7/multiprocessing/forking.py", line 121, in __init__                                                                                                               │····················
    self.pid = os.fork()                                                                                                                                                                      │····················
OSError: [Errno 12] Cannot allocate memory

Comment 3 John Fulton 2018-01-10 15:28:19 UTC
- relevant patches merged upstream for pike
- a related, but not the same bug, is https://bugzilla.redhat.com/show_bug.cgi?id=1527205

Comment 10 Yogev Rabl 2018-03-13 19:18:15 UTC
verified on openstack-tripleo-common-7.6.9-2.el7ost.noarch

Comment 13 errata-xmlrpc 2018-03-28 17:27:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0607


Note You need to log in before you can comment on or make changes to this bug.