Bug 1267883

This is the THT part of the fix

+++ This bug was initially created as a clone of Bug #1240587 +++

Description of problem:
Unable to control the file_descriptors limit for rabbitmq-server via the director.

Version-Release number of selected component (if applicable):
Current.

How reproducible:
Always.

Steps to Reproduce:
1. Deploy an overcloud using OSP-d.

Actual results:
rabbitmq-sever is launched via the rabbitmq resource agent using the following file_descriptors limit :

# pcs resource show rabbitmq
 Resource: rabbitmq (class=ocf provider=heartbeat type=rabbitmq-cluster)
  Attributes: set_policy="ha-all ^(?!amq\.).* {"ha-mode":"all"}" 
  Operations: start interval=0s timeout=100 (rabbitmq-start-timeout-100)
              stop interval=0s timeout=90 (rabbitmq-stop-timeout-90)
              monitor interval=10 timeout=40 (rabbitmq-monitor-interval-10)

# rabbitmqctl report | grep -A3 file_descriptors
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
--
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
--
 {file_descriptors,[{total_limit,924},
                    {total_used,152},
                    {sockets_limit,829},
                    {sockets_used,150}]},

Expected results:
rabbitmq-server is launched via the rabbitmq resource agent but is able to control the file_descriptors limit via a parameter. As requested in BZ#1240561.

Additional info:


--- Additional comment from David Vossel on 2015-07-07 11:31:00 EDT ---

(In reply to Rafael Rosa from comment #3)
> Can the file_descriptors limit be increased manually as a work around? I'm
> trying to assess the urgency and if needs to be a blocker for GA. I would
> rather deal with it on A1 or A2, given our time constraints.

yes. the fix is trivial. 

echo "ulimit -S -n 4096" >> /etc/rabbitmq/rabbitmq-env.conf

after restarting rabbitmq we can see this setting is used.

rabbitmqctl status | grep -C 3 total_limit
 {vm_memory_limit,667051622},
 {disk_free_limit,50000000},
 {disk_free,11562962944},
 {file_descriptors,[{total_limit,3996},
                    {total_used,3},
                    {sockets_limit,3594},
                    {sockets_used,1}]},

--- Additional comment from Emilien Macchi on 2015-07-14 18:00:25 EDT ---

if it can help, you can also use this parameter[1] to customize the limit.

[1] https://github.com/puppetlabs/puppetlabs-rabbitmq/blob/95498e4174915dbce81c81bd42d6ec3b87df14b3/manifests/init.pp#L63

--- Additional comment from John Eckersberg on 2015-08-19 17:05:39 EDT ---

reposting from https://bugzilla.redhat.com/show_bug.cgi?id=1255091#c3 ... don't want it to get lost:

upstream pr - https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/381

--- Additional comment from Emilien Macchi on 2015-08-19 17:27:27 EDT ---

I merged it.

--- Additional comment from James Slagle on 2015-08-27 12:18:26 EDT ---

we still need the tripleo-heat-templates update to set this parameter

--- Additional comment from  on 2015-09-24 10:02:10 EDT ---

Failed QA -Tested on:
openstack-puppet-modules-2015.1.8-17.el7ost.noarch

steps:
 - Changed the values from 16384 to 8192 on instack: /etc/security/limits.d/rabbitmq-server.conf
 - Deployed the overcloud (tried with and without tuskar)
 - Overcloud controller rabbitmq FD limit was still 16384 

Tried editing:
 - /usr/share/openstack-puppet/modules/rabbitmq/manifests/params.pp
 - Deployed the overcloud (tried with and without tuskar)
 - Overcloud controller rabbitmq FD limit was still 16384

--- Additional comment from Martin Magr on 2015-09-24 10:49:21 EDT ---

According to comment #15 tht does not have support. If you would try to run puppet manifest manually without RHOSd, the parameter would be changed.

--- Additional comment from Marius Cornea on 2015-09-30 17:55:23 EDT ---

The tripleo-heat-templates patch attached to the bug doesn't seem to be present in the 09-25 puddle. Was it updated so it can take the fd limit as parameter? I'm trying to verify this but we should be able to pass the fd limit as a parameter, not run puppet manually.