Bug 1240587 - Unable to control the file_descriptors limit for rabbitmq-server via the director.
Summary: Unable to control the file_descriptors limit for rabbitmq-server via the dire...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules
Version: 7.0 (Kilo)
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: z2
: 7.0 (Kilo)
Assignee: Ivan Chavero
QA Contact: Marius Cornea
URL:
Whiteboard:
: 1255091 1257400 (view as bug list)
Depends On: 1240561
Blocks: 1241628 1267883
TreeView+ depends on / blocked
 
Reported: 2015-07-07 09:51 UTC by Lee Yarwood
Modified: 2015-10-08 12:23 UTC (History)
16 users (show)

Fixed In Version: openstack-puppet-modules-2015.1.8-13.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the director contained no direct mechanism to control the file_descriptors limit for rabbitmq-server because the file limits snippet was not installed on Red Hat Enterprise Linux 7, which uses systemd. With this update, the file limits snippet is installed as /etc/security/limits.d/rabbitmq-server.conf unconditionally, which allows users to control the limit.
Clone Of:
: 1241628 1267883 (view as bug list)
Environment:
Last Closed: 2015-10-08 12:23:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 201796 0 None None None Never
Red Hat Product Errata RHBA-2015:1872 0 normal SHIPPED_LIVE openstack-puppet-modules bug fix and enhancement update 2015-10-08 16:14:03 UTC

Description Lee Yarwood 2015-07-07 09:51:22 UTC
Description of problem:
Unable to control the file_descriptors limit for rabbitmq-server via the director.

Version-Release number of selected component (if applicable):
Current.

How reproducible:
Always.

Steps to Reproduce:
1. Deploy an overcloud using OSP-d.

Actual results:
rabbitmq-sever is launched via the rabbitmq resource agent using the following file_descriptors limit :

# pcs resource show rabbitmq
 Resource: rabbitmq (class=ocf provider=heartbeat type=rabbitmq-cluster)
  Attributes: set_policy="ha-all ^(?!amq\.).* {"ha-mode":"all"}" 
  Operations: start interval=0s timeout=100 (rabbitmq-start-timeout-100)
              stop interval=0s timeout=90 (rabbitmq-stop-timeout-90)
              monitor interval=10 timeout=40 (rabbitmq-monitor-interval-10)

# rabbitmqctl report | grep -A3 file_descriptors
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
--
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
--
 {file_descriptors,[{total_limit,924},
                    {total_used,152},
                    {sockets_limit,829},
                    {sockets_used,150}]},

Expected results:
rabbitmq-server is launched via the rabbitmq resource agent but is able to control the file_descriptors limit via a parameter. As requested in BZ#1240561.

Additional info:

Comment 9 Emilien Macchi 2015-07-14 22:00:25 UTC
if it can help, you can also use this parameter[1] to customize the limit.

[1] https://github.com/puppetlabs/puppetlabs-rabbitmq/blob/95498e4174915dbce81c81bd42d6ec3b87df14b3/manifests/init.pp#L63

Comment 11 James Slagle 2015-08-19 21:02:44 UTC
*** Bug 1255091 has been marked as a duplicate of this bug. ***

Comment 12 James Slagle 2015-08-19 21:04:01 UTC
this is a blocker for a1 IMO, as any deployment of decent scale (20 nodes) is going to start bumping into the 1024 limit

Comment 13 John Eckersberg 2015-08-19 21:05:39 UTC
reposting from https://bugzilla.redhat.com/show_bug.cgi?id=1255091#c3 ... don't want it to get lost:

upstream pr - https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/381

Comment 14 Emilien Macchi 2015-08-19 21:27:27 UTC
I merged it.

Comment 15 James Slagle 2015-08-27 16:18:26 UTC
we still need the tripleo-heat-templates update to set this parameter

Comment 16 James Slagle 2015-08-27 16:19:22 UTC
*** Bug 1257398 has been marked as a duplicate of this bug. ***

Comment 17 James Slagle 2015-08-27 16:22:34 UTC
*** Bug 1257400 has been marked as a duplicate of this bug. ***

Comment 20 Udi Shkalim 2015-09-24 14:02:10 UTC
Failed QA -Tested on:
openstack-puppet-modules-2015.1.8-17.el7ost.noarch

steps:
 - Changed the values from 16384 to 8192 on instack: /etc/security/limits.d/rabbitmq-server.conf
 - Deployed the overcloud (tried with and without tuskar)
 - Overcloud controller rabbitmq FD limit was still 16384 

Tried editing:
 - /usr/share/openstack-puppet/modules/rabbitmq/manifests/params.pp
 - Deployed the overcloud (tried with and without tuskar)
 - Overcloud controller rabbitmq FD limit was still 16384

Comment 21 Martin Magr 2015-09-24 14:49:21 UTC
According to comment #15 tht does not have support. If you would try to run puppet manifest manually without RHOSd, the parameter would be changed.

Comment 24 Marius Cornea 2015-10-02 13:28:50 UTC
stack@instack:~>>> cat templates/rabbit.yaml 
parameters:
    RabbitFDLimit: 7192

stack@instack:~>>> openstack overcloud deploy --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --ntp-server clock.redhat.com --libvirt-type qemu -e ~/templates/rabbit.yaml 

stack@instack:~>>> for ip in $(nova list  | grep controller |  awk '{print $12;}' | cut -d "=" -f2); do ssh heat-admin@$ip 'sudo rabbitmqctl report | grep -A3 file_descriptors';done
 {file_descriptors,[{total_limit,7092},
                    {total_used,3},
                    {sockets_limit,6380},
                    {sockets_used,1}]},
--
 {file_descriptors,[{total_limit,7092},
                    {total_used,4},
                    {sockets_limit,6380},
                    {sockets_used,2}]},
--
 {file_descriptors,[{total_limit,7092},
                    {total_used,197},
                    {sockets_limit,6380},
                    {sockets_used,195}]},
 {file_descriptors,[{total_limit,7092},
                    {total_used,3},
                    {sockets_limit,6380},
                    {sockets_used,1}]},
--
 {file_descriptors,[{total_limit,7092},
                    {total_used,197},
                    {sockets_limit,6380},
                    {sockets_used,195}]},
--
 {file_descriptors,[{total_limit,7092},
                    {total_used,4},
                    {sockets_limit,6380},
                    {sockets_used,2}]},
 {file_descriptors,[{total_limit,7092},
                    {total_used,197},
                    {sockets_limit,6380},
                    {sockets_used,195}]},
--
 {file_descriptors,[{total_limit,7092},
                    {total_used,4},
                    {sockets_limit,6380},
                    {sockets_used,2}]},
--
 {file_descriptors,[{total_limit,7092},
                    {total_used,3},
                    {sockets_limit,6380},
                    {sockets_used,1}]},

Comment 26 errata-xmlrpc 2015-10-08 12:23:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1872


Note You need to log in before you can comment on or make changes to this bug.