Bug 1282491 - [RabbitMQ] update default file descriptor setting
Summary: [RabbitMQ] update default file descriptor setting
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 7.0 (Kilo)
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: Upstream M3
: 10.0 (Newton)
Assignee: Chris Jones
QA Contact: Tomas Jamrisko
URL:
Whiteboard:
Depends On:
Blocks: 1353943 1356490
TreeView+ depends on / blocked
 
Reported: 2015-11-16 15:21 UTC by Joe Talerico
Modified: 2016-12-14 15:17 UTC (History)
23 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20160820164503.6c537d2.1.el7ost
Doc Type: Bug Fix
Doc Text:
Prior to this update, the RabbitMQ maximum open file descriptors was set to 4096. Consequently, customers with larger deployments could hit this limit and face stability issues. With this update, the maximum open file descriptor limit for RabbitMQ has been increased to 65536. As a result, larger deployments should now be significantly less likely to run into this issue.
Clone Of:
: 1353943 (view as bug list)
Environment:
Last Closed: 2016-12-14 15:17:43 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 332127 None None None 2016-06-22 01:47:36 UTC

Description Joe Talerico 2015-11-16 15:21:02 UTC
Description of problem:
Current OSP Deployments deploy w/ ~4k file descriptors. I could of sworn previously we deployed with 16k.

From Rabbit:
 {file_descriptors,[{total_limit,3996},
                    {total_used,945},
                    {sockets_limit,3594},
                    {sockets_used,943}]},

Chatting with eck we should deploy with 16k, instead of 4k.

Comment 2 Joe Talerico 2015-11-16 15:23:04 UTC
eck, can you ack the 16k default?

Comment 3 Peter Lemenkov 2015-11-16 15:25:10 UTC
(In reply to Joe Talerico from comment #0)
> Description of problem:
> Current OSP Deployments deploy w/ ~4k file descriptors. I could of sworn
> previously we deployed with 16k.
> 
> From Rabbit:
>  {file_descriptors,[{total_limit,3996},
>                     {total_used,945},
>                     {sockets_limit,3594},
>                     {sockets_used,943}]},
> 
> Chatting with eck we should deploy with 16k, instead of 4k.

I'm perfectly fine with even more. Let's say 32k or 64. However let's start from 16k first.

Comment 4 John Eckersberg 2015-11-16 15:26:08 UTC
(In reply to Joe Talerico from comment #2)
> eck, can you ack the 16k default?

ACK all the things

Comment 5 Peter Lemenkov 2015-11-17 21:21:02 UTC
Ok since we agree about this, let's reassign it to osp-director. This is a place where all the OpenStack configs lives.

Comment 7 Sadique Puthen 2016-03-23 06:15:11 UTC
We hit the 4k limit for two important customers. Rakutan Marketing and Telefonica. Let us make sure that we increase it to 64k

Please note hitting this limit and spending time to troubleshoot it again and again affects customer satisfaction. So nominating for Async and increasing priority to Urgent.

Comment 8 Mike Burns 2016-03-23 13:43:35 UTC
Jarda, want to comment on prioritization?

Comment 12 Mike Burns 2016-04-07 20:57:01 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 14 Rob Young 2016-04-21 13:45:20 UTC
This bug was recently brought to my attention and the problem being reported is a well-known issue with MySQL and MariaDB. We should correct our default deployment to 32k, ideally 64k as soon as is reasonable as most workloads will hit or exceed the 4k limit running less than peak activity.

Comment 15 Rob Young 2016-04-21 13:48:14 UTC
This bug was recently brought to my attention. We should correct our default deployment to 32k, ideally 64k as soon as is reasonable as most workloads will hit or exceed the 4k limit running less than peak activity. Will defer to Jarda on what is reasonable in this case.

Comment 19 Chris Jones 2016-06-21 13:09:31 UTC
RHOS 8/9/10 currently have a limit of 16k files.

Patch proposed upstream to increase the limit to 64k files: https://review.openstack.org/#/c/332127/

Comment 22 Leonid Natapov 2016-09-14 12:04:22 UTC
openstack-tripleo-heat-templates-5.0.0-0.20160823140311.72404b.1.el7ost.noarch

Checked on clean osp10 deployment

{file_descriptors,[{total_limit,65436},

Need to be checked also on upgraded deployment osp9--->osp10

Comment 24 Marian Krcmarik 2016-10-19 20:32:18 UTC
Verified after upgrade from 9 to 10.

Comment 27 errata-xmlrpc 2016-12-14 15:17:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.