Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 971610 - Scaled app head gear hits nofile ulimits...
Scaled app head gear hits nofile ulimits...
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Rob Millner
libra bugs
Depends On:
Blocks: 1047437
  Show dependency treegraph
Reported: 2013-06-06 18:51 EDT by Thomas Wiest
Modified: 2015-05-14 19:21 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1047437 (view as bug list)
Last Closed: 2013-06-24 10:52:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Thomas Wiest 2013-06-06 18:51:00 EDT
Description of problem:
We found while trying to scale the summit presentation that the head haproxy gear is hitting the nofile ulimit (which is set to 4096 by default). 

We added these lines to this file /etc/security/limits.d/84-51ae792fe0b8cdddf2000177.conf:
51ae792fe0b8cdddf2000177               soft    nofile          16384
51ae792fe0b8cdddf2000177               hard    nofile          16384

And that helped out tremendously.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. create a scaled app with at least 4 backed gears (5 gears total)
2. pound on the app
3. watch netstat -nap on the head gear's node and notice that the connections are being limited to the head gear.
4. bump the limit
5. notice that the head gear can scale better

Actual results:
Slow head gear because of file descriptors are being limited. This makes the backend gears not be able to contribute enough.

Expected results:
fast connections
Comment 1 Rob Millner 2013-06-12 14:26:02 EDT
Only the hard limit needs to be increased.  Haproxy will request an increase up to the hard limit based on the number of connections its configured for.

Release ticket updated.

Pull request for devenv:

To Q/A:
1. create a scaled app.
2. log into your gear and edit haproxy/conf/haproxy.cfg
3. change the value of "maxconn" to 8000
4. restart your app
5. log back into the gear and inspect haproxy/logs/haproxy.log

No errors like "FD limit (1024) too low for maxconn=8000..."
Comment 2 openshift-github-bot 2013-06-12 17:35:44 EDT
Commit pushed to master at https://github.com/openshift/li

Bug 971610 - Increase the hard limit on file descriptors to 16k
Comment 3 Meng Bo 2013-06-14 02:50:53 EDT
Checked on devenv_3360,

Step as comment#1,

After restart app, no such errors in haproxy.log

Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.