Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1047437

Summary: Scaled app head gear hits nofile ulimits...
Product: OpenShift Container Platform Reporter: Ma xiaoqiang <xiama>
Component: ContainersAssignee: Brenton Leanhardt <bleanhar>
Status: CLOSED DEFERRED QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: medium    
Version: 2.0.0CC: bmeng, chunchen, jeder, libra-onpremise-devel, mfisher, rmillner, twiest, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 971610 Environment:
Last Closed: 2014-02-04 16:28:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 971610    
Bug Blocks:    

Description Ma xiaoqiang 2013-12-31 06:46:53 UTC 
This problem exists on puddle[2.0.1/2013-12-16.3] and puddle[1.2.7/2013-12-16.2]

+++ This bug was initially created as a clone of Bug #971610 +++

Description of problem:
We found while trying to scale the summit presentation that the head haproxy gear is hitting the nofile ulimit (which is set to 4096 by default). 

We added these lines to this file /etc/security/limits.d/84-51ae792fe0b8cdddf2000177.conf:
51ae792fe0b8cdddf2000177               soft    nofile          16384
51ae792fe0b8cdddf2000177               hard    nofile          16384

And that helped out tremendously.


Version-Release number of selected component (if applicable):
rhc-node-1.9.15-1.el6oso.x86_64


How reproducible:
very


Steps to Reproduce:
1. create a scaled app with at least 4 backed gears (5 gears total)
2. pound on the app
3. watch netstat -nap on the head gear's node and notice that the connections are being limited to the head gear.
4. bump the limit
5. notice that the head gear can scale better


Actual results:
Slow head gear because of file descriptors are being limited. This makes the backend gears not be able to contribute enough.


Expected results:
fast connections

--- Additional comment from Rob Millner on 2013-06-12 14:26:02 EDT ---

Only the hard limit needs to be increased.  Haproxy will request an increase up to the hard limit based on the number of connections its configured for.

Release ticket updated.

Pull request for devenv:
https://github.com/openshift/li/pull/1618



To Q/A:
1. create a scaled app.
2. log into your gear and edit haproxy/conf/haproxy.cfg
3. change the value of "maxconn" to 8000
4. restart your app
5. log back into the gear and inspect haproxy/logs/haproxy.log

Expected:
No errors like "FD limit (1024) too low for maxconn=8000..."

--- Additional comment from openshift-github-bot on 2013-06-12 17:35:44 EDT ---

Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/d4b70a6a4a31fd33ac0d25a76d9d42630bd16bf3
Bug 971610 - Increase the hard limit on file descriptors to 16k

--- Additional comment from Meng Bo on 2013-06-14 02:50:53 EDT ---

Checked on devenv_3360,

Step as comment#1,

After restart app, no such errors in haproxy.log

Move bug to verified.