Bug 1605153 - [3.10]failed to create fsnotify watcher: too many open files
Summary: [3.10]failed to create fsnotify watcher: too many open files
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: 3.10.z
Assignee: Vadim Rutkovsky
QA Contact: Weihua Meng
URL:
Whiteboard:
Depends On: 1559061
Blocks: 1605148 1605152
TreeView+ depends on / blocked
 
Reported: 2018-07-20 09:57 UTC by Weihua Meng
Modified: 2018-07-30 20:22 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1559061
Environment:
Last Closed: 2018-07-30 20:22:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2263 0 None None None 2018-07-30 20:22:54 UTC

Comment 1 Scott Dodson 2018-07-20 12:49:40 UTC
In openshift-ansible-3.10.20-1 and later

Comment 3 Weihua Meng 2018-07-23 06:53:19 UTC
Hi, Avesh

I found those changes are only applied to nodes, masters are not applied.
Is it expected?

on master
# sysctl -a | grep fs.inotify.max_user_
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 65536

on node:
# sysctl -a | grep fs.inotify.max_user_
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 65536


openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch.rpm

Comment 4 Johnny Liu 2018-07-23 09:57:12 UTC
Based on comment 3, change back to ASSIGNED status to let dev have a quick glance because it is very close to 3.10 release date.

Comment 5 Scott Dodson 2018-07-23 12:29:38 UTC
(In reply to Weihua Meng from comment #3)
> Hi, Avesh
> 
> I found those changes are only applied to nodes, masters are not applied.
> Is it expected?
> 
> on master
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 128
> fs.inotify.max_user_watches = 65536
> 
> on node:
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 8192
> fs.inotify.max_user_watches = 65536
> 
> 
> openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch.rpm

This bug seems to be related to tailing log files on builds and builds should not run on masters so I think this is fine for now. I'll follow up with the performance and node teams to see if we should change this in the future though.

Comment 6 Avesh Agarwal 2018-07-23 13:05:21 UTC
(In reply to Weihua Meng from comment #3)
> Hi, Avesh
> 
> I found those changes are only applied to nodes, masters are not applied.
> Is it expected?

Yes.

> 
> on master
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 128
> fs.inotify.max_user_watches = 65536
> 
> on node:
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 8192
> fs.inotify.max_user_watches = 65536
> 
> 
> openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch.rpm

Comment 7 Weihua Meng 2018-07-23 14:15:00 UTC
Verified per comment 5 & comment 6

Comment 9 errata-xmlrpc 2018-07-30 20:22:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2263


Note You need to log in before you can comment on or make changes to this bug.