Bug 1605153

Summary:	[3.10]failed to create fsnotify watcher: too many open files
Product:	OpenShift Container Platform	Reporter:	Weihua Meng <wmeng>
Component:	Installer	Assignee:	Vadim Rutkovsky <vrutkovs>
Status:	CLOSED ERRATA	QA Contact:	Weihua Meng <wmeng>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.10.0	CC:	acomabon, aos-bugs, avagarwa, jialiu, jokerman, lmeyer, lucarval, mmccomas, rmeggins, sdodson, vrutkovs
Target Milestone:	---
Target Release:	3.10.z
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1559061	Environment:
Last Closed:	2018-07-30 20:22:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1559061
Bug Blocks:	1605148, 1605152

Comment 1 Scott Dodson 2018-07-20 12:49:40 UTC

In openshift-ansible-3.10.20-1 and later

Comment 3 Weihua Meng 2018-07-23 06:53:19 UTC

Hi, Avesh

I found those changes are only applied to nodes, masters are not applied.
Is it expected?

on master
# sysctl -a | grep fs.inotify.max_user_
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 65536

on node:
# sysctl -a | grep fs.inotify.max_user_
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 65536


openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch.rpm

Comment 4 Johnny Liu 2018-07-23 09:57:12 UTC

Based on comment 3, change back to ASSIGNED status to let dev have a quick glance because it is very close to 3.10 release date.

Comment 5 Scott Dodson 2018-07-23 12:29:38 UTC

(In reply to Weihua Meng from comment #3)
> Hi, Avesh
> 
> I found those changes are only applied to nodes, masters are not applied.
> Is it expected?
> 
> on master
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 128
> fs.inotify.max_user_watches = 65536
> 
> on node:
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 8192
> fs.inotify.max_user_watches = 65536
> 
> 
> openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch.rpm

This bug seems to be related to tailing log files on builds and builds should not run on masters so I think this is fine for now. I'll follow up with the performance and node teams to see if we should change this in the future though.

Comment 6 Avesh Agarwal 2018-07-23 13:05:21 UTC

(In reply to Weihua Meng from comment #3)
> Hi, Avesh
> 
> I found those changes are only applied to nodes, masters are not applied.
> Is it expected?

Yes.

> 
> on master
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 128
> fs.inotify.max_user_watches = 65536
> 
> on node:
> # sysctl -a | grep fs.inotify.max_user_
> fs.inotify.max_user_instances = 8192
> fs.inotify.max_user_watches = 65536
> 
> 
> openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch.rpm

Comment 7 Weihua Meng 2018-07-23 14:15:00 UTC

Verified per comment 5 & comment 6

Comment 9 errata-xmlrpc 2018-07-30 20:22:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2263