Bug 1653738

Summary:	Build is failing on unable to push the image
Product:	OpenShift Container Platform	Reporter:	Vladislav Walek <vwalek>
Component:	Containers	Assignee:	Mrunal Patel <mpatel>
Status:	CLOSED NOTABUG	QA Contact:	weiwei jiang <wjiang>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, jokerman, mmccomas, mpatel, vwalek, wzheng
Target Milestone:	---
Target Release:	3.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-03-08 19:49:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vladislav Walek 2018-11-27 14:16:17 UTC

Description of problem:

The build is failing on push the image to the registry.

Pushing image docker-registry.default.svc.cluster.local:5000/test/ruby-ex:latest ...
error: build error: Failed to push image: unauthorized: authentication required

However, I was able to check the logs which secret is used - builder-dockercfg-<>.
We tried to use the token from the secret and login and push manually the image, it works as expected.

Nevertheless, the build is now failing on:
Running 'bundle install --retry 2 --deployment --without development:test' ...
failed to create fsnotify watcher: too many open files

Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.9

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Ben Parees 2018-11-27 15:01:18 UTC

The error you're now getting points to a node/runtime resource problem, because we're simply running a docker container and it is hitting the too many open files restriction.  Transferring to the runtimes team to help you investigate why you would be hitting this limitation.

Error from the log:

I1122 08:02:41.495624       1 sti.go:681] ---> Running 'bundle install --retry 2 --deployment --without development:test' ...
failed to create fsnotify watcher: too many open files


If you get past that error (w/ help from the runtime team) and start hitting the push error again, we can come back and investigate that.

Comment 3 Mrunal Patel 2018-11-28 22:11:07 UTC

Can you get this value sysctl fs.inotify.max_user_instances? Both on the node and in the container?

Comment 7 Mrunal Patel 2018-12-14 19:57:29 UTC

Can you try bumping up that limit and trying the build?
sysctl -w fs.inotify.max_user_instances = 10000