Bug 1653738 - Build is failing on unable to push the image
Summary: Build is failing on unable to push the image
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.9.z
Assignee: Mrunal Patel
QA Contact: weiwei jiang
Depends On:
TreeView+ depends on / blocked
Reported: 2018-11-27 14:16 UTC by Vladislav Walek
Modified: 2019-09-19 23:16 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-03-08 19:49:28 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Vladislav Walek 2018-11-27 14:16:17 UTC
Description of problem:

The build is failing on push the image to the registry.

Pushing image docker-registry.default.svc.cluster.local:5000/test/ruby-ex:latest ...
error: build error: Failed to push image: unauthorized: authentication required

However, I was able to check the logs which secret is used - builder-dockercfg-<>.
We tried to use the token from the secret and login and push manually the image, it works as expected.

Nevertheless, the build is now failing on:
Running 'bundle install --retry 2 --deployment --without development:test' ...
failed to create fsnotify watcher: too many open files

Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.9

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 2 Ben Parees 2018-11-27 15:01:18 UTC
The error you're now getting points to a node/runtime resource problem, because we're simply running a docker container and it is hitting the too many open files restriction.  Transferring to the runtimes team to help you investigate why you would be hitting this limitation.

Error from the log:

I1122 08:02:41.495624       1 sti.go:681] ---> Running 'bundle install --retry 2 --deployment --without development:test' ...
failed to create fsnotify watcher: too many open files

If you get past that error (w/ help from the runtime team) and start hitting the push error again, we can come back and investigate that.

Comment 3 Mrunal Patel 2018-11-28 22:11:07 UTC
Can you get this value sysctl fs.inotify.max_user_instances? Both on the node and in the container?

Comment 7 Mrunal Patel 2018-12-14 19:57:29 UTC
Can you try bumping up that limit and trying the build?
sysctl -w fs.inotify.max_user_instances = 10000

Note You need to log in before you can comment on or make changes to this bug.