Bug 1928850

Summary: Unable to pull images due to limited quota on Docker Hub
Product: OpenShift Container Platform Reporter: Xavier Coulon <xcoulon>
Component: BuildAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: wewang <wewang>
Severity: low Docs Contact: Rolfe Dlugy-Hegwer <rdlugyhe>
Priority: unspecified    
Version: 4.6CC: alkazako, aos-bugs, eparis, gmontero, jokerman, obulatov, pbergene
Target Milestone: ---Keywords: Reopened
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Some users pulling images from Docker Hub get a `container image registry lookup failed...toomanyrequests: You have reached your pull rate limit` error. This error happens because the 'docker.io' login they used to call the 'oc new-app' does not have sufficient paid support with 'docker.io'. The resulting application is subject to image pull throttling, which can produce failures. The current release updates the 'oc new-app' help to remind users how defaulting works for the image registry and repository specs, so users can, when possible, use non-default image references to avoid similar errors.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:44:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Xavier Coulon 2021-02-15 16:31:36 UTC
Description of problem:

When too many users try to pull images from Docker Hub, some of them end up with a error message such as


container image registry lookup failed: ...:latest: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

This can also happen via the `oc new-app` command. 
This kind of problem may occur on a cluster with a lot of users, such as (but not only) Dev Sandbox

Version-Release number of selected component (if applicable): Dev Sandbox running on OSD 4.6.15

How reproducible:

Steps to Reproduce:
1. Pull too many images from Docker Hub, or run too many times the `oc new-app` command in a short amount of time 

Actual results: error message reported above

Expected results: being able to deploy

Additional info:

Comment 1 Oleg Bulatov 2021-02-15 17:20:55 UTC
This is not a problem with OpenShift Image Registry. This is a limitation of Docker Hub, see https://www.docker.com/increase-rate-limit.

Comment 2 Alexey Kazakov 2021-02-15 17:27:04 UTC
Of course it's Docker Hub limitation but shouldn't oc new-app stop using Docker Hub and start using images from quay.io or something instead?

Comment 3 Oleg Bulatov 2021-02-15 19:56:10 UTC
Can you show how you use `oc new-app`? `oc new-app` is usually provided with an image explicitly and our docs usually demonstrate how to use it with docker.io images. We have a task to update docs [1].

[1]: https://issues.redhat.com/browse/RHDEVDOCS-2465

Comment 4 Alexey Kazakov 2021-02-15 20:45:38 UTC
I guess it's matter of updating openshift templates.

For example there a few examples suggested by `oc new-app` help. And I tried a random one and it uses an image from docker.io:
oc new-app --docker-image=python

Comment 5 Oleg Bulatov 2021-02-15 22:37:50 UTC
afaik new-app belongs to Build

Comment 6 Gabe Montero 2021-02-15 22:57:00 UTC
(In reply to Alexey Kazakov from comment #4)
> I guess it's matter of updating openshift templates.
> For example there a few examples suggested by `oc new-app` help. And I tried
> a random one and it uses an image from docker.io:
> oc new-app --docker-image=python

Ahh ... yeah, if you just say "python" for a docker image, without a qualified registry, the image clients will still go to docker.io first.

For example:

 $ podman pull python
Completed short name "python" with unqualified-search registries (origin: /etc/containers/registries.conf)
Trying to pull registry.fedoraproject.org/python:latest...
  manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/python:latest...
  name unknown: Repo not found
Trying to pull registry.centos.org/python:latest...
  manifest unknown: manifest unknown
Trying to pull docker.io/library/python:latest...
Getting image source signatures

So yeah we'll take this on to clarify the `oc new-app` help ... that said, I'm looking at the latest 4.7 oc, and `oc help new-app` does NOT list 
oc new-app --docker-image=python so it may have been corrected there.

At most, we'll put a warning in the help that if you do not do registry/repo/image:tag, it will default to docker.io and you need to consider image throttling.
And if needed we'll remove examples like oc new-app --docker-image=python

What we are NOT going to do is try to parse the --docker-image flag and if the just put the image, try and substitute a quay.io reference.  That would change 
behavior.  It is still conceivable a user might want to go to docker.io

One small clarification @Alexey - when you say `oc new-app` help, you literally mean running `oc help new-app` or `oc new-app --help` from the command line, right?

Or did you see this help somehow through the console, and I need to look at what it is generating for help as well?

Comment 7 Alexey Kazakov 2021-02-15 23:35:26 UTC
Gabe thanks for clarification. The biggest concern here is probably User Experience...
We got a few questions/complaints from our users who tried to use oc new-app in our Sandbox cluster and faced that docker hub limitation.
I personally just tried to do the following:

'$ oc new-app` (since I don't remember the exact new-app syntax)
I got the following response:

error: You must specify one or more images, image streams, templates, or source code locations to create an application.

To list all local templates and image streams, use:

  oc new-app -L

To search templates, image streams, and Docker images that match the arguments provided, use:

  oc new-app -S php
  oc new-app -S --template=ruby
  oc new-app -S --image-stream=mysql
  oc new-app -S --docker-image=python

See 'oc new-app -h' for help and examples


So, next thing I tried:
`oc new-app --docker-image=python`

The app was not deployed and it's not easy for inexperienced users to figure out what is wrong.

So, anything which could help users who are trying things out to create a new app properly (better docs / help message / error (warning) messaging, etc) would be useful.

Comment 8 Gabe Montero 2021-02-15 23:44:16 UTC
Perfect - thanks Alexey.  I know what to go after now.

Yeah `oc new-app` just on its own and then `oc help new-app` produce different output.

The former's output needs some TLC.

Comment 11 wewang 2021-02-25 02:48:51 UTC
Verified in version:
[wewang@wangwen Downloads]$ ./oc version
Client Version: 4.8.0-0.nightly-2021-02-24-063313
Server Version: 4.5.0-0.nightly-2021-02-20-065540
Kubernetes Version: v1.18.3+c3b5327

[wewang@wangwen Downloads]$ ./oc new-app --help
  # Create an application based on the source code in the current git repository (with a public remote) and a Docker
  oc new-app . --docker-image=registry/repo/langimage

      --docker-image=[]: Name of a Docker image to include in the app.  Note:  not specifying a registry or repository
means defaults in place for client image pulls are employed.

To search templates, image streams, and Docker images that match the arguments provided, use:

  oc new-app -S php
  oc new-app -S --template=rails
  oc new-app -S --image-stream=mysql
  oc new-app -S --docker-image=registry.access.redhat.com/ubi8/python-38

For details on how to use the results from those searches to provide images, image streams, templates, or source code locations as inputs into 'oc new-app', use:

  oc help new-app

Comment 12 Rolfe Dlugy-Hegwer 2021-04-09 10:49:22 UTC
Supprting information for release notes:

Cause: image registry searches from 'oc new-app' via the standard clients can default registry and repository to 'docker.io' and 'library' if those elements of the images reference are not specified.

Consequence: if the user invoking 'oc new-app' does not have a sufficient level of paid support with 'docker.io', the resulting application created by 'oc new-app' can be suspect to image pull throttling that can result in failures.

Fix: Updates to 'oc new-app' help have been made to remind users to be cognizant of how defaulting works for registry and repository in image specs.

Result: Users are better informed as they try out 'oc new-app' and should encounter fewer errors.

Comment 20 errata-xmlrpc 2021-07-27 22:44:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.