Bug 1398330 - oc new-app always returns "Internal error occurred" when it failed to find imageStream in disconnected environment
Summary: oc new-app always returns "Internal error occurred" when it failed to find im...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Martin Milata
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-24 13:17 UTC by Kenjiro Nakayama
Modified: 2021-09-09 12:00 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Master returns internal server error HTTP code when docker image lookup fails due to unreachable registry. This happens for every image lookup in disconnected OCP environments. Consequence: oc new-app reports the internal server error as a warning to user, which can make them think there is something wrong with their OCP deployment. Fix: Change the wording of the error oc new-app prints to not include the string "internal server error". Result: The warning that is printed doesn't sound more severe than it is.
Clone Of:
Environment:
Last Closed: 2017-04-12 19:17:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1378647 0 medium CLOSED Failed to "oc new-app --search" at the offline environment disconnected to the Internet 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1404957 0 high CLOSED [RFE] Configurable default docker registry used by oc new-app 2021-09-09 12:02:42 UTC
Red Hat Product Errata RHBA-2017:0884 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.5 RPM Release Advisory 2017-04-12 22:50:07 UTC

Internal Links: 1378647 1404957

Description Kenjiro Nakayama 2016-11-24 13:17:04 UTC
Description of problem:
===
- oc new-app always returns "Internal error occurred" when it failed to find imageStream in disconnected environment
- For example, when users run oc new-app to build Java code, it returns this error:

~~~
  oc new-app <ssh git repo> --loglevel==4 returns
  helpers.go:110 error can't look up Docker image "jee": Internal error occurred: get https://registry-1.docker.io/vt: dial tcp: lookup registry-1.docker.io on <IP> no such host
  error: no match for "jee"
~~~

- We know that "# oc new-app <SSH_GIT> --image-stream=jboss-eap70-openshift" could work. But our problem is that the error message says "Internal error occurred" and "lookup registry-1.docker.io".
- These messages mislead the users, so they try to research the internal error and why oc command tried to lookup docker.io in disconnected env.

Version-Release number of selected component (if applicable):
===
- OCP 3.3

How reproducible:
===
Steps to Reproduce:
1. Deploy OpenShift with disconnected environment
2. oc new-app <GIT_CODE> (e.g. oc new-app <JAVA_CODE>)

Actual results:
===
- If imageStream was not found, it returns above error.

Expected results:
===
- The error message should not say "Internal error occurred". And oc command should not lookup docker.io with some configuration in disconnected environment.

Comment 3 Ben Parees 2016-11-25 17:01:09 UTC
Martin I believe you recently did some other fixes around handling new-app lookup operations when dockerhub is not available, can you take a look at this one?  Thanks.

Comment 5 Martin Milata 2016-11-29 13:19:28 UTC
This was fixed in #1378647. The lookup failure is still printed as a warning but if the image can be found in other places new-app looks then the command succeeds.

The fix should be included in OCP 3.4 and is on its way to 3.2.z (bug #1388522) and 3.3.z (bug #1388524).

OK to close as duplicate?

Comment 6 Ben Parees 2016-11-29 15:07:26 UTC
Yup.  Thanks Martin!

*** This bug has been marked as a duplicate of bug 1378647 ***

Comment 8 Martin Milata 2016-12-01 15:44:03 UTC
The warning is printed in default loglevel - in online env it's a legitimate problem the user should probably be aware of (and afaik there's on easy way to determine we're running in offline env).

I wasn't able to find references to registry-1.docker.io in openshift codebase - I suspect it's a docker default when registry is not explicitly given?

Comment 9 Kenjiro Nakayama 2016-12-01 16:39:00 UTC
registry-1.docker.io is hardcorded here: 
https://github.com/openshift/origin/blob/v1.3.0/pkg/image/api/helper.go#L23-L36

Comment 10 Kenjiro Nakayama 2016-12-02 00:44:30 UTC
> The warning is printed in default loglevel - in online env it's a legitimate problem the user should probably be aware of (and afaik there's on easy way to determine we're running in offline env).

OK, so the fix in bz#1378647 has not been implemented as the customer expected. It still outputs "Internal error occurred" and the message is produced whenever imageStream was not found.

I'm sorry, but this fix request is not to determine we're running in offline env, but to avoid misleading the users.

Comment 11 Martin Milata 2016-12-12 16:07:27 UTC
What's the problem? Wording of the warning or that the warning is printed at all? Something else?

Comment 13 Martin Milata 2016-12-13 17:38:38 UTC
My understanding was that the core issue here has been "oc new-app <ssh git repo>" failing regardless of whether the imagestream existed or not - that's what I've been able to reproduce with OCP 3.3. Fixing bug #1378647 resulted in the command succeeding with a warning when the imagestream was found.

Agreed that "Internal error occurred" doesn't exactly inspire confidence in the user. It shouldn't be difficult to change the wording to not imply that there's a OpenShift programming error. Note that the error originates from the server which performs the docker registry lookup.

I don't think we can get rid of the warning entirely - without the server knowing whether it should be able to connect to the host there's no way to determine whether it's actual error or not.

Regarding making the default docker registry configurable, I think that is fair amount of work and would suggest opening separate RFE BZ.

Comment 14 Kenjiro Nakayama 2016-12-14 08:29:23 UTC
> Regarding making the default docker registry configurable, I think that is fair amount of work and would suggest opening separate RFE BZ.

I think there are two approach that:

  a) Replaceable with other registry like quay.io
  b) Disable to use(look up) docker.io by default

Would you like me to file one RFE or separated two RFEs? Ideal solution is a), but if b) could be done sooner by using environmental value or something, I believe that the customer use it.

Comment 15 Ben Parees 2016-12-14 14:02:22 UTC
one RFE for both of those requests is fine.

@martin in the meantime if you can make the current warning a little more user friendly that would let us close this existing bug.

Comment 16 Martin Milata 2016-12-14 22:41:57 UTC
PR: https://github.com/openshift/origin/pull/12269

Changes the warning from:

  Docker registry lookup failed: Internal error occurred: get https://registry-1.docker.io/vt: dial tcp: lookup registry-1.docker.io on <IP> no such host

To:

  Docker registry lookup failed: get https://registry-1.docker.io/vt: dial tcp: lookup registry-1.docker.io on <IP> no such host

Comment 17 Ben Parees 2016-12-14 22:45:10 UTC
Kenjiro is that output acceptable (pending the RFE)?

Comment 18 Kenjiro Nakayama 2016-12-15 08:39:51 UTC
Yes, that's fine. Thank you.

Comment 19 Kenjiro Nakayama 2016-12-15 08:42:07 UTC
As for making the default docker registry configurable, I created bz#1404957

Comment 20 openshift-github-bot 2016-12-16 14:18:44 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/fa63dcb76d0352aeb1182c1fcf9d33213d654c6e
Don't mention internal error when docker registry is unreachable

Should make oc new-app output less scary when used in disconnected
environment.

Before:

$ oc new-app somegitrepo
...dockerimagelookup.go:220] Docker registry lookup failed: Internal error occurred: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

After:

$ oc new-app somegitrepo
...dockerimagelookup.go:220] Docker registry lookup failed: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Fixes bug 1398330.

Comment 21 Troy Dawson 2017-01-20 23:30:03 UTC
This has been merged into ocp and is in OCP v3.5.0.7 or newer.

Comment 22 Dongbo Yan 2017-01-22 06:25:14 UTC
Verified

openshift v3.5.0.7+390ef18
kubernetes v1.5.2+43a9be4
etcd 3.1.0-rc.0

# oc new-app --search mariadb
W0122 01:23:05.669604   54906 dockerimagelookup.go:220] Docker registry lookup failed: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Templates (oc new-app --template=<template>)

Comment 24 errata-xmlrpc 2017-04-12 19:17:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0884


Note You need to log in before you can comment on or make changes to this bug.