Bug 1703262 - confusing/messy error message on build image pull failure
Summary: confusing/messy error message on build image pull failure
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.1.0
Assignee: Nalin Dahyabhai
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-25 22:41 UTC by Ben Parees
Modified: 2019-06-04 10:48 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:48:05 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:48:13 UTC

Description Ben Parees 2019-04-25 22:41:55 UTC
Description of problem:
Verbose/unhelpful error message when an image cannot be pulled.

A failure to pull an image in a build(due to missing creds) results in an error like this:

error: build error: After retrying 2 times, Pull image still failed due to error: while pulling “docker://registry.redhat.io/rhoar-nodejs/nodejs-10@sha256:cd0003f4abfa61f4e801ca498acb20120008ecd3ea7ce7bb618e06ab7b2f8b1d” as “registry.redhat.io/rhoar-nodejs/nodejs-10@sha256:cd0003f4abfa61f4e801ca498acb20120008ecd3ea7ce7bb618e06ab7b2f8b1d”: Error determining manifest MIME type for docker://registry.redhat.io/rhoar-nodejs/nodejs-10@sha256:cd0003f4abfa61f4e801ca498acb20120008ecd3ea7ce7bb618e06ab7b2f8b1d: unable to retrieve auth token: invalid username/password

There are a few problems here..some i think are coming from buildah and some from the devex bits, but i'm starting the bug with containers/buildah:

1) we're printing the image pullspec 3 times.  
2) The "error determining MIME type" is confusing/misleading from the real error
3) since we're wrapping errors inside errors we get a bunch of colons

Ideally we'd get a clear InvalidCredentialsError from buildah, which the build code could then cleanly report instead of blindly wrapping/printing it.

Comment 1 Daniel Walsh 2019-04-26 08:08:48 UTC
The problem is that we have attempted to pull from three registries, therefore three messages.   The first two indicating that the image did not exists there, and the
third with the invalid username/password.

If you run an errors.Cause(err) then you will only get the last one I believe.

Unable to retrieve auth token: invalid username/password

Comment 2 Nalin Dahyabhai 2019-04-26 13:56:33 UTC
Each instance of the image name in this error includes the same registry name, so the duplication is not a result of the search list.  If I'm reading it right, using github.com/pkg/errors.Cause() to strip away the additional background information from this one should yield a github.com/containers/image/docker.ErrUnauthorizedForCredentials error.

Comment 3 Adam Kaplan 2019-04-29 19:17:17 UTC
PR: https://github.com/openshift/builder/pull/66

Comment 4 Adam Kaplan 2019-04-29 19:18:03 UTC
@Nalin leaving it up to you to move to MODIFIED if the builder PR takes care of it.

Comment 5 Nalin Dahyabhai 2019-04-29 21:25:55 UTC
If we only get a 401 error, we now log messages that look like:

Pulling image registry.redhat.io/no-such-base-image ...
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: unable to retrieve auth token: invalid username/password

Note that if we receive multiple errors, we receive a longer message which includes the original description for the 401 error along with whatever other errors we encountered.

Comment 6 Ben Parees 2019-04-29 21:34:34 UTC
Thanks Nalin, that looks way better... Adam i think you should still track a low severity bug to further clean up the builder side of it, so we can get the error down to something like:  "error: failed to pull image: unable to retrieve auth token: invalid username/password"

Comment 8 weiwei jiang 2019-05-06 09:49:35 UTC
Checked with and this issue has been fixed.

# openshift-sti-build version 
openshift-sti-build v4.1.0-201905032232+59e5dc1-dirty

# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-05-070156   True        False         15m     Cluster version is 4.1.0-0.nightly-2019-05-05-070156

# oc get nodes -o wide 
NAME                                              STATUS   ROLES    AGE   VERSION             INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                   KERNEL-VERSION         CONTAINER-RUNTIME
ip-10-0-133-56.ap-northeast-1.compute.internal    Ready    worker   23m   v1.13.4+db7b699c3   10.0.133.56    <none>        Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa)   4.18.0-80.el8.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8
ip-10-0-137-75.ap-northeast-1.compute.internal    Ready    master   33m   v1.13.4+db7b699c3   10.0.137.75    <none>        Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa)   4.18.0-80.el8.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8
ip-10-0-149-8.ap-northeast-1.compute.internal     Ready    worker   23m   v1.13.4+db7b699c3   10.0.149.8     <none>        Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa)   4.18.0-80.el8.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8
ip-10-0-158-112.ap-northeast-1.compute.internal   Ready    master   33m   v1.13.4+db7b699c3   10.0.158.112   <none>        Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa)   4.18.0-80.el8.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8
ip-10-0-169-206.ap-northeast-1.compute.internal   Ready    worker   23m   v1.13.4+db7b699c3   10.0.169.206   <none>        Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa)   4.18.0-80.el8.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8
ip-10-0-174-121.ap-northeast-1.compute.internal   Ready    master   33m   v1.13.4+db7b699c3   10.0.174.121   <none>        Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa)   4.18.0-80.el8.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8


# oc logs -f ruby-ex-1-build
Caching blobs under "/var/cache/blobs".
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
error: build error: After retrying 2 times, Pull image still failed due to error: while pulling "docker://centos/ruby-25-centosnotexist" as "centos/ruby-25-centosnotexist": Error determining manifest MIME type for docker://centos/ruby-25-centosnotexist:latest: Error reading manifest latest in docker.io/centos/ruby-25-centosnotexist: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

Comment 10 errata-xmlrpc 2019-06-04 10:48:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.