Description of problem: Verbose/unhelpful error message when an image cannot be pulled. A failure to pull an image in a build(due to missing creds) results in an error like this: error: build error: After retrying 2 times, Pull image still failed due to error: while pulling “docker://registry.redhat.io/rhoar-nodejs/nodejs-10@sha256:cd0003f4abfa61f4e801ca498acb20120008ecd3ea7ce7bb618e06ab7b2f8b1d” as “registry.redhat.io/rhoar-nodejs/nodejs-10@sha256:cd0003f4abfa61f4e801ca498acb20120008ecd3ea7ce7bb618e06ab7b2f8b1d”: Error determining manifest MIME type for docker://registry.redhat.io/rhoar-nodejs/nodejs-10@sha256:cd0003f4abfa61f4e801ca498acb20120008ecd3ea7ce7bb618e06ab7b2f8b1d: unable to retrieve auth token: invalid username/password There are a few problems here..some i think are coming from buildah and some from the devex bits, but i'm starting the bug with containers/buildah: 1) we're printing the image pullspec 3 times. 2) The "error determining MIME type" is confusing/misleading from the real error 3) since we're wrapping errors inside errors we get a bunch of colons Ideally we'd get a clear InvalidCredentialsError from buildah, which the build code could then cleanly report instead of blindly wrapping/printing it.
The problem is that we have attempted to pull from three registries, therefore three messages. The first two indicating that the image did not exists there, and the third with the invalid username/password. If you run an errors.Cause(err) then you will only get the last one I believe. Unable to retrieve auth token: invalid username/password
Each instance of the image name in this error includes the same registry name, so the duplication is not a result of the search list. If I'm reading it right, using github.com/pkg/errors.Cause() to strip away the additional background information from this one should yield a github.com/containers/image/docker.ErrUnauthorizedForCredentials error.
PR: https://github.com/openshift/builder/pull/66
@Nalin leaving it up to you to move to MODIFIED if the builder PR takes care of it.
If we only get a 401 error, we now log messages that look like: Pulling image registry.redhat.io/no-such-base-image ... Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: unable to retrieve auth token: invalid username/password Note that if we receive multiple errors, we receive a longer message which includes the original description for the 401 error along with whatever other errors we encountered.
Thanks Nalin, that looks way better... Adam i think you should still track a low severity bug to further clean up the builder side of it, so we can get the error down to something like: "error: failed to pull image: unable to retrieve auth token: invalid username/password"
Checked with and this issue has been fixed. # openshift-sti-build version openshift-sti-build v4.1.0-201905032232+59e5dc1-dirty # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-0.nightly-2019-05-05-070156 True False 15m Cluster version is 4.1.0-0.nightly-2019-05-05-070156 # oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-133-56.ap-northeast-1.compute.internal Ready worker 23m v1.13.4+db7b699c3 10.0.133.56 <none> Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa) 4.18.0-80.el8.x86_64 cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8 ip-10-0-137-75.ap-northeast-1.compute.internal Ready master 33m v1.13.4+db7b699c3 10.0.137.75 <none> Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa) 4.18.0-80.el8.x86_64 cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8 ip-10-0-149-8.ap-northeast-1.compute.internal Ready worker 23m v1.13.4+db7b699c3 10.0.149.8 <none> Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa) 4.18.0-80.el8.x86_64 cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8 ip-10-0-158-112.ap-northeast-1.compute.internal Ready master 33m v1.13.4+db7b699c3 10.0.158.112 <none> Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa) 4.18.0-80.el8.x86_64 cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8 ip-10-0-169-206.ap-northeast-1.compute.internal Ready worker 23m v1.13.4+db7b699c3 10.0.169.206 <none> Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa) 4.18.0-80.el8.x86_64 cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8 ip-10-0-174-121.ap-northeast-1.compute.internal Ready master 33m v1.13.4+db7b699c3 10.0.174.121 <none> Red Hat Enterprise Linux CoreOS 410.8.20190505.0 (Ootpa) 4.18.0-80.el8.x86_64 cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8 # oc logs -f ruby-ex-1-build Caching blobs under "/var/cache/blobs". Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... error: build error: After retrying 2 times, Pull image still failed due to error: while pulling "docker://centos/ruby-25-centosnotexist" as "centos/ruby-25-centosnotexist": Error determining manifest MIME type for docker://centos/ruby-25-centosnotexist:latest: Error reading manifest latest in docker.io/centos/ruby-25-centosnotexist: errors: denied: requested access to the resource is denied unauthorized: authentication required
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758