Bug 1918879 - need better debug for bad pull secrets
Summary: need better debug for bad pull secrets
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.6.z
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.7.0
Assignee: Gabe Montero
QA Contact: XiuJuan Wang
Rolfe Dlugy-Hegwer
URL:
Whiteboard:
Depends On:
Blocks: 1919359
TreeView+ depends on / blocked
 
Reported: 2021-01-21 16:24 UTC by Gabe Montero
Modified: 2021-02-24 15:55 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, build errors caused by invalid build pull secrets, whose auth keys were not base64-encoded, did not propagate through the build stack. As a result, determining the root cause of these errors was difficult. The current release fixes this issue, so these types of build errors propagate through the build stack. Now, determining the root cause of invalid build pull secret keys is easier for users.
Clone Of:
: 1919359 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:55:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
The full build log (290.00 KB, application/x-tar)
2021-01-27 08:54 UTC, XiuJuan Wang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift builder pull 200 0 None closed Bug 1918879: better surface dockerconfigjson errors causing image pull errors 2021-02-18 01:25:37 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:55:38 UTC

Comment 1 Gabe Montero 2021-01-22 16:08:14 UTC
For testing this XiuJuan import a pull secret and then edit it so you corrupt/break the encoded value associated with the.dockerconfigjson key so we cannot json unmarshal it

the log

log.V(0).Infof("error trying to parse file %s: %s", filePath, err.Error())

should show up in the build log

and the error 

 fmt.Errorf("%s; also, error processing dockerconfigjson: %s", err.Error(), dockerConfigCredsErr.Error())

should be propagated up and ultimately visible from the log 

the bonus scenario is to do this with a build pull secret, but the registry handled by that pull secret can also be handled by the node credentials (I *think* registry.redhat.io is such a registry).

you should see the log log.V(0).Infof("error trying to parse file %s: %s", filePath, err.Error()) still, but presumably the build could still work since the pull is authenticated via the node credentials instead of the supply pull secret

Comment 3 XiuJuan Wang 2021-01-25 08:48:19 UTC
Gabe,
I create a secret which includes break line in auth encode. which failed to import imagestream(This should be a regression issue bug #1776599).

But I can't see such log from build log after create this secret.
log.V(0).Infof("error trying to parse file %s: %s", filePath, err.Error())
fmt.Errorf("%s; also, error processing dockerconfigjson: %s", err.Error(), dockerConfigCredsErr.Error())


###ruby-26-rhel7:v1 import failed after mysecret created. 
$oc tag  registry.redhat.io/rhscl/ruby-26-rhel7:latest ruby-26-rhel7:latest
$oc create secret generic mysecret --from-file=.dockerconfigjson=/tmp/kubeconfig/tbr-invalid.json  --type=kubernetes.io/dockerconfigjson
$oc tag  registry.redhat.io/rhscl/ruby-26-rhel7:latest ruby-26-rhel7:v1
$oc describe is 
Name:			ruby-26-rhel7
Namespace:		invalidsecret
Created:		39 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2021-01-25T08:27:54Z
Image Repository:	image-registry.openshift-image-registry.svc:5000/invalidsecret/ruby-26-rhel7
Image Lookup:		local=false
Unique Images:		1
Tags:			2

latest
  tagged from registry.redhat.io/rhscl/ruby-26-rhel7:latest

  * registry.redhat.io/rhscl/ruby-26-rhel7@sha256:6f9dfd3df4edc256daecdc135dc7d9fd642e1dfd6724f8c1a90af0034df77d73
      35 seconds ago

v1
  tagged from registry.redhat.io/rhscl/ruby-26-rhel7:latest

  ! error: Import failed (InternalError): Internal error occurred: registry.redhat.io/rhscl/ruby-26-rhel7:latest: illegal base64 data at input byte 1059
      5 seconds ago

$oc start-build ruby-ex --build-loglevel=8
$oc logs -f build/ruby-ex-2 | grep "error trying to pars"
$oc logs -f build/ruby-ex-2 | grep "error processing dockerconfigjson"
$oc logs -f build/ruby-ex-2 | less

I0125 08:42:30.692334       1 sti.go:231] Locating docker config paths for type PULL_DOCKERCFG_PATH
I0125 08:42:30.692362       1 sti.go:231] Getting docker config in paths : [/var/run/secrets/openshift.io/pull]
Explicitly pulling image registry.redhat.io/rhscl/ruby-26-rhel7@sha256:6f9dfd3df4edc256daecdc135dc7d9fd642e1dfd6724f8c1a90af0034df77d73
Asked to pull fresh copy of "registry.redhat.io/rhscl/ruby-26-rhel7@sha256:6f9dfd3df4edc256daecdc135dc7d9fd642e1dfd6724f8c1a90af0034df77d73".
I0125 08:42:30.692416       1 daemonless.go:165] looking for config.json at /var/run/secrets/openshift.io/pull/config.json
I0125 08:42:30.692448       1 cfg.go:154] error reading file: open /var/run/secrets/openshift.io/pull/config.json: no such file or directory
I0125 08:42:30.692459       1 daemonless.go:165] looking for .dockerconfigjson at /var/run/secrets/openshift.io/pull/.dockerconfigjson
I0125 08:42:30.692474       1 cfg.go:154] error reading file: open /var/run/secrets/openshift.io/pull/.dockerconfigjson: no such file or directory
I0125 08:42:30.692484       1 daemonless.go:165] looking for .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg
I0125 08:42:30.692588       1 daemonless.go:165] found valid .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg
I0125 08:42:30.692759       1 config.go:110] looking for .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg
I0125 08:42:30.692895       1 config.go:125] found .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg

Comment 4 Gabe Montero 2021-01-25 13:04:59 UTC
you need to do a start build on a build that tries to pull the image as its builder image, where the secret for that builder image is the corrupted secret

this bug fix is for improving error logging on the build

and in case I was not clear last time, since OCP has node level credentials for registry.redhat.io, and you use an image from there for your builder image ... doing a build with that would fall under getting
log.V(0).Infof("error trying to parse file %s: %s", filePath, err.Error()) for the bad build pull secret, but should still work since we'll find node level credentials for registry.redhat.io

to get both 

log.V(0).Infof("error trying to parse file %s: %s", filePath, err.Error())
fmt.Errorf("%s; also, error processing dockerconfigjson: %s", err.Error(), dockerConfigCredsErr.Error())


you should reference a builder image from your own private image registry that needs credentials ... and then the build pull secret for you own private registry should be bad

if you are using tests from https://bugzilla.redhat.com/show_bug.cgi?id=1776599 to construct a bad pull secret, that is OK, but otherwise, the two bugs are unrelated

hopefully that makes it more clear?

thanks

Comment 10 XiuJuan Wang 2021-01-27 08:54:45 UTC
Created attachment 1751180 [details]
The full build log

Comment 12 XiuJuan Wang 2021-01-28 02:18:06 UTC
Gabe,
I could mark this bug as verified now.
I was totally wrong to use cmd `oc secret link builder invaliadsecret --for=pull,mount` to add pull secret.

Today I used cmd `oc set build-secret --pull bc/ruby-hello-world invalidsecret28`, the pullsecret is added in bc.

        from:
          kind: ImageStreamTag
          name: myruby:v1
          namespace: xiuwang
        pullSecret:
          name: invalidsecret28

$oc get secret invalidsecret28 -o json | jq  '.data[".dockerconfigjson"]'  --raw-output   | base64 -d
{
  "auths": {
    "upshift.mirror-registry.qe.devcluster.openshift.com:5001": {
      "auth": "dummy:dummy-is-not-base64"
    }
  }  
}


Error info is from build log

I0128 02:01:20.267708       1 daemonless.go:166] looking for .dockerconfigjson at /var/run/secrets/openshift.io/pull/.dockerconfigjson
error trying to parse file /var/run/secrets/openshift.io/pull/.dockerconfigjson: illegal base64 data at input byte 5
F0128 02:01:25.348897       1 helpers.go:115] error: build error: After retrying 2 times, Pull image still failed due to error: while pulling "docker://upshift.mirror-registry.qe.devcluster.openshift.com:5001/rhscl/ruby-25-rhel7@sha256:8307f4493929f02168feda1cf178bcd1afb2036f21f9bdfb47f1e1c72bdd7d2c" as "upshift.mirror-registry.qe.devcluster.openshift.com:5001/rhscl/ruby-25-rhel7@sha256:8307f4493929f02168feda1cf178bcd1afb2036f21f9bdfb47f1e1c72bdd7d2c": Error initializing source docker://upshift.mirror-registry.qe.devcluster.openshift.com:5001/rhscl/ruby-25-rhel7@sha256:8307f4493929f02168feda1cf178bcd1afb2036f21f9bdfb47f1e1c72bdd7d2c: Error reading manifest sha256:8307f4493929f02168feda1cf178bcd1afb2036f21f9bdfb47f1e1c72bdd7d2c in upshift.mirror-registry.qe.devcluster.openshift.com:5001/rhscl/ruby-25-rhel7: unauthorized: authentication required; also, error processing dockerconfigjson: [open /var/run/secrets/openshift.io/pull/config.json: no such file or directory, illegal base64 data at input byte 5, open /var/run/secrets/openshift.io/pull/.dockercfg: no such file or directory]


ON cluster 4.7.0-0.nightly-2021-01-27-192705

Comment 18 errata-xmlrpc 2021-02-24 15:55:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.