Bug 1506175 - Should not meet "lookup failed" and "incorrect username or password" when new-app with public image in project having fake docker secret
Summary: Should not meet "lookup failed" and "incorrect username or password" when new...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 3.9.0
Assignee: Mangirdas Judeikis
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-25 10:29 UTC by Xingxing Xia
Modified: 2018-06-18 18:16 UTC (History)
7 users (show)

Fixed In Version: v3.9.0-0.53.0
Doc Type: Bug Fix
Doc Text:
Cause: Secret with wrong password causes pull fail for all images. Consequence: Any public image from same registry pull will fail. Fix: Added retry logic for 401 error when the password is wrong. Result: if the image is public, it will get pulled and the wrong secret will be ignored.
Clone Of:
Environment:
Last Closed: 2018-06-18 18:16:53 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2013 normal SHIPPED_LIVE Important: OpenShift Container Platform 3.9 security, bug fix, and enhancement update 2018-06-27 22:01:43 UTC

Description Xingxing Xia 2017-10-25 10:29:01 UTC
Description of problem:
Met "lookup failed" and "incorrect username or password" when new-app

Test against 3.7 OCP, free-stg, the command can succeed and don't have the problem

Version-Release number of selected component (if applicable):
v3.7.0-0.176.0 (online version 3.6.0.38)

How reproducible:
Always

Steps to Reproduce:
1. Login with user who has a flaky project xxia21-proj (created on 20171016)

2.  Then run in the flaky project
$ oc new-app openshift/hello-openshift
W1025 15:03:32.386759   12162 dockerimagelookup.go:220] Docker registry lookup failed: Get https://registry-1.docker.io/v2/openshift/hello-openshift/manifests/latest: unauthorized: incorrect username or password
...

3. In the project, try to deploy openshift/hello-openshift in web console "Deploy Image" page
And try any other image, like centos/mysql-57-centos7:latest

4. Login with user who has no project and create project, then repeat above

Actual results:
2 & 3: CLI/web both show the same error for both of the example images
4. Succeed

Expected results:
2 & 3: should succeed like step 4

Additional info:
Doubt old projects have the issue, but another guy tried another project created about 12 days ago and didn't reproduce. So this project xxia21-proj is flaky

Comment 2 Xingxing Xia 2017-11-07 09:49:48 UTC
Could anybody help check? The issue still exists in today's free-int v3.7.0-0.191.0 (online version 3.6.0.38)

I did not do any operation that could be destructive to the project. Thus it is flaky to have the issue.

Not sure which Component it is close to, maybe Build/Upgrade? CC'ing Ben and Scott :)

Comment 3 Ben Parees 2017-11-07 14:58:44 UTC
It sounds to me like the broken project has a bad docker secret in it.  What docker secrets exist in the broken project?

Comment 4 Xingxing Xia 2017-11-08 03:00:30 UTC
Ah, thank you, you're right
$ oc get secret
mysecret                   kubernetes.io/dockercfg               1         21d
$ oc extract secret/mysecret
.dockercfg
$ cat .dockercfg
{"docker.io":{"username":"xxia","password":"abcd","email":"xxia@gmail.com","auth":"eHhpYTphYmNk"}}

After removing the secret, `oc new-app openshift/hello-openshift` works. It happens in OCP too (both v3.6.173.0.62 and v3.7.0-0.191.0 are checked)

The secret (user/password are fake) was created during case test. I didn't imagine it affects `oc new-app` for _public_ image. IMO tt is bug, the reasons are:
a) `oc run testdc --image=openshift/hello-openshift` is not affected
b) DC yaml has array spec.template.spec.imagePullSecrets, put above secret in DC that uses _public_ image, the DC deployment is not affected:
$ vi mydc.yaml
apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    run: mydc
  name: mydc
spec:
  replicas: 1
  selector:
    run: mydc
  template:
    metadata:
      labels:
        run: mydc
    spec:
      containers:
      - image: openshift/hello-openshift
        name: mydc
      imagePullSecrets:
      - name: mysecret # above fake secret
$ oc create -f mydc.yaml
$ oc get pod
NAME           READY     STATUS    RESTARTS   AGE
mydc-1-hv953   1/1       Running   0          8m

Thus, `oc new-app` with public image should be not affected by the fake secret

Comment 5 Ben Parees 2017-11-08 03:12:55 UTC
Agreed, if the image is public, new-app should be able to access it despite the bad secret.  Severity is low though since it is still a bad secret and the fix is to delete the secret or correct it.

Comment 6 Xingxing Xia 2017-11-08 07:14:12 UTC
Given the cause in comment 4, in "Description" step 3, web console shows same error. Not sure if your fix would automatically solve the error in web. If not, web separate fix may be needed, CC'ing Samuel fyi :)

Comment 7 Ben Parees 2018-01-10 21:41:20 UTC
https://github.com/openshift/origin/pull/18012

Comment 8 Mangirdas Judeikis 2018-02-26 11:34:37 UTC
Do we need QA for this one? Automated tests was included,

Comment 9 Ben Parees 2018-02-26 16:34:21 UTC
> Do we need QA for this one? Automated tests was included,

yes, all bugs that resulted in a code fix go back through QE so they can verify the fix+update their test cases if necessary.

Comment 10 Xingxing Xia 2018-02-27 03:34:26 UTC
Verified in v3.9.0-0.53.0
$ cat .dockercfg
{"docker.io":{"username":"xxia","password":"abcd","email":"xxia@gmail.com","auth":"eHhpYTphYmNk"}}
$ oc secrets new mysecret .dockercfg=.dockercfg
$ oc get secret mysecret
mysecret                   kubernetes.io/dockercfg               1         2s

$ oc new-app openshift/hello-openshift
--> Found Docker image b94da9e (4 hours old) from Docker Hub for "openshift/hello-openshift"
...


Web console "Deploy Image" page also has no the reported error now

Comment 11 Mangirdas Judeikis 2018-03-14 10:44:39 UTC
Is something missing from my side on this?

Comment 12 Ben Parees 2018-03-14 13:48:41 UTC
If you're getting emails about it, it's probably to fill in the doc text.

Choose the doc type in the upper right (probably "Bug Fix") and then fill in the template for the doc text in the next field.


Note You need to log in before you can comment on or make changes to this bug.