Bug 1506175

Summary: Should not meet "lookup failed" and "incorrect username or password" when new-app with public image in project having fake docker secret
Product: OpenShift Container Platform Reporter: Xingxing Xia <xxia>
Component: BuildAssignee: Mangirdas Judeikis <mjudeiki>
Status: CLOSED CURRENTRELEASE QA Contact: Wenjing Zheng <wzheng>
Severity: low Docs Contact:
Priority: medium    
Version: 3.7.0CC: aos-bugs, bparees, jokerman, mfojtik, mmccomas, smunilla, spadgett
Target Milestone: ---   
Target Release: 3.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v3.9.0-0.53.0 Doc Type: Bug Fix
Doc Text:
Cause: Secret with wrong password causes pull fail for all images. Consequence: Any public image from same registry pull will fail. Fix: Added retry logic for 401 error when the password is wrong. Result: if the image is public, it will get pulled and the wrong secret will be ignored.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-18 18:16:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xingxing Xia 2017-10-25 10:29:01 UTC
Description of problem:
Met "lookup failed" and "incorrect username or password" when new-app

Test against 3.7 OCP, free-stg, the command can succeed and don't have the problem

Version-Release number of selected component (if applicable):
v3.7.0-0.176.0 (online version 3.6.0.38)

How reproducible:
Always

Steps to Reproduce:
1. Login with user who has a flaky project xxia21-proj (created on 20171016)

2.  Then run in the flaky project
$ oc new-app openshift/hello-openshift
W1025 15:03:32.386759   12162 dockerimagelookup.go:220] Docker registry lookup failed: Get https://registry-1.docker.io/v2/openshift/hello-openshift/manifests/latest: unauthorized: incorrect username or password
...

3. In the project, try to deploy openshift/hello-openshift in web console "Deploy Image" page
And try any other image, like centos/mysql-57-centos7:latest

4. Login with user who has no project and create project, then repeat above

Actual results:
2 & 3: CLI/web both show the same error for both of the example images
4. Succeed

Expected results:
2 & 3: should succeed like step 4

Additional info:
Doubt old projects have the issue, but another guy tried another project created about 12 days ago and didn't reproduce. So this project xxia21-proj is flaky

Comment 2 Xingxing Xia 2017-11-07 09:49:48 UTC
Could anybody help check? The issue still exists in today's free-int v3.7.0-0.191.0 (online version 3.6.0.38)

I did not do any operation that could be destructive to the project. Thus it is flaky to have the issue.

Not sure which Component it is close to, maybe Build/Upgrade? CC'ing Ben and Scott :)

Comment 3 Ben Parees 2017-11-07 14:58:44 UTC
It sounds to me like the broken project has a bad docker secret in it.  What docker secrets exist in the broken project?

Comment 4 Xingxing Xia 2017-11-08 03:00:30 UTC
Ah, thank you, you're right
$ oc get secret
mysecret                   kubernetes.io/dockercfg               1         21d
$ oc extract secret/mysecret
.dockercfg
$ cat .dockercfg
{"docker.io":{"username":"xxia","password":"abcd","email":"xxia","auth":"eHhpYTphYmNk"}}

After removing the secret, `oc new-app openshift/hello-openshift` works. It happens in OCP too (both v3.6.173.0.62 and v3.7.0-0.191.0 are checked)

The secret (user/password are fake) was created during case test. I didn't imagine it affects `oc new-app` for _public_ image. IMO tt is bug, the reasons are:
a) `oc run testdc --image=openshift/hello-openshift` is not affected
b) DC yaml has array spec.template.spec.imagePullSecrets, put above secret in DC that uses _public_ image, the DC deployment is not affected:
$ vi mydc.yaml
apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    run: mydc
  name: mydc
spec:
  replicas: 1
  selector:
    run: mydc
  template:
    metadata:
      labels:
        run: mydc
    spec:
      containers:
      - image: openshift/hello-openshift
        name: mydc
      imagePullSecrets:
      - name: mysecret # above fake secret
$ oc create -f mydc.yaml
$ oc get pod
NAME           READY     STATUS    RESTARTS   AGE
mydc-1-hv953   1/1       Running   0          8m

Thus, `oc new-app` with public image should be not affected by the fake secret

Comment 5 Ben Parees 2017-11-08 03:12:55 UTC
Agreed, if the image is public, new-app should be able to access it despite the bad secret.  Severity is low though since it is still a bad secret and the fix is to delete the secret or correct it.

Comment 6 Xingxing Xia 2017-11-08 07:14:12 UTC
Given the cause in comment 4, in "Description" step 3, web console shows same error. Not sure if your fix would automatically solve the error in web. If not, web separate fix may be needed, CC'ing Samuel fyi :)

Comment 7 Ben Parees 2018-01-10 21:41:20 UTC
https://github.com/openshift/origin/pull/18012

Comment 8 Mangirdas Judeikis 2018-02-26 11:34:37 UTC
Do we need QA for this one? Automated tests was included,

Comment 9 Ben Parees 2018-02-26 16:34:21 UTC
> Do we need QA for this one? Automated tests was included,

yes, all bugs that resulted in a code fix go back through QE so they can verify the fix+update their test cases if necessary.

Comment 10 Xingxing Xia 2018-02-27 03:34:26 UTC
Verified in v3.9.0-0.53.0
$ cat .dockercfg
{"docker.io":{"username":"xxia","password":"abcd","email":"xxia","auth":"eHhpYTphYmNk"}}
$ oc secrets new mysecret .dockercfg=.dockercfg
$ oc get secret mysecret
mysecret                   kubernetes.io/dockercfg               1         2s

$ oc new-app openshift/hello-openshift
--> Found Docker image b94da9e (4 hours old) from Docker Hub for "openshift/hello-openshift"
...


Web console "Deploy Image" page also has no the reported error now

Comment 11 Mangirdas Judeikis 2018-03-14 10:44:39 UTC
Is something missing from my side on this?

Comment 12 Ben Parees 2018-03-14 13:48:41 UTC
If you're getting emails about it, it's probably to fill in the doc text.

Choose the doc type in the upper right (probably "Bug Fix") and then fill in the template for the doc text in the next field.