Bug 1333510

Summary: Project is missing a push secret
Product: OpenShift Container Platform Reporter: Stefanie Forrester <dakini>
Component: apiserver-authAssignee: David Eads <deads>
Status: CLOSED ERRATA QA Contact: Bing Li <bingli>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.2.0CC: aos-bugs, bingli, bparees, dakini, deads, haowang, jliggitt, maszulik, tdawson, vdinh
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-27 09:32:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1303130    

Description Stefanie Forrester 2016-05-05 17:09:23 UTC
Description of problem:

In INT, we have a single project that is missing its push secret, and therefore cannot push to the registry during builds. It's unknown how it got into that state. The project name is 'vdinh-project'.

However, the user *can* push directly to the registry by doing a 'docker login' and pushing to the exposed registry's FQDN.

Version-Release number of selected component (if applicable):


How reproducible:

Unknown. It's only been seen on this project.

Steps to Reproduce:
1. Create an app using the web console (in affected project)
2. Watch build logs.
3. Build fails during push.

Actual results:

"No push secret provided"

Expected results:

This project should use a push secret, like the rest of the projects do.

Additional info:

# Failed push from affected project:

I0505 12:18:20.891112       1 sti.go:334] Successfully built vdinh-project/nodejs-mongodb-example-1:1eabf729
I0505 12:18:20.919841       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build286764454
I0505 12:18:20.919866       1 fs.go:156] Removing directory '/tmp/s2i-build286764454'
I0505 12:18:20.934538       1 sti.go:269] No push secret provided
I0505 12:18:20.934559       1 sti.go:271] Pushing 172.30.94.234:5000/vdinh-project/nodejs-mongodb-example:latest image ...
F0505 12:18:21.025637       1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Post https://172.30.94.234:5000/v2/vdinh-project/nodejs-mongodb-example/blobs/uploads/: no basic auth credentials

# Successful push from another project:

I0505 12:17:23.789968       1 sti.go:334] Successfully built dakinitest/cakephp-example-1:f087edc0
I0505 12:17:23.818022       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build211711886
I0505 12:17:23.818064       1 fs.go:156] Removing directory '/tmp/s2i-build211711886'
I0505 12:17:23.882053       1 sti.go:267] Using provided push secret for pushing 172.30.94.234:5000/dakinitest/cakephp-example:latest image
I0505 12:17:23.882090       1 sti.go:271] Pushing 172.30.94.234:5000/dakinitest/cakephp-example:latest image ...
I0505 12:19:20.772406 1 sti.go:287] Successfully pushed 172.30.94.234:5000/dakinitest/cakephp-example:latest

Comment 3 Stefanie Forrester 2016-05-05 17:59:30 UTC
Possibly related. https://bugzilla.redhat.com/show_bug.cgi?id=1324179

Comment 4 Michal Fojtik 2016-06-20 14:54:04 UTC
The related BZ was moved to VERIFIED, is this still an issue?

Comment 5 Stefanie Forrester 2016-06-20 15:40:03 UTC
I think this issue is resolved.

Comment 6 Wei Sun 2016-06-21 02:18:43 UTC
According to #Comment 5 , verified this bug.

Comment 7 Vu Dinh 2016-07-08 19:21:56 UTC
I would like to reopen this issue as we still encounter this problem on the INT cluster.

Vu

Comment 8 Bing Li 2016-07-11 06:30:01 UTC
I have met this issue several times in online too. Sometimes image would be fail to pushed after succcessfully built, but it could secceed when I just start build again with the same buildconfig. Logs are same with Stefanie described in this bug.

Comment 9 Maciej Szulik 2016-07-11 11:42:06 UTC
David can you look into it, iirc all the secrets are auto-generated, even if one removes them they should be re-created instantaneously. At least that's what I've experienced last Friday when dealing with other bug with secrets.

Comment 10 David Eads 2016-07-11 14:00:34 UTC
@bparees is this just the ordering problem between the secrets and the build getting started before they're present?  If so, don't you already have cards/issues/bugs about it?

Comment 11 Ben Parees 2016-07-11 14:48:30 UTC
There appear to be two things being discussed in this bug.  The original problem was a case of a project in which the secrets *never* got created:

"In INT, we have a single project that is missing its push secret, and therefore cannot push to the registry during builds. It's unknown how it got into that state. The project name is 'vdinh-project'."

You're correct that we have other bugs around the ordering problems of retrying when secrets don't exist "yet" but will soon.

https://github.com/openshift/origin/issues/4518
https://bugzilla.redhat.com/show_bug.cgi?id=1333030  (in which i disagree that builds should take any special action in this scenario).


I'm assigning this back to you for the original issue of secrets never being created/recreated for the project.  If you feel it's resolved, feel free to put it ON_QA.

Comment 12 Jordan Liggitt 2016-07-11 14:57:41 UTC
https://github.com/openshift/ose/pull/278 resolved the original issue with push secrets not getting created

Comment 13 Bing Li 2016-07-12 03:08:16 UTC
I think the issue of secrets never being created/recreated has been resolved on dev-preview-stg.
But in dev-preview-int, sometimes secrets still can't be created succussfully now, like below:

# oc get sa
NAME       SECRETS   AGE
builder    1         18m
default    1         18m
deployer   2         18m
# oc get secret
NAME                       TYPE                                  DATA      AGE
builder-token-4cghx        kubernetes.io/service-account-token   3         19m
default-token-1a1h5        kubernetes.io/service-account-token   3         19m
default-token-jrebj        kubernetes.io/service-account-token   3         19m
deployer-dockercfg-kp2ao   kubernetes.io/dockercfg               1         19m
deployer-token-s9wcu       kubernetes.io/service-account-token   3         19m
deployer-token-wefka       kubernetes.io/service-account-token   3         19m

Comment 17 Ben Parees 2016-07-13 13:38:06 UTC
1333030 is not the right spot.  1333030 is referring to a case where we know the secret is not available and apparently some people expect the build to retry until the secret is available (I don't agree, hence why that bug has not moved forward).

but a case where the secret actually does exist, but the build doesn't find/use it should be opened as a new bug.

Comment 21 David Eads 2016-08-01 12:39:22 UTC
This one seems fixed.

Comment 22 Bing Li 2016-08-02 09:48:28 UTC
The issue of "secrets not being created" has be fixed.
Move this bug to verified. Thanks!

Comment 24 errata-xmlrpc 2016-09-27 09:32:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933