Bug 1333510 - Project is missing a push secret
Summary: Project is missing a push secret
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: David Eads
QA Contact: Bing Li
URL:
Whiteboard:
Depends On:
Blocks: OSOPS_V3
TreeView+ depends on / blocked
 
Reported: 2016-05-05 17:09 UTC by Stefanie Forrester
Modified: 2016-09-28 00:55 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-27 09:32:17 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC

Description Stefanie Forrester 2016-05-05 17:09:23 UTC
Description of problem:

In INT, we have a single project that is missing its push secret, and therefore cannot push to the registry during builds. It's unknown how it got into that state. The project name is 'vdinh-project'.

However, the user *can* push directly to the registry by doing a 'docker login' and pushing to the exposed registry's FQDN.

Version-Release number of selected component (if applicable):


How reproducible:

Unknown. It's only been seen on this project.

Steps to Reproduce:
1. Create an app using the web console (in affected project)
2. Watch build logs.
3. Build fails during push.

Actual results:

"No push secret provided"

Expected results:

This project should use a push secret, like the rest of the projects do.

Additional info:

# Failed push from affected project:

I0505 12:18:20.891112       1 sti.go:334] Successfully built vdinh-project/nodejs-mongodb-example-1:1eabf729
I0505 12:18:20.919841       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build286764454
I0505 12:18:20.919866       1 fs.go:156] Removing directory '/tmp/s2i-build286764454'
I0505 12:18:20.934538       1 sti.go:269] No push secret provided
I0505 12:18:20.934559       1 sti.go:271] Pushing 172.30.94.234:5000/vdinh-project/nodejs-mongodb-example:latest image ...
F0505 12:18:21.025637       1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Post https://172.30.94.234:5000/v2/vdinh-project/nodejs-mongodb-example/blobs/uploads/: no basic auth credentials

# Successful push from another project:

I0505 12:17:23.789968       1 sti.go:334] Successfully built dakinitest/cakephp-example-1:f087edc0
I0505 12:17:23.818022       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build211711886
I0505 12:17:23.818064       1 fs.go:156] Removing directory '/tmp/s2i-build211711886'
I0505 12:17:23.882053       1 sti.go:267] Using provided push secret for pushing 172.30.94.234:5000/dakinitest/cakephp-example:latest image
I0505 12:17:23.882090       1 sti.go:271] Pushing 172.30.94.234:5000/dakinitest/cakephp-example:latest image ...
I0505 12:19:20.772406 1 sti.go:287] Successfully pushed 172.30.94.234:5000/dakinitest/cakephp-example:latest

Comment 3 Stefanie Forrester 2016-05-05 17:59:30 UTC
Possibly related. https://bugzilla.redhat.com/show_bug.cgi?id=1324179

Comment 4 Michal Fojtik 2016-06-20 14:54:04 UTC
The related BZ was moved to VERIFIED, is this still an issue?

Comment 5 Stefanie Forrester 2016-06-20 15:40:03 UTC
I think this issue is resolved.

Comment 6 Wei Sun 2016-06-21 02:18:43 UTC
According to #Comment 5 , verified this bug.

Comment 7 Vu Dinh 2016-07-08 19:21:56 UTC
I would like to reopen this issue as we still encounter this problem on the INT cluster.

Vu

Comment 8 Bing Li 2016-07-11 06:30:01 UTC
I have met this issue several times in online too. Sometimes image would be fail to pushed after succcessfully built, but it could secceed when I just start build again with the same buildconfig. Logs are same with Stefanie described in this bug.

Comment 9 Maciej Szulik 2016-07-11 11:42:06 UTC
David can you look into it, iirc all the secrets are auto-generated, even if one removes them they should be re-created instantaneously. At least that's what I've experienced last Friday when dealing with other bug with secrets.

Comment 10 David Eads 2016-07-11 14:00:34 UTC
@bparees is this just the ordering problem between the secrets and the build getting started before they're present?  If so, don't you already have cards/issues/bugs about it?

Comment 11 Ben Parees 2016-07-11 14:48:30 UTC
There appear to be two things being discussed in this bug.  The original problem was a case of a project in which the secrets *never* got created:

"In INT, we have a single project that is missing its push secret, and therefore cannot push to the registry during builds. It's unknown how it got into that state. The project name is 'vdinh-project'."

You're correct that we have other bugs around the ordering problems of retrying when secrets don't exist "yet" but will soon.

https://github.com/openshift/origin/issues/4518
https://bugzilla.redhat.com/show_bug.cgi?id=1333030  (in which i disagree that builds should take any special action in this scenario).


I'm assigning this back to you for the original issue of secrets never being created/recreated for the project.  If you feel it's resolved, feel free to put it ON_QA.

Comment 12 Jordan Liggitt 2016-07-11 14:57:41 UTC
https://github.com/openshift/ose/pull/278 resolved the original issue with push secrets not getting created

Comment 13 Bing Li 2016-07-12 03:08:16 UTC
I think the issue of secrets never being created/recreated has been resolved on dev-preview-stg.
But in dev-preview-int, sometimes secrets still can't be created succussfully now, like below:

# oc get sa
NAME       SECRETS   AGE
builder    1         18m
default    1         18m
deployer   2         18m
# oc get secret
NAME                       TYPE                                  DATA      AGE
builder-token-4cghx        kubernetes.io/service-account-token   3         19m
default-token-1a1h5        kubernetes.io/service-account-token   3         19m
default-token-jrebj        kubernetes.io/service-account-token   3         19m
deployer-dockercfg-kp2ao   kubernetes.io/dockercfg               1         19m
deployer-token-s9wcu       kubernetes.io/service-account-token   3         19m
deployer-token-wefka       kubernetes.io/service-account-token   3         19m

Comment 17 Ben Parees 2016-07-13 13:38:06 UTC
1333030 is not the right spot.  1333030 is referring to a case where we know the secret is not available and apparently some people expect the build to retry until the secret is available (I don't agree, hence why that bug has not moved forward).

but a case where the secret actually does exist, but the build doesn't find/use it should be opened as a new bug.

Comment 21 David Eads 2016-08-01 12:39:22 UTC
This one seems fixed.

Comment 22 Bing Li 2016-08-02 09:48:28 UTC
The issue of "secrets not being created" has be fixed.
Move this bug to verified. Thanks!

Comment 24 errata-xmlrpc 2016-09-27 09:32:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933


Note You need to log in before you can comment on or make changes to this bug.