Description of problem:
In INT, we have a single project that is missing its push secret, and therefore cannot push to the registry during builds. It's unknown how it got into that state. The project name is 'vdinh-project'.
However, the user *can* push directly to the registry by doing a 'docker login' and pushing to the exposed registry's FQDN.
Version-Release number of selected component (if applicable):
Unknown. It's only been seen on this project.
Steps to Reproduce:
1. Create an app using the web console (in affected project)
2. Watch build logs.
3. Build fails during push.
"No push secret provided"
This project should use a push secret, like the rest of the projects do.
# Failed push from affected project:
I0505 12:18:20.891112 1 sti.go:334] Successfully built vdinh-project/nodejs-mongodb-example-1:1eabf729
I0505 12:18:20.919841 1 cleanup.go:23] Removing temporary directory /tmp/s2i-build286764454
I0505 12:18:20.919866 1 fs.go:156] Removing directory '/tmp/s2i-build286764454'
I0505 12:18:20.934538 1 sti.go:269] No push secret provided
I0505 12:18:20.934559 1 sti.go:271] Pushing 172.30.94.234:5000/vdinh-project/nodejs-mongodb-example:latest image ...
F0505 12:18:21.025637 1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Post https://172.30.94.234:5000/v2/vdinh-project/nodejs-mongodb-example/blobs/uploads/: no basic auth credentials
# Successful push from another project:
I0505 12:17:23.789968 1 sti.go:334] Successfully built dakinitest/cakephp-example-1:f087edc0
I0505 12:17:23.818022 1 cleanup.go:23] Removing temporary directory /tmp/s2i-build211711886
I0505 12:17:23.818064 1 fs.go:156] Removing directory '/tmp/s2i-build211711886'
I0505 12:17:23.882053 1 sti.go:267] Using provided push secret for pushing 172.30.94.234:5000/dakinitest/cakephp-example:latest image
I0505 12:17:23.882090 1 sti.go:271] Pushing 172.30.94.234:5000/dakinitest/cakephp-example:latest image ...
I0505 12:19:20.772406 1 sti.go:287] Successfully pushed 172.30.94.234:5000/dakinitest/cakephp-example:latest
Possibly related. https://bugzilla.redhat.com/show_bug.cgi?id=1324179
The related BZ was moved to VERIFIED, is this still an issue?
I think this issue is resolved.
According to #Comment 5 , verified this bug.
I would like to reopen this issue as we still encounter this problem on the INT cluster.
I have met this issue several times in online too. Sometimes image would be fail to pushed after succcessfully built, but it could secceed when I just start build again with the same buildconfig. Logs are same with Stefanie described in this bug.
David can you look into it, iirc all the secrets are auto-generated, even if one removes them they should be re-created instantaneously. At least that's what I've experienced last Friday when dealing with other bug with secrets.
@bparees is this just the ordering problem between the secrets and the build getting started before they're present? If so, don't you already have cards/issues/bugs about it?
There appear to be two things being discussed in this bug. The original problem was a case of a project in which the secrets *never* got created:
"In INT, we have a single project that is missing its push secret, and therefore cannot push to the registry during builds. It's unknown how it got into that state. The project name is 'vdinh-project'."
You're correct that we have other bugs around the ordering problems of retrying when secrets don't exist "yet" but will soon.
https://bugzilla.redhat.com/show_bug.cgi?id=1333030 (in which i disagree that builds should take any special action in this scenario).
I'm assigning this back to you for the original issue of secrets never being created/recreated for the project. If you feel it's resolved, feel free to put it ON_QA.
https://github.com/openshift/ose/pull/278 resolved the original issue with push secrets not getting created
I think the issue of secrets never being created/recreated has been resolved on dev-preview-stg.
But in dev-preview-int, sometimes secrets still can't be created succussfully now, like below:
# oc get sa
NAME SECRETS AGE
builder 1 18m
default 1 18m
deployer 2 18m
# oc get secret
NAME TYPE DATA AGE
builder-token-4cghx kubernetes.io/service-account-token 3 19m
default-token-1a1h5 kubernetes.io/service-account-token 3 19m
default-token-jrebj kubernetes.io/service-account-token 3 19m
deployer-dockercfg-kp2ao kubernetes.io/dockercfg 1 19m
deployer-token-s9wcu kubernetes.io/service-account-token 3 19m
deployer-token-wefka kubernetes.io/service-account-token 3 19m
1333030 is not the right spot. 1333030 is referring to a case where we know the secret is not available and apparently some people expect the build to retry until the secret is available (I don't agree, hence why that bug has not moved forward).
but a case where the secret actually does exist, but the build doesn't find/use it should be opened as a new bug.
This one seems fixed.
The issue of "secrets not being created" has be fixed.
Move this bug to verified. Thanks!
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.