Description of problem:
While trying to deploy an application from a template if the ImageStream object appears before the BuildConfig then the builds fails with an error like this:
Pushing image 172.30.220.103:5000/bfqc8526/workshop2:latest ...
error: build error: Failed to push image: unauthorized: authentication required
However if the order is swapped and the BuildConfig object appears first the build finishes successfully.
Also when the build fails, if a new build is started through the CLI with oc start-build <bc> then it finishes successfully and through the web console further rebuilds also work, however if the new build is started from the web interface after it has failed previously from the CLI or web, then it keeps failing.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Import a template where BuildConfig appears before the ImageStream object
2. Deploy an application using this template
Build error while pushing the resulting image: error: build error: Failed to push image: unauthorized: authentication required
Build finished successfully.
mostly this sounds like what is described here:
however in that scenario, subsequent build attempts should succeed.
I would like to see level 5 build logs from the second failing build.
also i'm confused by this summary:
"Also when the build fails, if a new build is started through the CLI with oc start-build <bc> then it finishes successfully and through the web console further rebuilds also work, however if the new build is started from the web interface after it has failed previously from the CLI or web, then it keeps failing."
1) the build fails (when the buildconfig precedes the imagestream in the template)
2) oc start-build or starting a build via the web console results in a successful build completion
when does "after it has failed previously from the CLI or web" come into play? I think you might be saying:
1) the initial build fails
2) starting a build via the web console - build fails
3) starting a build via CLI or web will now always fail
1) the initial build fails
2) start a build via CLI - build succeeds
3) starting a build via CLI or web will now always succeed
Is that accurate? If so it is extremely strange since the codepaths for starting builds should be identical for the CLI and webconsole. Again, full level 5 logs for the failing builds are needed.
After some discussion on IRC this is the known issue i linked, the confusion was due to using the "Rebuild" button which clones the existing broken build (it's missing the pushsecret) so it continues to fail. Starting a new build succeeds as expected.
Copying the most relevant description of the problem from the github issue:
during the generation of the build object, we try to resolve the output IST so we can decide what secret we need to use with it. the resolution fails so we don't setup any push secret, but that does not (and should not) fail the creation of the build object.
So a build object gets created with no pushsecret.
Then the build controller sees it and tries to process the build object. part of processing the build object requires resolving the output IST (so we know where to push). Since we can't do this (no IST yet), the build hangs in a retry state until the IST gets created.
Finally the IST gets created and the build proceeds, but the push secret was never set up.
The right fix for this is to resolve the output push location at the time we create the build (which means hanging the build object creation until the output destination can be resolved), but i think there are some flow challenges with doing that. Another option would be to update the push secret when we resolve the reference in the build controller, instead of when creating the build in the buildconfig controller.
either way it's a real bug, you need to create your IST before launching the build if you don't want to hit it, or you need to run a second build after the IST is created.
Commit pushed to master at https://github.com/openshift/origin
resolve image push secret in controller, not instantiate
just checking, would this fix come in 3.6 or 3.5 ? The target is marked as 3.7.
Neither, it's coming in 3.7 too risky a change to put in 3.6 this late and ot critical enough of an issue.
verified in openshift v3.7.0-0.143.3
1. Extract buildconfig and imagestream
oc new-app https://github.com/wewang58/helloworld-v3.git -o json > helloworld-v3.json
# cat helloworld-v3.json | jq .items > imageStream.json
# cat helloworld-v3.json | jq .items > buildConfig.json
2. cretae build(buildconfig) before imagestream.
# oc create -f buildConfig.json
# oc create -f imageStream.json
3. Check the image pushed successfully
[root@dhcp-128-91 .ssh]# oc get builds
NAME TYPE FROM STATUS STARTED DURATION
helloworld-v3-1 Source Git@f4be318 Complete Less than a second ago 7s
[root@dhcp-128-91 .ssh]# oc build-logs helloworld-v3-1
Command "build-logs" is deprecated, use "oc logs build/<build-name>" instead.
Cloning "https://github.com/wewang58/helloworld-v3.git" ...
Commit: f4be3180053297f74bdfdecb5aec27f7ac229276 (empty)
Author: Kenjiro Nakayama <nakayamakenjiro>
Date: Fri Sep 4 14:06:33 2015 +0900
---> Installing application source...
Pushing image docker-registry.default.svc:5000/wewang/helloworld-v3:latest ...
Pushed 1/6 layers, 21% complete
Pushed 2/6 layers, 50% complete
Pushed 3/6 layers, 67% complete
Pushed 4/6 layers, 83% complete
Pushed 5/6 layers, 83% complete
Pushed 6/6 layers, 100% complete
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.