Bug 1443163 - Failed to push image error at the end of the build when creating an application from a template
Summary: Failed to push image error at the end of the build when creating an applicati...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.4.0
Hardware: Unspecified
OS: Linux
medium
low
Target Milestone: ---
: 3.7.0
Assignee: Ben Parees
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-18 16:08 UTC by Joel Rosental R.
Modified: 2020-08-13 09:04 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: ImageStream output references and their corresponding secrets were resolved during build creation time. Consequence: If the output imagestream did not exist yet, no push secret would be be computed, resulting in a build failure during push. Fix: The ImageStream output and push secret will be computed when preparing to run the build, under logic which will retry until the imagestream is available. Result: Builds which are started before the output imagestream exists will no longer fail during the push phase.
Clone Of:
Environment:
Last Closed: 2017-11-28 21:53:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3009161 None None None 2017-11-16 22:26:45 UTC
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Joel Rosental R. 2017-04-18 16:08:47 UTC
Description of problem:
While trying to deploy an application from a template if the ImageStream object appears before the BuildConfig then the builds fails with an error like this:

Pushing image 172.30.220.103:5000/bfqc8526/workshop2:latest ...
error: build error: Failed to push image: unauthorized: authentication required

However if the order is swapped and the BuildConfig object appears first the build finishes successfully.

Also when the build fails, if a new build is started through the CLI with oc start-build <bc> then it finishes successfully and through the web console further rebuilds also work, however if the new build is started from the web interface after it has failed previously from the CLI or web, then it keeps failing.


Version-Release number of selected component (if applicable):
3.4

How reproducible:
always

Steps to Reproduce:
1. Import a template where BuildConfig appears before the ImageStream object
2. Deploy an application using this template


Actual results:
Build error while pushing the resulting image: error: build error: Failed to push image: unauthorized: authentication required

Expected results:
Build finished successfully.

Additional info:

Comment 2 Ben Parees 2017-04-18 17:31:57 UTC
mostly this sounds like what is described here:
https://github.com/openshift/origin/issues/4518#issuecomment-140460107

however in that scenario, subsequent build attempts should succeed.

I would like to see level 5 build logs from the second failing build.

Comment 3 Ben Parees 2017-04-18 17:36:35 UTC
also i'm confused by this summary:

"Also when the build fails, if a new build is started through the CLI with oc start-build <bc> then it finishes successfully and through the web console further rebuilds also work, however if the new build is started from the web interface after it has failed previously from the CLI or web, then it keeps failing."


1) the build fails (when the buildconfig precedes the imagestream in the template)
2) oc start-build or starting a build via the web console results in a successful build completion

when does "after it has failed previously from the CLI or web" come into play?  I think you might be saying:

1) the initial build fails
2) starting a build via the web console - build fails
3) starting a build via CLI or web will now always fail

vs

1) the initial build fails
2) start a build via CLI - build succeeds
3) starting a build via CLI or web will now always succeed

Is that accurate?  If so it is extremely strange since the codepaths for starting builds should be identical for the CLI and webconsole.  Again, full level 5 logs for the failing builds are needed.

Comment 4 Ben Parees 2017-04-19 14:44:24 UTC
After some discussion on IRC this is the known issue i linked, the confusion was due to using the "Rebuild" button which clones the existing broken build (it's missing the pushsecret) so it continues to fail.  Starting a new build succeeds as expected.

Copying the most relevant description of the problem from the github issue:

during the generation of the build object, we try to resolve the output IST so we can decide what secret we need to use with it. the resolution fails so we don't setup any push secret, but that does not (and should not) fail the creation of the build object.

So a build object gets created with no pushsecret.

Then the build controller sees it and tries to process the build object. part of processing the build object requires resolving the output IST (so we know where to push). Since we can't do this (no IST yet), the build hangs in a retry state until the IST gets created.

Finally the IST gets created and the build proceeds, but the push secret was never set up.

The right fix for this is to resolve the output push location at the time we create the build (which means hanging the build object creation until the output destination can be resolved), but i think there are some flow challenges with doing that. Another option would be to update the push secret when we resolve the reference in the build controller, instead of when creating the build in the buildconfig controller.

either way it's a real bug, you need to create your IST before launching the build if you don't want to hit it, or you need to run a second build after the IST is created.

Comment 6 openshift-github-bot 2017-07-19 21:40:47 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/10f3a0181fefeaea3cdf34137dea5dd558fce1cf
resolve image push secret in controller, not instantiate

https://bugzilla.redhat.com/show_bug.cgi?id=1443163

bug 1443163

Comment 7 Ben Parees 2017-07-19 21:44:51 UTC
https://github.com/openshift/origin/pull/15102

Comment 8 Vladislav Walek 2017-07-21 08:39:58 UTC
Hello,
just checking, would this fix come in 3.6 or 3.5 ? The target is marked as 3.7.
Thank you

Comment 9 Ben Parees 2017-07-21 11:50:33 UTC
Neither, it's coming in 3.7  too risky a change to put in 3.6 this late and  ot critical enough of an issue.

Comment 12 wewang 2017-10-13 06:58:35 UTC
verified in openshift v3.7.0-0.143.3
kubernetes v1.7.0+80709908fd
etcd 3.2.1

steps:# 
1.  Extract buildconfig and imagestream
oc new-app https://github.com/wewang58/helloworld-v3.git -o json > helloworld-v3.json
# cat helloworld-v3.json | jq .items[0] > imageStream.json
# cat helloworld-v3.json | jq .items[1] > buildConfig.json

2.  cretae build(buildconfig) before imagestream.
# oc create -f buildConfig.json
# oc create -f imageStream.json

3. Check the image pushed successfully
[root@dhcp-128-91 .ssh]# oc get builds
NAME              TYPE      FROM          STATUS     STARTED                  DURATION
helloworld-v3-1   Source    Git@f4be318   Complete   Less than a second ago   7s
[root@dhcp-128-91 .ssh]# oc build-logs helloworld-v3-1
Command "build-logs" is deprecated, use "oc logs build/<build-name>" instead.
Cloning "https://github.com/wewang58/helloworld-v3.git" ...
	Commit:	f4be3180053297f74bdfdecb5aec27f7ac229276 (empty)
	Author:	Kenjiro Nakayama <nakayamakenjiro@gmail.com>
	Date:	Fri Sep 4 14:06:33 2015 +0900
---> Installing application source...
Pushing image docker-registry.default.svc:5000/wewang/helloworld-v3:latest ...
Pushed 1/6 layers, 21% complete
Pushed 2/6 layers, 50% complete
Pushed 3/6 layers, 67% complete
Pushed 4/6 layers, 83% complete
Pushed 5/6 layers, 83% complete
Pushed 6/6 layers, 100% complete
Push successful

Comment 16 errata-xmlrpc 2017-11-28 21:53:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.