Bug 1393549

Summary: Infinite build loop
Product: OpenShift Online Reporter: Miloslav Nenadal <nenadalm>
Component: BuildAssignee: Gabe Montero <gmontero>
Status: CLOSED CURRENTRELEASE QA Contact: Wang Haoran <haowang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.xCC: aos-bugs, bparees, cewong, gmontero, nenadalm, shiywang, tdawson, xiuwang, yufchang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-01 20:40:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Miloslav Nenadal 2016-11-09 19:54:51 UTC
Description of problem:
I have issue that new builds keep creating without me doing anything (once one build finishes, new is started). This process continues until I cancel it.

Url with builds (doesn't work as the project was removed - is here just to make sure that people understand where I see the issue): https://console.preview.openshift.com/console/project/warehouse/browse/builds

Steps to Reproduce:
delete current project warehouse
$ oc new-project warehouse
$ oc new-app nenadalm/warehouse~https://github.com/nenadalm/Warehouse.git
wait until build fails (due to memory)
$ oc edit buildconfigs
change 'specs.resources = {}' to 'specs.resources.limits.memory = "1Gi"'
$ oc start-build warehouse 

Actual results:
1) build is created
2) build successfully finishes
3) go to step 1


Expected results:
1) build is created
2) build successfully finishes

Comment 1 Cesar Wong 2016-11-09 19:59:36 UTC
Miloslav, can you include the output of 'oc version' ? 

Also, if instead of:
oc new-app nenadalm/warehouse~https://github.com/nenadalm/Warehouse.git

You run:
oc new-app nenadalm/warehouse~https://github.com/nenadalm/Warehouse.git --name=warehouse2

Does that resolve the issue? It seems that the same ImageStream is getting used for input and output. So a new tag will kick off a build which will create a new tag and so on.

Comment 2 Miloslav Nenadal 2016-11-09 20:34:52 UTC
Thanks. That solved the issue (should be fixed anyway probably somehow - either by generating different name or by showing some error).

$ oc version
oc v1.3.0
kubernetes v1.3.0+52492b4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://api.preview.openshift.com:443
openshift v3.3.1.3
kubernetes v1.3.0+52492b4

Comment 3 Cesar Wong 2016-11-09 20:40:49 UTC
Thanks. Yes, we should detect the cycle and generate a different name or at least warn you.

Comment 4 Ben Parees 2016-11-11 18:29:19 UTC
I swear we have logic that's supposed to do that already, but perhaps there are cases it's not catching.

apparently we do it for new-build but not for new-app:

$ oc new-build nenadalm/warehouse~https://github.com/nenadalm/Warehouse.git
--> Found Docker image 61b5e8b (2 days old) from Docker Hub for "nenadalm/warehouse"

    * An image stream will be created as "warehouse:latest" that will track the source image
    * A source build using source code from https://github.com/nenadalm/Warehouse.git will be created
      * The resulting image will be pushed to image stream "warehouse:latest"
      * Every time "warehouse:latest" changes a new build will be triggered

error: output image of "nenadalm/warehouse:latest" should be different than input, set a different tag with --to

Comment 5 Gabe Montero 2016-11-22 22:21:05 UTC
The comment here:  https://github.com/openshift/origin/blob/master/pkg/generate/app/cmd/newapp.go#L696

would imply the new-build vs. new-app distinction is intentional.

A search on `ExpectToBuild` seems to confirm that.  It is only set to true here:  https://github.com/openshift/origin/blob/master/pkg/cmd/cli/cmd/newbuild.go#L102

It stems from commit 79a03113 from back in March.  The PR is https://github.com/openshift/origin/pull/7936 and the associated issue is https://github.com/openshift/origin/issues/7718

There's this comment wrt the rationale:  https://github.com/openshift/origin/pull/7936#r55853107

At this point, do we not care about the warning vs. error distinction, and just remove the ExpectToBuild check?  Or just use ExpectToBuild to alter the use of fmt.Errorf vs. fmt.Printf("WARNING ...") ?

Comment 6 Ben Parees 2016-11-22 23:13:29 UTC
Thanks for digging up the history, Gabe.

I'm not sure the history clears up for me *why* we intentionally only did it for new-app.  It looks like it had to do w/ us having a "--to" flag for new-build and not for new-app, but given that you can use --name to avoid this problem with new-app, it seems like we ought to be catching it and reporting it just like we do for new-build.

I guess it also had to do w/ the fact that when new-app is instantiating a template, we should not care if you defined a cycle in your template (that's your problem at that point) though a warning would be nice.

So in sum, for new-app:

1) we need to make sure when we detect a cycle, it really is a cycle (the issue with https://github.com/openshift/origin/issues/7718 is that what it thought was a cycle was in fact *not* a cycle, it thought two things were equivalent that are not equivalent)

2) if we're instantiating a template, we can only warn you that there is a cycle, we can't error out.

3) if we're not instantiating a template, we can tell you to use --name to avoid creating a cycle. (equivalent to the --to that we recommend for new-build).

Comment 7 Gabe Montero 2016-12-01 21:14:01 UTC
PR https://github.com/openshift/origin/pull/12104 has been submitted to fix this issue.

As an aside, when I run 

   oc new-app nenadalm/warehouse~https://github.com/nenadalm/Warehouse.git

I now get

   error: output image of "nenadalm/warehouse:latest" should be different than input, override artifact names with --name

Comment 8 openshift-github-bot 2016-12-04 22:38:28 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/4c9330566770ff5f96745eade6bc44d99944a72d
new-app: appropriately warn/error on circular builds

Bug 1393549
Bugzilla link https://bugzilla.redhat.com/show_bug.cgi?id=1393549
Detailed explanation
Citing of cirular builds were performed for new-build but not new-app;
With new-app, if template, simply warn, if not template fail command

Comment 9 Gabe Montero 2016-12-05 01:09:50 UTC
With the fix merged upstream, and given this was opened from online usage, per the process, moving the bug to modified for the productization team's attention.

Comment 11 Ben Parees 2017-01-26 22:04:17 UTC
I don't know if 3.5 releases have been spun yet.  That's a question for Troy.

Comment 12 Troy Dawson 2017-01-26 22:15:13 UTC
If you can see that the fix for this is in ose/master, then yes, rpms and images have been created for this.  This is in OCP v3.5.0.8 or newer.

Comment 13 Gabe Montero 2017-01-27 03:16:28 UTC
OK - thanks Ben and Troy.

Moving to on qa then.

Comment 14 XiuJuan Wang 2017-02-06 08:43:13 UTC
Test in openshift v3.5.0.16+a26133a

@Gabe
Input image name is same with output name but different tag, will show error to create is. If should this issue be fixed.

#oc  new-app  ruby-ex:2.0 https://github.com/openshift/ruby-ex.git
--> Creating resources ...
    error: imagestreams "ruby-ex" already exists


input/output image is totally same,
'oc new-app' will prompt:
error: output image of "**" should be different than input, override artifact names with --name
‘oc new-build’ will prompt:
error: output image of "***" should be different than input, set a different tag with --to

Comment 15 Ben Parees 2017-02-06 13:30:37 UTC
please open a low severity bug, new-app should tolerate the existing imagestream since it's just going to create a new tag within that imagestream.

Comment 16 XiuJuan Wang 2017-02-07 05:49:47 UTC
Open another low bug https://bugzilla.redhat.com/show_bug.cgi?id=1419801
Verified this bug as comment #14