Bug 1268000

Summary: sometimes two builds are started when using "add to project" from web console
Product: OpenShift Container Platform Reporter: Erik M Jacobs <ejacobs>
Component: Image RegistryAssignee: Maciej Szulik <maszulik>
Status: CLOSED CURRENTRELEASE QA Contact: Wei Sun <wsun>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.0.0CC: aos-bugs, bparees, dmcphers
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-23 14:43:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
build configuration json
none
json output of two builds
none
last 1000 lines of master's log at loglevel 5 none

Description Erik M Jacobs 2015-10-01 14:27:30 UTC
openshift-node-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-master-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
tuned-profiles-openshift-node-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-sdn-ovs-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64

See attached files.

Comment 1 Erik M Jacobs 2015-10-01 14:28:13 UTC
Created attachment 1079140 [details]
build configuration json

Comment 2 Erik M Jacobs 2015-10-01 14:28:48 UTC
Created attachment 1079141 [details]
json output of two builds

Comment 3 Erik M Jacobs 2015-10-01 14:29:23 UTC
Created attachment 1079143 [details]
last 1000 lines of master's log at loglevel 5

Comment 5 Ben Parees 2015-10-01 15:42:49 UTC
build was was the initial build for a configchange trigger:
Oct 01 10:17:59 ose3-master.example.com openshift-master[16062]: I1001 10:17:59.145781   16062 generator.go:167] Generating Build from BuildConfig: tooz/sinatra-example, LastVersion: 0

build 2 was due to an image change trigger:
Oct 01 10:19:31 ose3-master.example.com openshift-master[16062]: I1001 10:19:31.841295   16062 generator.go:167] Generating Build from BuildConfig: tooz/sinatra-example, TriggeredBy: DockerImage/registry.access.redhat.com/openshift3/ruby-20-rhel7:latest with stream: ImageStreamTag/ruby:2.0

the question is why build 2 wasn't prevented by the logic that should be checking if we already ran a build for a given imagestreamtag.  i'm guessing the imagestreamtag didn't exist at the time the configchange triggered build was created.

Comment 6 Ben Parees 2015-10-01 15:57:21 UTC
so initially we resolve the from reference as:
 Resolved ImageStreamTag ruby:2.0 to image ea16bfe0829377fb1c0191d2953a1e104c9eb05d3691bfe9443bf6a12931fe25 with reference registry.access.redhat.com/openshift3/ruby-20-rhel7:v3.0.2.0 in namespace openshift

then we later get a trigger for:
 TriggeredBy: DockerImage/registry.access.redhat.com/openshift3/ruby-20-rhel7:latest with stream: ImageStreamTag/ruby:2.0

which means the IST for ruby:2.0 changed, which doesn't make much sense to me.

when I just imported the imagestreams now, i get only:
 "tag": "2.0",
                        "items": [
                            {
                                "created": "2015-10-01T15:53:10Z",
                                "dockerImageReference": "registry.access.redhat.com/openshift3/ruby-20-rhel7:latest",
                                "image": "ea16bfe0829377fb1c0191d2953a1e104c9eb05d3691bfe9443bf6a12931fe25"
                            }
                        ]


which is what i'd expect.  over to the registry team to see if they can shed some light on why we'd be seeing the IST change here.

Comment 11 Maciej Szulik 2015-10-21 09:07:23 UTC
I'm currently reworking our import code as part of [1]. Unfortunately current implementation (as well as my PR) does not contain valuable logs, will work on adding them. I'll try to analyze the code though and figure out what might be the problem.

[1] https://github.com/openshift/origin/pull/4853

Comment 12 Maciej Szulik 2015-10-27 15:06:09 UTC
This is applicable only to rhel7 images, I can't reproduce the problem with centos7 counterparts. It's still relevant in current master 814c05e869d16a1252855bab7244b72989e1857c, though.

Comment 13 Maciej Szulik 2015-10-28 10:43:13 UTC
The problem is following, first execution of build saves wrong image in LastTriggeredImageID (registry.access.redhat.com/openshift3/ruby-20-rhel7:v3.0.2.0). When the imagechange trigger is fired it checks that against registry.access.redhat.com/openshift3/ruby-20-rhel7:latest to find it's different and kick another build. 

The problem seems to be in [1] which returns wrong istag. It should return .../ruby-20-rhel7:latest but for some reason returns ruby-20-rhel7:v3.0.2.0 instead. I'm looking what is causing this.

[1] https://github.com/openshift/origin/blob/ab3a5d9d577cb378fc67d346c53801f82a737b01/pkg/build/generator/generator.go#L450

Comment 14 Maciej Szulik 2015-10-30 10:10:17 UTC
The fix is in https://github.com/openshift/origin/pull/5495.

Comment 16 openshift-github-bot 2015-11-02 13:39:34 UTC
Commits pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/d94907ee0c9ef27a0849ce81098585e6d8d2e894
Bug 1268000 - replace Image.DockerImageReference with value from status.

https://github.com/openshift/origin/commit/14aa7baaabde7748dfceb26851355c1e178c9937
Merge pull request #5495 from soltysh/bug1268000

Merged by openshift-bot

Comment 17 Wei Sun 2015-11-04 06:07:18 UTC
Created new app sinatra-example on web console using the latest ose env (openshift v3.0.2.905) today,did not see two builds.

So verified this bug.