Bug 1268000 - sometimes two builds are started when using "add to project" from web console
Summary: sometimes two builds are started when using "add to project" from web console
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Maciej Szulik
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-01 14:27 UTC by Erik M Jacobs
Modified: 2015-11-23 14:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-23 14:43:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
build configuration json (1.85 KB, text/plain)
2015-10-01 14:28 UTC, Erik M Jacobs
no flags Details
json output of two builds (4.18 KB, text/plain)
2015-10-01 14:28 UTC, Erik M Jacobs
no flags Details
last 1000 lines of master's log at loglevel 5 (331.18 KB, text/plain)
2015-10-01 14:29 UTC, Erik M Jacobs
no flags Details

Description Erik M Jacobs 2015-10-01 14:27:30 UTC
openshift-node-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-master-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
tuned-profiles-openshift-node-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-sdn-ovs-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64

See attached files.

Comment 1 Erik M Jacobs 2015-10-01 14:28:13 UTC
Created attachment 1079140 [details]
build configuration json

Comment 2 Erik M Jacobs 2015-10-01 14:28:48 UTC
Created attachment 1079141 [details]
json output of two builds

Comment 3 Erik M Jacobs 2015-10-01 14:29:23 UTC
Created attachment 1079143 [details]
last 1000 lines of master's log at loglevel 5

Comment 5 Ben Parees 2015-10-01 15:42:49 UTC
build was was the initial build for a configchange trigger:
Oct 01 10:17:59 ose3-master.example.com openshift-master[16062]: I1001 10:17:59.145781   16062 generator.go:167] Generating Build from BuildConfig: tooz/sinatra-example, LastVersion: 0

build 2 was due to an image change trigger:
Oct 01 10:19:31 ose3-master.example.com openshift-master[16062]: I1001 10:19:31.841295   16062 generator.go:167] Generating Build from BuildConfig: tooz/sinatra-example, TriggeredBy: DockerImage/registry.access.redhat.com/openshift3/ruby-20-rhel7:latest with stream: ImageStreamTag/ruby:2.0

the question is why build 2 wasn't prevented by the logic that should be checking if we already ran a build for a given imagestreamtag.  i'm guessing the imagestreamtag didn't exist at the time the configchange triggered build was created.

Comment 6 Ben Parees 2015-10-01 15:57:21 UTC
so initially we resolve the from reference as:
 Resolved ImageStreamTag ruby:2.0 to image ea16bfe0829377fb1c0191d2953a1e104c9eb05d3691bfe9443bf6a12931fe25 with reference registry.access.redhat.com/openshift3/ruby-20-rhel7:v3.0.2.0 in namespace openshift

then we later get a trigger for:
 TriggeredBy: DockerImage/registry.access.redhat.com/openshift3/ruby-20-rhel7:latest with stream: ImageStreamTag/ruby:2.0

which means the IST for ruby:2.0 changed, which doesn't make much sense to me.

when I just imported the imagestreams now, i get only:
 "tag": "2.0",
                        "items": [
                            {
                                "created": "2015-10-01T15:53:10Z",
                                "dockerImageReference": "registry.access.redhat.com/openshift3/ruby-20-rhel7:latest",
                                "image": "ea16bfe0829377fb1c0191d2953a1e104c9eb05d3691bfe9443bf6a12931fe25"
                            }
                        ]


which is what i'd expect.  over to the registry team to see if they can shed some light on why we'd be seeing the IST change here.

Comment 11 Maciej Szulik 2015-10-21 09:07:23 UTC
I'm currently reworking our import code as part of [1]. Unfortunately current implementation (as well as my PR) does not contain valuable logs, will work on adding them. I'll try to analyze the code though and figure out what might be the problem.

[1] https://github.com/openshift/origin/pull/4853

Comment 12 Maciej Szulik 2015-10-27 15:06:09 UTC
This is applicable only to rhel7 images, I can't reproduce the problem with centos7 counterparts. It's still relevant in current master 814c05e869d16a1252855bab7244b72989e1857c, though.

Comment 13 Maciej Szulik 2015-10-28 10:43:13 UTC
The problem is following, first execution of build saves wrong image in LastTriggeredImageID (registry.access.redhat.com/openshift3/ruby-20-rhel7:v3.0.2.0). When the imagechange trigger is fired it checks that against registry.access.redhat.com/openshift3/ruby-20-rhel7:latest to find it's different and kick another build. 

The problem seems to be in [1] which returns wrong istag. It should return .../ruby-20-rhel7:latest but for some reason returns ruby-20-rhel7:v3.0.2.0 instead. I'm looking what is causing this.

[1] https://github.com/openshift/origin/blob/ab3a5d9d577cb378fc67d346c53801f82a737b01/pkg/build/generator/generator.go#L450

Comment 14 Maciej Szulik 2015-10-30 10:10:17 UTC
The fix is in https://github.com/openshift/origin/pull/5495.

Comment 16 openshift-github-bot 2015-11-02 13:39:34 UTC
Commits pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/d94907ee0c9ef27a0849ce81098585e6d8d2e894
Bug 1268000 - replace Image.DockerImageReference with value from status.

https://github.com/openshift/origin/commit/14aa7baaabde7748dfceb26851355c1e178c9937
Merge pull request #5495 from soltysh/bug1268000

Merged by openshift-bot

Comment 17 Wei Sun 2015-11-04 06:07:18 UTC
Created new app sinatra-example on web console using the latest ose env (openshift v3.0.2.905) today,did not see two builds.

So verified this bug.


Note You need to log in before you can comment on or make changes to this bug.