Bug 1268000 - sometimes two builds are started when using "add to project" from web console
sometimes two builds are started when using "add to project" from web console
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry (Show other bugs)
3.0.0
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Maciej Szulik
Wei Sun
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-01 10:27 EDT by Erik M Jacobs
Modified: 2015-11-23 09:43 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-23 09:43:53 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
build configuration json (1.85 KB, text/plain)
2015-10-01 10:28 EDT, Erik M Jacobs
no flags Details
json output of two builds (4.18 KB, text/plain)
2015-10-01 10:28 EDT, Erik M Jacobs
no flags Details
last 1000 lines of master's log at loglevel 5 (331.18 KB, text/plain)
2015-10-01 10:29 EDT, Erik M Jacobs
no flags Details

  None (edit)
Description Erik M Jacobs 2015-10-01 10:27:30 EDT
openshift-node-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-master-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
tuned-profiles-openshift-node-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64
openshift-sdn-ovs-3.0.2.0-0.git.5.ee06ab6.el7ose.x86_64

See attached files.
Comment 1 Erik M Jacobs 2015-10-01 10:28 EDT
Created attachment 1079140 [details]
build configuration json
Comment 2 Erik M Jacobs 2015-10-01 10:28 EDT
Created attachment 1079141 [details]
json output of two builds
Comment 3 Erik M Jacobs 2015-10-01 10:29 EDT
Created attachment 1079143 [details]
last 1000 lines of master's log at loglevel 5
Comment 5 Ben Parees 2015-10-01 11:42:49 EDT
build was was the initial build for a configchange trigger:
Oct 01 10:17:59 ose3-master.example.com openshift-master[16062]: I1001 10:17:59.145781   16062 generator.go:167] Generating Build from BuildConfig: tooz/sinatra-example, LastVersion: 0

build 2 was due to an image change trigger:
Oct 01 10:19:31 ose3-master.example.com openshift-master[16062]: I1001 10:19:31.841295   16062 generator.go:167] Generating Build from BuildConfig: tooz/sinatra-example, TriggeredBy: DockerImage/registry.access.redhat.com/openshift3/ruby-20-rhel7:latest with stream: ImageStreamTag/ruby:2.0

the question is why build 2 wasn't prevented by the logic that should be checking if we already ran a build for a given imagestreamtag.  i'm guessing the imagestreamtag didn't exist at the time the configchange triggered build was created.
Comment 6 Ben Parees 2015-10-01 11:57:21 EDT
so initially we resolve the from reference as:
 Resolved ImageStreamTag ruby:2.0 to image ea16bfe0829377fb1c0191d2953a1e104c9eb05d3691bfe9443bf6a12931fe25 with reference registry.access.redhat.com/openshift3/ruby-20-rhel7:v3.0.2.0 in namespace openshift

then we later get a trigger for:
 TriggeredBy: DockerImage/registry.access.redhat.com/openshift3/ruby-20-rhel7:latest with stream: ImageStreamTag/ruby:2.0

which means the IST for ruby:2.0 changed, which doesn't make much sense to me.

when I just imported the imagestreams now, i get only:
 "tag": "2.0",
                        "items": [
                            {
                                "created": "2015-10-01T15:53:10Z",
                                "dockerImageReference": "registry.access.redhat.com/openshift3/ruby-20-rhel7:latest",
                                "image": "ea16bfe0829377fb1c0191d2953a1e104c9eb05d3691bfe9443bf6a12931fe25"
                            }
                        ]


which is what i'd expect.  over to the registry team to see if they can shed some light on why we'd be seeing the IST change here.
Comment 11 Maciej Szulik 2015-10-21 05:07:23 EDT
I'm currently reworking our import code as part of [1]. Unfortunately current implementation (as well as my PR) does not contain valuable logs, will work on adding them. I'll try to analyze the code though and figure out what might be the problem.

[1] https://github.com/openshift/origin/pull/4853
Comment 12 Maciej Szulik 2015-10-27 11:06:09 EDT
This is applicable only to rhel7 images, I can't reproduce the problem with centos7 counterparts. It's still relevant in current master 814c05e869d16a1252855bab7244b72989e1857c, though.
Comment 13 Maciej Szulik 2015-10-28 06:43:13 EDT
The problem is following, first execution of build saves wrong image in LastTriggeredImageID (registry.access.redhat.com/openshift3/ruby-20-rhel7:v3.0.2.0). When the imagechange trigger is fired it checks that against registry.access.redhat.com/openshift3/ruby-20-rhel7:latest to find it's different and kick another build. 

The problem seems to be in [1] which returns wrong istag. It should return .../ruby-20-rhel7:latest but for some reason returns ruby-20-rhel7:v3.0.2.0 instead. I'm looking what is causing this.

[1] https://github.com/openshift/origin/blob/ab3a5d9d577cb378fc67d346c53801f82a737b01/pkg/build/generator/generator.go#L450
Comment 14 Maciej Szulik 2015-10-30 06:10:17 EDT
The fix is in https://github.com/openshift/origin/pull/5495.
Comment 16 openshift-github-bot 2015-11-02 08:39:34 EST
Commits pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/d94907ee0c9ef27a0849ce81098585e6d8d2e894
Bug 1268000 - replace Image.DockerImageReference with value from status.

https://github.com/openshift/origin/commit/14aa7baaabde7748dfceb26851355c1e178c9937
Merge pull request #5495 from soltysh/bug1268000

Merged by openshift-bot
Comment 17 Wei Sun 2015-11-04 01:07:18 EST
Created new app sinatra-example on web console using the latest ose env (openshift v3.0.2.905) today,did not see two builds.

So verified this bug.

Note You need to log in before you can comment on or make changes to this bug.