Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1708648

Summary:	Release images with different commits for images built from the same Git repository
Product:	OpenShift Container Platform	Reporter:	Gabe Montero <gmontero>
Component:	Release	Assignee:	Tim Bielawa <tbielawa>
Status:	CLOSED ERRATA	QA Contact:	Mike Fiedler <mifiedle>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.1.0	CC:	aos-bugs, bparees, jokerman, lmeyer, miabbott, mifiedle, mmccomas, sjenning, smunilla, sponnaga, vlaad, wking
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	buildcop
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-04 10:48:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1707681

Description Gabe Montero 2019-05-10 13:15:07 UTC

Description of problem:

https://prow.svc.ci.openshift.org/?job=*release-*-ocp-*-e2e-aws* jobs started failing about 17:00 May 9

List of recent runs can be found via https://search.svc.ci.openshift.org/?search=Cluster+operator+machine-config+is+reporting+a+failure&maxAge=168h&context=2&type=all

Or putting in "Cluster operator machine-config is reporting a failure" in the serch ci tool



In dumping the build logs from the recent runs, I'm consistently seeing error logs of this type:

level=fatal msg="failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Failed to resync 4.1.0-0.nightly-2019-05-09-213253 because: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f6c5d0693e1b59b15b350d12d9c75dc4 expected b92b681d55bbe071d0f6f9dda080b3a77a879ff9 has 51229c70416ca18819e3a4a0bee4dce35157a1f8, retrying: timed out waiting for the condition"

Most recent run was https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.1/192/

Comment 1 Gabe Montero 2019-05-10 13:16:58 UTC

From Colin Walters ... slack discussion in https://coreos.slack.com/messages/C999USB0D/

Comment 2 W. Trevor King 2019-05-10 13:20:22 UTC

This is narrowly a dup of bug 1707928, but I'm going to re-purpose this one to be about the larger ART/CI issue of different commits of the same repository entirely being used for different referenced images within the same release image (e.g. see [1], [2]).

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1707928#c18
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1707928#c35

Comment 3 W. Trevor King 2019-05-10 13:48:40 UTC

[1] should help debug CI/oc for this issue.  See [2] for why I want more detail on ImageStream fetching.

[1]: https://github.com/openshift/ci-operator/pull/343
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1707928#c3

Comment 4 W. Trevor King 2019-05-10 18:08:55 UTC

[1] has a guard for this so 'oc adm release new ...' will fail-fast when it happens.  I think we want the guard, but should keep this bug open until we can keep this from happening by fixing the tooling that feeds 'oc adm release new ...'.  Luke has some ideas about fixing partial syncs that might help with the root cause (at least for ART builds).

[1]: https://github.com/openshift/origin/pull/22816

Comment 6 Vikas Laad 2019-05-13 16:39:37 UTC

*** Bug 1709365 has been marked as a duplicate of this bug. ***

Comment 7 Sudha Ponnaganti 2019-05-14 00:51:26 UTC

@tim - This is a test blocker

Comment 8 Luke Meyer 2019-05-14 16:58:48 UTC

Waiting for https://github.com/openshift/machine-config-operator/pull/743 to merge and then can rebuild.

Comment 9 Luke Meyer 2019-05-14 21:07:33 UTC

https://github.com/openshift/machine-config-operator/pull/743 was really a separate issue.

Comment 12 Mike Fiedler 2019-05-15 02:12:39 UTC

Verified on 4.1.0-0.nightly-2019-05-14-202907 - install successful

Comment 14 errata-xmlrpc 2019-06-04 10:48:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758