Bug 1781302 - [4.2.z] During a 4.2 to 4.3 upgrade skew test, build e2e tests continuously fail
Summary: [4.2.z] During a 4.2 to 4.3 upgrade skew test, build e2e tests continuously fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.2.z
Assignee: Adam Kaplan
QA Contact: wewang
URL:
Whiteboard:
Depends On: 1781290
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-09 18:03 UTC by Adam Kaplan
Modified: 2020-01-07 17:55 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1781290
Environment:
Last Closed: 2020-01-07 17:55:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 24273 0 None closed Bug 1781302: Handle build controller skew in tests 2020-05-29 06:22:10 UTC
Red Hat Product Errata RHBA-2020:0014 0 None None None 2020-01-07 17:55:25 UTC

Description Adam Kaplan 2019-12-09 18:03:36 UTC
+++ This bug was initially created as a clone of Bug #1781290 +++

We run skew tests from 4.2 to 4.3 (where we install 4.2, upgrade to 4.3, but stop before updating nodes).  We then run the 4.2 e2e tests (which verifies that you didn't regress the API function).

Sometime in the last few weeks, the build subsystem e2e tests started failing continuously :

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-old-rhcos-e2e-aws-4.3/160

[Feature:Builds][pruning] prune builds based on settings in the buildconfig [Conformance] buildconfigs should have a default history limit set when created via the group api [Suite:openshift/conformance/parallel/minimal] 

fail [github.com/openshift/origin/test/extended/builds/build_pruning.go:43]: Unexpected error:
    <*errors.errorString | 0xc0000a7a00>: {
        s: "Timed out waiting for internal registry hostname to be published",
    }
    Timed out waiting for internal registry hostname to be published
occurred

There are three cases, this bug must be fixed in the appropriate place:

1. We don't correctly run these e2e tests when the control plane is 4.3 - fix in the 4.2 tests to tolerate a 4.3 control plane so that we can see it pass
2. We regressed a product API that a 4.2 client fails against a 4.3 server - must fix for ship because we don't regress APIs
3. We don't correctly work with the registry when the control plane is 4.3 and the nodes are 4.2 - must fix because you would break if someone ran during an upgrade.

This bug may not be deferred.

--- Additional comment from Clayton Coleman on 2019-12-09 17:27:57 UTC ---

Dec  9 12:47:39.809: INFO: Waiting up to 2 minutes for the internal registry hostname to be published
Dec  9 12:47:42.186: INFO: did not find the sequence in the OCM pod logs around the build controller getting started after the internal registry hostname has been set in the OCM config

--- Additional comment from Adam Kaplan on 2019-12-09 18:01:50 UTC ---

> There are three cases, this bug must be fixed in the appropriate place:
> 
> 1. We don't correctly run these e2e tests when the control plane is 4.3 - fix in the 4.2 tests to tolerate a 4.3 control plane so that we can see it pass
> 2. We regressed a product API that a 4.2 client fails against a 4.3 server - must fix for ship because we don't regress APIs
> 3. We don't correctly work with the registry when the control plane is 4.3 and the nodes are 4.2 - must fix because you would break if someone ran during an upgrade.

I suspect #1 - we found on 4.3 we started flaking because the test depended on a specific controller start sequence. We are in fact getting the right internal registry hostname synced:

```
2019-12-09T12:34:29.970319148Z I1209 12:34:29.970252       1 build_controller.go:474] Starting build controller
2019-12-09T12:34:29.970319148Z I1209 12:34:29.970305       1 build_controller.go:476] OpenShift image registry hostname: image-registry.openshift-image-registry.svc:5000
2019-12-09T12:34:30.033985328Z I1209 12:34:30.033931       1 deleted_token_secrets.go:72] caches synced
2019-12-09T12:34:30.034172215Z I1209 12:34:30.034145       1 docker_registry_service.go:154] caches synced
2019-12-09T12:34:30.034209042Z I1209 12:34:30.034188       1 create_dockercfg_secrets.go:220] urls found
2019-12-09T12:34:30.034209042Z I1209 12:34:30.034202       1 create_dockercfg_secrets.go:226] caches synced
2019-12-09T12:34:30.034705813Z I1209 12:34:30.034664       1 docker_registry_service.go:284] Updating registry URLs from map[172.30.107.118:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}] to map[172.30.107.118:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}]
```

Comment 2 wewang 2019-12-17 03:12:30 UTC
Verified in 4.2.0-0.nightly-2019-12-14-011342, steps are similar with https://bugzilla.redhat.com/show_bug.cgi?id=1781290#c5

Comment 4 errata-xmlrpc 2020-01-07 17:55:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0014


Note You need to log in before you can comment on or make changes to this bug.