Bug 1781302

Summary: [4.2.z] During a 4.2 to 4.3 upgrade skew test, build e2e tests continuously fail
Product: OpenShift Container Platform Reporter: Adam Kaplan <adam.kaplan>
Component: BuildAssignee: Adam Kaplan <adam.kaplan>
Status: CLOSED ERRATA QA Contact: wewang <wewang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: aos-bugs, ccoleman, wewang, wzheng
Target Milestone: ---   
Target Release: 4.2.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1781290 Environment:
Last Closed: 2020-01-07 17:55:11 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1781290    
Bug Blocks:    

Description Adam Kaplan 2019-12-09 18:03:36 UTC
+++ This bug was initially created as a clone of Bug #1781290 +++

We run skew tests from 4.2 to 4.3 (where we install 4.2, upgrade to 4.3, but stop before updating nodes).  We then run the 4.2 e2e tests (which verifies that you didn't regress the API function).

Sometime in the last few weeks, the build subsystem e2e tests started failing continuously :

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-old-rhcos-e2e-aws-4.3/160

[Feature:Builds][pruning] prune builds based on settings in the buildconfig [Conformance] buildconfigs should have a default history limit set when created via the group api [Suite:openshift/conformance/parallel/minimal] 

fail [github.com/openshift/origin/test/extended/builds/build_pruning.go:43]: Unexpected error:
    <*errors.errorString | 0xc0000a7a00>: {
        s: "Timed out waiting for internal registry hostname to be published",
    }
    Timed out waiting for internal registry hostname to be published
occurred

There are three cases, this bug must be fixed in the appropriate place:

1. We don't correctly run these e2e tests when the control plane is 4.3 - fix in the 4.2 tests to tolerate a 4.3 control plane so that we can see it pass
2. We regressed a product API that a 4.2 client fails against a 4.3 server - must fix for ship because we don't regress APIs
3. We don't correctly work with the registry when the control plane is 4.3 and the nodes are 4.2 - must fix because you would break if someone ran during an upgrade.

This bug may not be deferred.

--- Additional comment from Clayton Coleman on 2019-12-09 17:27:57 UTC ---

Dec  9 12:47:39.809: INFO: Waiting up to 2 minutes for the internal registry hostname to be published
Dec  9 12:47:42.186: INFO: did not find the sequence in the OCM pod logs around the build controller getting started after the internal registry hostname has been set in the OCM config

--- Additional comment from Adam Kaplan on 2019-12-09 18:01:50 UTC ---

> There are three cases, this bug must be fixed in the appropriate place:
> 
> 1. We don't correctly run these e2e tests when the control plane is 4.3 - fix in the 4.2 tests to tolerate a 4.3 control plane so that we can see it pass
> 2. We regressed a product API that a 4.2 client fails against a 4.3 server - must fix for ship because we don't regress APIs
> 3. We don't correctly work with the registry when the control plane is 4.3 and the nodes are 4.2 - must fix because you would break if someone ran during an upgrade.

I suspect #1 - we found on 4.3 we started flaking because the test depended on a specific controller start sequence. We are in fact getting the right internal registry hostname synced:

```
2019-12-09T12:34:29.970319148Z I1209 12:34:29.970252       1 build_controller.go:474] Starting build controller
2019-12-09T12:34:29.970319148Z I1209 12:34:29.970305       1 build_controller.go:476] OpenShift image registry hostname: image-registry.openshift-image-registry.svc:5000
2019-12-09T12:34:30.033985328Z I1209 12:34:30.033931       1 deleted_token_secrets.go:72] caches synced
2019-12-09T12:34:30.034172215Z I1209 12:34:30.034145       1 docker_registry_service.go:154] caches synced
2019-12-09T12:34:30.034209042Z I1209 12:34:30.034188       1 create_dockercfg_secrets.go:220] urls found
2019-12-09T12:34:30.034209042Z I1209 12:34:30.034202       1 create_dockercfg_secrets.go:226] caches synced
2019-12-09T12:34:30.034705813Z I1209 12:34:30.034664       1 docker_registry_service.go:284] Updating registry URLs from map[172.30.107.118:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}] to map[172.30.107.118:5000:{} image-registry.openshift-image-registry.svc.cluster.local:5000:{} image-registry.openshift-image-registry.svc:5000:{}]
```

Comment 2 wewang 2019-12-17 03:12:30 UTC
Verified in 4.2.0-0.nightly-2019-12-14-011342, steps are similar with https://bugzilla.redhat.com/show_bug.cgi?id=1781290#c5

Comment 4 errata-xmlrpc 2020-01-07 17:55:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0014