Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1955414

Summary: 4.8 -> 4.7 rollbacks broken on unrecognized flowschema openshift-etcd-operator
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: EtcdAssignee: Sam Batschelet <sbatsche>
Status: CLOSED ERRATA QA Contact: ge liu <geliu>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.8CC: aos-bugs, mfojtik, xxia
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1955418 (view as bug list) Environment:
Last Closed: 2021-07-27 23:05:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1955418    

Description W. Trevor King 2021-04-30 04:42:57 UTC 
Recent job [1] stuck going back to 4.7.9:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-rollback/1387886269920448512/artifacts/e2e-aws-upgrade-rollback/gather-extra/artifacts/clusterversion.json | jq -r '.items[].status.history[] | .startedTime + " " + (.completionTime // "-") + " " + .state + " " + .version'
  2021-04-29T23:51:44Z - Partial 4.7.9
  2021-04-29T22:45:09Z 2021-04-29T23:51:44Z Partial 4.8.0-0.ci-2021-04-29-151002
  2021-04-29T22:16:20Z 2021-04-29T22:42:02Z Completed 4.7.9

Stuck on the etcd-operator FlowSchema:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-rollback/1387886269920448512/artifacts/e2e-aws-upgrade-rollback/gather-extra/artifacts/clusterversion.json | jq -r '.items[].status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + .reason + ": " + .message'
  2021-04-29T22:16:20Z RetrievedUpdates=False NoChannel: The update channel has not been configured.
  2021-04-29T22:42:02Z Available=True : Done applying 4.7.9
  2021-04-30T01:55:35Z Failing=True UpdatePayloadResourceTypeMissing: Could not update flowschema "openshift-etcd-operator" (73 of 668): the server does not recognize this resource, check extension API servers
  2021-04-29T22:45:09Z Progressing=True UpdatePayloadResourceTypeMissing: Unable to apply 4.7.9: a required extension is not available to update

Sounds like the issue was that 4.7 supported both v1alpha1 and v1beta1 FlowSchema, but 4.8 only supports v1beta1 [2].  So on rollback, the etcd operator is trying to push out  its 4.7 v1alpha1 FlowSchema, and the 4.8 API-server is saying "v1alpha1?  No idea...".  Fix is probably bumping 4.7's manifest to use v1beta1, but... That may make life exciting for 4.6 -> 4.7, unless the 4.6 API-server also understands v1beta1, which it doesn't sound like it does.  Possibly close this as a WONTFIX dup of 1954481, but I thought I'd file a bug explaining why the 4.8 -> 4.7 rollback jobs were broken.  Original motivation for this FlowSchema manifest is in [3].

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-rollback/1387886269920448512
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1954481#c2
[3]: https://github.com/openshift/cluster-etcd-operator/pull/462
Comment 1 W. Trevor King 2021-04-30 04:49:54 UTC 
Hmm, or we can move the FlowSchema manifest after the kube-apiserver manifests, and then use v1beta1 in 4.7.z?  4.6 -> 4.7 updates would have their v1alpha1 FlowSchema thrown out on kube-apiserver bump, but then shortly thereafter restored via the v1beta1 manifest.  4.8 -> 4.7 downgrades would no longer have any version bump, because v1beta1 is compatible with both versions.  Moving to etcd to see if we can get that to work...
Comment 3 W. Trevor King 2021-04-30 05:08:07 UTC 
4.8->4.8 rollbacks are happy now [1,2].  I think the fix has to go into 4.7.z.

[1]: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-informing#periodic-ci-openshift-release-master-ci-4.8-e2e-aws-upgrade-rollback
[2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-aws-upgrade-rollback/1387886269983363072
Comment 5 ge liu 2021-05-24 08:10:36 UTC 
according to e2e log in comment3, this issue be fixed, thanks
Comment 8 errata-xmlrpc 2021-07-27 23:05:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438