Bug 1955418

+++ This bug was initially created as a clone of Bug #1955414 +++

Recent job [1] stuck going back to 4.7.9:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-rollback/1387886269920448512/artifacts/e2e-aws-upgrade-rollback/gather-extra/artifacts/clusterversion.json | jq -r '.items[].status.history[] | .startedTime + " " + (.completionTime // "-") + " " + .state + " " + .version'
  2021-04-29T23:51:44Z - Partial 4.7.9
  2021-04-29T22:45:09Z 2021-04-29T23:51:44Z Partial 4.8.0-0.ci-2021-04-29-151002
  2021-04-29T22:16:20Z 2021-04-29T22:42:02Z Completed 4.7.9

Stuck on the etcd-operator FlowSchema:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-rollback/1387886269920448512/artifacts/e2e-aws-upgrade-rollback/gather-extra/artifacts/clusterversion.json | jq -r '.items[].status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + .reason + ": " + .message'
  2021-04-29T22:16:20Z RetrievedUpdates=False NoChannel: The update channel has not been configured.
  2021-04-29T22:42:02Z Available=True : Done applying 4.7.9
  2021-04-30T01:55:35Z Failing=True UpdatePayloadResourceTypeMissing: Could not update flowschema "openshift-etcd-operator" (73 of 668): the server does not recognize this resource, check extension API servers
  2021-04-29T22:45:09Z Progressing=True UpdatePayloadResourceTypeMissing: Unable to apply 4.7.9: a required extension is not available to update

Sounds like the issue was that 4.7 supported both v1alpha1 and v1beta1 FlowSchema, but 4.8 only supports v1beta1 [2].  So on rollback, the etcd operator is trying to push out  its 4.7 v1alpha1 FlowSchema, and the 4.8 API-server is saying "v1alpha1?  No idea...".  Fix is probably bumping 4.7's manifest to use v1beta1, but... That may make life exciting for 4.6 -> 4.7, unless the 4.6 API-server also understands v1beta1, which it doesn't sound like it does.  Possibly close this as a WONTFIX dup of 1954481, but I thought I'd file a bug explaining why the 4.8 -> 4.7 rollback jobs were broken.  Original motivation for this FlowSchema manifest is in [3].

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-rollback/1387886269920448512
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1954481#c2
[3]: https://github.com/openshift/cluster-etcd-operator/pull/462

--- Additional comment from W. Trevor King on 2021-04-30 04:49:54 UTC ---

Hmm, or we can move the FlowSchema manifest after the kube-apiserver manifests, and then use v1beta1 in 4.7.z?  4.6 -> 4.7 updates would have their v1alpha1 FlowSchema thrown out on kube-apiserver bump, but then shortly thereafter restored via the v1beta1 manifest.  4.8 -> 4.7 downgrades would no longer have any version bump, because v1beta1 is compatible with both versions.  Moving to etcd to see if we can get that to work...