Since ~ 4.3.0-rc.0 -> 4.4.0-0.ci-2020-01-13-202732 [1]: Cluster did not complete upgrade: timed out waiting for the condition: Working towards 4.4.0-0.ci-2020-01-13-202732: 24% complete with: Jan 13 21:21:47.341 E clusterversion/version changed Failing to True: UpdatePayloadResourceTypeMissing: Could not update kubestorageversionmigrator "cluster" (125 of 514): the server does not recognize this resource, check extension API servers and similar. We need to drop [2] from the 4.4 release image or soften its types so they aren't pushed so early in the update that the 4.3 API servers are still running. [1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14188 [2]: https://github.com/openshift/kubernetes-kube-storage-version-migrator
If you want a full list of affected jobs: $ curl -s 'https://search.svc.ci.openshift.org/search?search=UpdatePayloadResourceTypeMissing:+Could+not+update+kubestorageversionmigrator' | jq -r '. | keys[]' https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14188 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14191 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14193 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14204 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14208 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14210 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14216 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14218 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14219 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14228 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14231 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14233 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14236 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14241 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14246 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14250 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14252 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14259 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14262 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14264 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14266 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14268 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14270 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14271 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14273 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14275 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14277 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14279 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14281 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14284 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14287 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14292 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14295 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14304 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14307 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14310 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14312 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14314 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14319 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14327 but basically this is all 4.3->4.4 since the new repo has been included in 4.4 release images.
Example of dropping an image from the 4.4 release payload: https://bugzilla.redhat.com/show_bug.cgi?id=1790244#c2
https://github.com/openshift/cluster-kube-storage-version-migrator-operator/pull/9
Per https://github.com/openshift/cluster-kube-storage-version-migrator-operator/pull/9, the kubernetes-kube-storage-version-migrator related CRD resources were moved to run-level 40, so we just check if the actual CRD resource manifest already be applied after OCP was updated from 4.3 to 4.4. First, 4.3.5-rc -> 4.4.0-0.nightly-2020-03-18-092817 upgrade passed, see below, $ oc get clusterversion -o json|jq ".items[0].status.history" [ { "completionTime": "2020-03-20T16:28:53Z", "image": "registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-03-18-092817", "startedTime": "2020-03-20T15:42:26Z", "state": "Completed", "verified": false, "version": "4.4.0-0.nightly-2020-03-18-092817" }, { "completionTime": "2020-03-20T15:30:26Z", "image": "quay.io/openshift-release-dev/ocp-release@sha256:64320fbf95d968fc6b9863581a92d373bc75f563a13ae1c727af37450579f61a", "startedTime": "2020-03-20T15:11:36Z", "state": "Completed", "verified": false, "version": "4.3.5" } ] Second, checking the changed run level kubernetes-kube-storage-version-migrator related CRD resources manifest, all is well. $ oc get CustomResourceDefinition | grep migration.k8s.io storagestates.migration.k8s.io 2020-03-20T15:51:49Z storageversionmigrations.migration.k8s.io 2020-03-20T15:51:46Z $ oc get ns -A | grep openshift-kube-storage-version-migrator-operator openshift-kube-storage-version-migrator-operator Active 35h $ oc get cm -A | grep openshift-kube-storage-version-migrator-operator openshift-kube-storage-version-migrator-operator config 1 35h openshift-kube-storage-version-migrator-operator openshift-kube-storage-version-migrator-operator-lock 0 35h $ oc get sa -A | grep kube-storage-version-migrator-operator openshift-kube-storage-version-migrator-operator builder 2 35h openshift-kube-storage-version-migrator-operator default 2 35h openshift-kube-storage-version-migrator-operator deployer 2 35h openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator 2 35h $ oc get CustomResourceDefinition | grep ubestorageversionmigrators.operator.openshift.io kubestorageversionmigrators.operator.openshift.io 2020-03-20T15:51:43Z $ oc get deployment -A |grep kube-storage-version-migrator-operator openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator 1/1 1 1 35h $ oc get service -A | grep migrator openshift-kube-storage-version-migrator-operator metrics ClusterIP 172.30.218.90 <none> 443/TCP 35h $ oc get co -A | grep kube-storage-version-migrator kube-storage-version-migrator 4.4.0-0.nightly-2020-03-18-092817 True False False 35h $ oc get ClusterRoleBinding | grep kube-storage-version-migrator-operator system:openshift:operator:kube-storage-version-migrator-operator 35h $ oc get KubeStorageVersionMigrator NAME AGE cluster 35h
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581