Description of problem: A customer installed upstream version of Cinder CSI driver, including volume snapshot support. It installs v1alpha1 version of snapshot CRDs, such as volumesnapshots.snapshot.storage.k8s.io. When upgrading to 4.4.0, where OCP introduces volume snapshots as technical preview, the upgrade got stuck at "the cluster operator csi-snapshot-controller has not yet successfully rolled out". $ oc describe co csi-snapshot-controller Name: csi-snapshot-controller [...] Status: Conditions: Last Transition Time: 2020-02-26T16:57:17Z Message: Degraded: failed to sync CRDs: CustomResourceDefinition.apiextensions.k8s.io "volumesnapshots.snapshot.storage.k8s.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions Reason: _OperatorSync Status: True Type: Degraded Last Transition Time: 2020-02-26T16:55:17Z Reason: NoData Status: Unknown Type: Progressing Last Transition Time: 2020-02-26T16:55:17Z Reason: NoData Status: Unknown Type: Available Last Transition Time: 2020-02-26T16:55:17Z Reason: NoData Status: Unknown Type: Upgradeable [...] The reason is that csi-snapshot-controller-operator wants to install v1beta1 version of the snapshot CRDs, which is not compatible with v1alpha1. Version-Release number of selected component (if applicable): 4.3.18 How reproducible: always
We cannot remove v1alpha1 CRDs during the upgrade - it would remove all snapshot CRs, i.e. data loss. We cannot convert v1alpha1 CRDs to v1beta1 - the CSI driver (not supported by us) understands only v1alpha1 and there is no automatic conversion to v1alpha1. What we can do is to mark the 4.3 cluster not upgradeable. cluster-storage-operator in 4.3 can monitor presence of v1alpha1 snapshot CRDs and mark cluster-storage-operator conditions as Upgradeable=false, with some description / link why and what to do with it.
Summit bz https://bugzilla.redhat.com/show_bug.cgi?id=1839639 for Release-nodes doc
Verify passed oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2020-07-07-233934 True False 5h Cluster version is 4.6.0-0.nightly-2020-07-07-233934 Last Transition Time: 2020-07-10T07:38:59Z Message: Unable to update cluster as v1alpha1 version of volumesnapshotcontents.snapshot.storage.k8s.io, volumesnapshots.snapshot.storage.k8s.io, volumesnapshotclasses.snapshot.storage.k8s.iois detected. Remove these CRDs to allow the upgrade to proceed. Reason: AsExpected Status: False Type: Upgradeable
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196