Bug 2101383 - Latest OCP 4.11 builds break OADP 1.0.3 CSI functionality
Summary: Latest OCP 4.11 builds break OADP 1.0.3 CSI functionality
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.11
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: OpenShift Storage Bugzilla Bot
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-27 10:37 UTC by mperetz
Modified: 2022-06-29 06:36 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-29 06:36:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description mperetz 2022-06-27 10:37:54 UTC
Description of problem: On latest OCP 4.11 builds CSIStorageCapacity API version changed to v1 instead of v1beta1. This is probably breaks the CSI functionality on OADP 1.0.3 release, as OADP 1.0.3 the velero searches for v1beta1.

On ocp-version:4.11.0-0.nightly-2022-06-15-222801 the issue didn't occur, but on latest build the velero pod panics with this error message:

msg="The 'EnableCSI' feature flag was specified, but CSI API group [snapshot.storage.k8s.io/v1beta1] was not found.


and seems like the only CRD with v1beta1 from this API group was CSIStorageCapacity.


For example, I checked on an OSP cluster with 4.11.0-0.nightly-2022-05-25-193227 
 and I saw that it has:
(venv) [mperetz@mperetz oadp-qe-automation]$ oc api-resources --api-group=storage.k8s.io
NAME                   SHORTNAMES   APIVERSION               NAMESPACED   KIND
csidrivers                          storage.k8s.io/v1        false        CSIDriver
csinodes                            storage.k8s.io/v1        false        CSINode
csistoragecapacities                storage.k8s.io/v1beta1   true         CSIStorageCapacity <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
storageclasses         sc           storage.k8s.io/v1        false        StorageClass
volumeattachments                   storage.k8s.io/v1        false        VolumeAttachment
(venv) [mperetz@mperetz oadp-qe-automation]$ 


while ocp-version:4.11.0-0.nightly-2022-06-15-222801  has
(venv) [mperetz@mperetz oadp-qe-automation]$ oc api-resources --api-group=storage.k8s.io
NAME                   SHORTNAMES   APIVERSION          NAMESPACED   KIND
csidrivers                          storage.k8s.io/v1   false        CSIDriver
csinodes                            storage.k8s.io/v1   false        CSINode
csistoragecapacities                storage.k8s.io/v1   true         CSIStorageCapacity <<<<<<<<<<<<<<<<<<<<<<<<<<<<
storageclasses         sc           storage.k8s.io/v1   false        StorageClass
volumeattachments                   storage.k8s.io/v1   false        VolumeAttachment


Version-Release number of selected component (if applicable): latest OCP 4.11 build (last checked: 4.11.0-0.nightly-2022-06-15-222801) and OADP 1.0.3 operator installed from production sources.

How reproducible: 100%

Steps to Reproduce:
1. On latest OCP 4.11, deploy OADP 1.0.3 from production sources.
2. After the OADP operator is installed the the controller pod is ready, create the following DPA CR (with CSI plugin):

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  namespace: openshift-adp
  name: example-velero
spec:
  backupImages: true
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi
      - aws
      noDefaultBackupLocation: true
    restic:
      enable: true

3. note that after a while the velero pods moves into crashloop, and this error appears on the velero pod logs:
msg="The 'EnableCSI' feature flag was specified, but CSI API group [snapshot.storage.k8s.io/v1beta1] was not found.

Actual results:

Expected results:

Master Log:

Node Log (of failed PODs):

msg="The 'EnableCSI' feature flag was specified, but CSI API group [snapshot.storage.k8s.io/v1beta1] was not found.

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 2 Jan Safranek 2022-06-27 14:46:05 UTC
I doubt Velero uses CSIStorageCapacity. It complains about "snapshot.storage.k8s.io/v1beta1" and that has been indeed removed in 4.11, as snapshot.storage.k8s.io/v1 is available since OCP 4.7 and v1beta1 was used as tech preview.

Why can't you switch to v1?

Comment 4 Dylan Murray 2022-06-28 13:16:39 UTC
Jan is correct that the issue is the 1.0.3 CSI plugin expected v1beta1 to exist and fails now that it's removed. For OADP 1.1 we are using the latest CSI plugin which uses `v1`. If we want 1.0.x to support 4.11 we'll need to backport this fix to oadp-1.0.

Comment 5 Jan Safranek 2022-06-28 13:27:50 UTC
Do you want to reassign the BZ to yourself? Otherwise I will close it, as there is nothing to fix on OCP side.

Comment 6 mperetz 2022-06-29 06:36:58 UTC
Closed as won't fix and opened a bug on OADP Jira project instead: https://issues.redhat.com/browse/OADP-585


Note You need to log in before you can comment on or make changes to this bug.