Bug 1922119

Summary: PVC snapshot creation failing on OCP4.6-OCS 4.7 cluster
Product: [Red Hat Storage] Red Hat OpenShift Container Storage Reporter: Jilju Joy <jijoy>
Component: csi-driverAssignee: Humble Chirammal <hchiramm>
Status: CLOSED ERRATA QA Contact: Jilju Joy <jijoy>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.7CC: branto, ebenahar, ebondare, hekumar, jsafrane, madam, mrajanna, muagarwa, ocs-bugs, owasserm
Target Milestone: ---Keywords: Automation, Regression
Target Release: OCS 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.7.0-721.ci Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-19 09:18:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jilju Joy 2021-01-29 09:47:26 UTC
Description of problem (please be detailed as possible and provide log
snippests):
Volume snapshot creation is failing on a cluster upgraded form OCS 4.6 to OCS 4.7. The OCP version is 4.6.

This error is repeated in csi-snapshotter container logs.


2021-01-25T17:29:01.348176975Z I0125 17:29:01.348136       1 reflector.go:255] Listing and watching *v1.VolumeSnapshotClass from github.com/kubernetes-csi/external-snapshotter/client/v4/informers/externalversions/factory.go:117
2021-01-25T17:29:01.350299415Z E0125 17:29:01.350271       1 reflector.go:138] github.com/kubernetes-csi/external-snapshotter/client/v4/informers/externalversions/factory.go:117: Failed to watch *v1.VolumeSnapshotClass: failed to list *v1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io)


Build url - https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/16688/

All snapshot tests failed:
1.tests/manage/pv_services/pvc_snapshot/test_pvc_snapshot.py::TestPvcSnapshot::test_pvc_snapshot[CephBlockPool]
2.tests/manage/pv_services/pvc_snapshot/test_pvc_snapshot.py::TestPvcSnapshot::test_pvc_snapshot[CephFile System]
3.tests/manage/pv_services/pvc_snapshot/test_rbd_block_pvc_snapshot.py::TestRbdBlockPvcSnapshot::test_rbd_block_pvc_snapshot
4.tests/manage/pv_services/pvc_snapshot/test_restore_snapshot_using_different_sc.py::TestRestoreSnapshotUsingDifferentSc::test_snapshot_restore_using_different_sc
5.tests/manage/pv_services/pvc_snapshot/test_snapshot_at_different_pvc_utlilization_level.py::TestSnapshotAtDifferentPvcUsageLevel::test_snapshot_at_different_usage_level
6.tests/manage/pv_services/pvc_snapshot/test_snapshot_restore_with_different_access_mode.py::TestSnapshotRestoreWithDifferentAccessMode::test_snapshot_restore_with_different_access_mode


logs - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j003ai3c33-uan/j003ai3c33-uan_20210125T151654/logs/testcases_1611601132/

List of snapshots in not ready state - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j003ai3c33-uan/j003ai3c33-uan_20210125T151654/logs/testcases_1611601132/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-5ac742001d236e9f6a2f54db2b389990fdef04064ac7dc0ae89a7f4937c779e1/namespaces/openshift-storage/oc_output/volumesnapshot_-A

Snapshot class yaml - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j003ai3c33-uan/j003ai3c33-uan_20210125T151654/logs/testcases_1611601132/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-5ac742001d236e9f6a2f54db2b389990fdef04064ac7dc0ae89a7f4937c779e1/namespaces/openshift-storage/oc_output/volumesnapshotclass

Version of all relevant components (if applicable):
$ oc get csv
NAME                         DISPLAY                       VERSION        REPLACES              PHASE
ocs-operator.v4.7.0-238.ci   OpenShift Container Storage   4.7.0-238.ci   ocs-operator.v4.6.1   Succeeded

Cluster version is 4.6.0-0.nightly-2021-01-22-123731


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Reporting first instance

Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:
Yes, the tests were passing in OCP4.6-OCS4.6 cluster.

Steps to Reproduce:
1. Create CephFS and RBD PVCs.
2. Create snapshots of the PVCs

OR
Run the test - 
tests/manage/pv_services/pvc_snapshot/test_pvc_snapshot.py


Actual results:
PVC snapshot creation fails


Expected results:
PVC snapshot creation should succeeded


Additional info:

Comment 3 Jilju Joy 2021-01-29 13:23:26 UTC
Reproduced this issue on fresh installation of OCS 4.7 on OCP 4.6.
OCP 4.6.0-0.nightly-2021-01-29-062917
OCS 4.7.0-241.ci

Failed tests:
tests/manage/pv_services/pvc_snapshot/test_pvc_snapshot.py::TestPvcSnapshot::test_pvc_snapshot[CephBlockPool]
tests/manage/pv_services/pvc_snapshot/test_pvc_snapshot.py::TestPvcSnapshot::test_pvc_snapshot[CephFile System]

Build url - https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/16823/

must-gather logs - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-bz1922119/jijoy-bz1922119_20210129T101925/logs/failed_testcase_ocs_logs_1611921996/

Comment 13 Elena Bondarenko 2021-02-09 08:23:52 UTC
I tested with OCP 4.6.0-0.nightly-2021-02-05-192240, ocs-operator.v4.7.0-250.ci. I tried creating the snapshots via the UI. Both snapshots of cephfs and rbd-based PVCs failed to be created successfully, they were stuck in Pending state with the events "Generated from snapshot-controller:
Waiting for a snapshot default/cephfspvc-snapshot to be created by the CSI driver" and "Waiting for a snapshot default/rbdpvc-snapshot to be created by the CSI driver" correspondingly. 

Automated tests failed as well: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/295/

Comment 16 Mudit Agarwal 2021-02-09 14:03:38 UTC
@Elena can you please try with latest OCS version. 
ocs-operator.v4.7.0-250.ci is quite old and will be having this issue. 

Please try with ocs-operator.4.7.0-721.ci


PS: I am putting latest tag in Fixed in version field because I am not sure about the exact version where the builds started pointing to 4.6 sidecar.

Comment 21 errata-xmlrpc 2021-05-19 09:18:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Container Storage 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2041