Bug 1860891
| Summary: | Deleting csi-cephfsplugin-provisioner pod during PVC deletion leaves behind PV in Released state | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Container Storage | Reporter: | Jilju Joy <jijoy> | |
| Component: | csi-driver | Assignee: | Madhu Rajanna <mrajanna> | |
| Status: | CLOSED ERRATA | QA Contact: | Jilju Joy <jijoy> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 4.4 | CC: | hchiramm, madam, mrajanna, muagarwa, nberry, ocs-bugs, sagrawal, srangana | |
| Target Milestone: | --- | Keywords: | Automation, Regression | |
| Target Release: | OCS 4.5.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | 4.5.0-515.ci | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1861257 (view as bug list) | Environment: | ||
| Last Closed: | 2020-09-15 10:18:25 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1861257 | |||
|
Description
Jilju Joy
2020-07-27 11:14:24 UTC
Yes, it looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1793387 because here also deletion is failing with ENOENT. @Jilju, do we have the system intact. Can we check whether the subvolume (csi-vol-185cfa6b-cd1d-11ea-ae58-0a580a83001) is present or not on the backing cephfs volume? (In reply to Mudit Agarwal from comment #3) > Yes, it looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1793387 > because here also deletion is failing with ENOENT. > > @Jilju, do we have the system intact. Sorry, the cluster is not available now. It was destroyed after automation execution. > Can we check whether the subvolume > (csi-vol-185cfa6b-cd1d-11ea-ae58-0a580a83001) is present or not on the > backing cephfs volume? In some ceph version, if the subvolume is not present, the ceph returns does not exist and in some version not found error message. sh-4.2# ceph version ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) sh-4.2# ceph fs subvolume getpath myfs csi-vol-a24a3d97-c7f4-11ea-8cfc-0242ac110012 --group_name csi Error ENOENT: subvolume 'csi-vol-a24a3d97-c7f4-11ea-8cfc-0242ac110012' does not exist sh-4.2# ceph version ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) In ceph 14.2.4 sh-4.2# ceph fs subvolume getpath myfs testing --group_name=csi Error ENOENT: Subvolume 'testing' not found This is a regression from ceph fs core side, we have fixed in ceph-csi to handle both cases in https://github.com/ceph/ceph-csi/pull/1247. will backport it to downstream Thanks Madhu. This can be hit based upon the ceph version but it is a corner case and would mostly be hit in a disruptive environment like this. Not a blocker for OCS4.5, can be pushed to OCS4.6. downstream PR for both 4.4 [1] and 4.5 [2] [1] https://github.com/openshift/ceph-csi/pull/5 [2] https://github.com/openshift/ceph-csi/pull/6 Yes above one should fix the issue. Providing the devel_ack because it is a regression and we already have a simple fix for the issue. Also, looks like this particular Test case will always fail with this ceph version that makes it a test blocker? @Madhu, please wait for all the acks before the merge. Verified in version: Cluster Version 4.5.0-0.nightly-2020-08-03-123303 OCS operator v4.5.0-515.ci rook_csi_ceph cephcsi@sha256:244099ffc77fe965cd258e105aeff127de08673830a679ecb2525d9220e161fb rook_ceph rook-ceph@sha256:6aaf689232cb7fcb44e37dc1c34b17c7cc81d5fe244cfb4277fafdb5a3865ee4 Executed test cases: tests/manage/pv_services/test_pvc_disruptive.py::TestPVCDisruption::test_pvc_disruptive[CephFileSystem-create_pvc-cephfsplugin_provisioner] tests/manage/pv_services/test_resource_deletion_during_pod_pvc_deletion.py::TestDeleteResourceDuringPodPvcDeletion::test_disruptive_during_pod_pvc_deletion[CephFileSystem-delete_pvcs-cephfsplugin_provisioner] tests/manage/pv_services/test_resource_deletion_during_pvc_pod_creation_and_io.py::TestResourceDeletionDuringCreationOperations::test_resource_deletion_during_pvc_pod_creation_and_io[CephFileSystem-cephfsplugin_provisioner] tests/manage/pv_services/test_resource_deletion_during_pvc_pod_deletion_and_io.py::TestResourceDeletionDuringMultipleDeleteOperations::test_disruptive_during_pod_pvc_deletion_and_io[CephFileSystem-cephfsplugin_provisioner] Test run - https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/10551/ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3754 |