Bug 1459461 - Volumes not getting deleted in Heketi-Backend
Summary: Volumes not getting deleted in Heketi-Backend
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Jan Safranek
QA Contact: Jianwei Hou
URL:
Whiteboard: aos-scalability-36
: 1461688 (view as bug list)
Depends On:
Blocks: 1445448 1463632
TreeView+ depends on / blocked
 
Reported: 2017-06-07 08:41 UTC by Tejas Chaphekar
Modified: 2023-09-14 03:58 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
no doc needed, this bug has never reached customers
Clone Of:
: 1463632 (view as bug list)
Environment:
Last Closed: 2017-08-10 05:26:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1716 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 09:02:50 UTC

Description Tejas Chaphekar 2017-06-07 08:41:31 UTC
Description of problem:
 Whenever oc delete pvc command is run, pvc gets deleted successfully, but the deletion of pv is failing

Version-Release number of selected component (if applicable):
 cns-3.6

How reproducible:

Everytime

Steps to Reproduce:
1. Create a SC
2. Create a PV, PVC
3. Delete a PVC

Actual results:

The volumes corresponding to PVC should be deleted

Expected results:

PVC is getting deleted but the PV goes into failed state

Additional info:

[root ~]# oc get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
storage-claim1   Bound     pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim2   Bound     pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim3   Bound     pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           fast           1h
[root ~]# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                            STORAGECLASS   REASON    AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim1   fast                     1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim2   fast                     1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           Delete          Bound     storage-project/storage-claim3   fast                     1h
[root ~]# oc delete pvc storage-claim3
persistentvolumeclaim "storage-claim3" deleted
[root ~]# oc get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
storage-claim1   Bound     pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim2   Bound     pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
[root ~]# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                            STORAGECLASS   REASON    AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim1   fast                     1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim2   fast                     1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           Delete          Failed    storage-project/storage-claim3   fast                     1h
[root ~]# oc describe pv pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Name:		pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Labels:		<none>
Annotations:	pv.beta.kubernetes.io/gid=2002
		pv.kubernetes.io/bound-by-controller=yes
		pv.kubernetes.io/provisioned-by=kubernetes.io/glusterfs
StorageClass:	fast
Status:		Failed
Claim:		storage-project/storage-claim3
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	2Gi
Message:	Volume has no class annotation
Source:
    Type:		Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:	glusterfs-dynamic-storage-claim3
    Path:		vol_6c6b19af3c63069854d33f57fce9c032
    ReadOnly:		false
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  33s		33s		1	persistent-volume-controller			Warning		VolumeFailedDelete	Volume has no class annotation

Comment 4 Mohamed Ashiq 2017-06-08 11:40:23 UTC
Fix is in kubernetes upstream but not yet in openshift upstream.

https://github.com/kubernetes/kubernetes/pull/44035/files

Issue is because annotations missing in the dynamically created pv's. 

Refer:
https://github.com/kubernetes/kubernetes/issues/43929

workaround:

oc edit pv <pv-name>

Add the below line in annotation section:

volume.beta.kubernetes.io/storage-class: "<strageclass-name>"

This will be fixed in openshift. I will keep track of the fix.

Comment 5 Humble Chirammal 2017-06-15 05:41:59 UTC
Tejas, Can you please check whether this issue is fixed in latest OCP builds ?

Comment 6 Humble Chirammal 2017-06-15 05:58:34 UTC
(In reply to Humble Chirammal from comment #5)
> Tejas, Can you please check whether this issue is fixed in latest OCP builds
> ?

Also please let me know the OCP build version used in your setup.

Comment 7 Jan Safranek 2017-06-15 10:11:23 UTC
This indeed looks like a bug in OpenShift, current 3.6 does not have https://github.com/kubernetes/kubernetes/pull/43982

Comment 8 Humble Chirammal 2017-06-15 10:16:13 UTC
Thanks Jan for update and filing PR # https://github.com/openshift/origin/pull/14667

Comment 9 Jan Safranek 2017-06-15 10:19:09 UTC
Origin PR: https://github.com/openshift/origin/pull/14667

Comment 10 Humble Chirammal 2017-06-16 05:40:22 UTC
Above PR is merged, so this issue should be fixed from next OCP 3.6 builds.

Comment 11 hchen 2017-06-16 16:13:10 UTC
*** Bug 1461688 has been marked as a duplicate of this bug. ***

Comment 13 Jianwei Hou 2017-06-29 08:52:40 UTC
Verified in v3.6.126.1

PV and volume are auto deleted after PVC is deleted.

Comment 15 errata-xmlrpc 2017-08-10 05:26:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Comment 16 Red Hat Bugzilla 2023-09-14 03:58:52 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.