Bug 1459461 - Volumes not getting deleted in Heketi-Backend [NEEDINFO]
Volumes not getting deleted in Heketi-Backend
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Jan Safranek
Jianwei Hou
aos-scalability-36
:
: 1461688 (view as bug list)
Depends On:
Blocks: 1445448 1463632
  Show dependency treegraph
 
Reported: 2017-06-07 04:41 EDT by Tejas Chaphekar
Modified: 2017-08-16 15 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: No Doc Update
Doc Text:
no doc needed, this bug has never reached customers
Story Points: ---
Clone Of:
: 1463632 (view as bug list)
Environment:
Last Closed: 2017-08-10 01:26:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
hchiramm: needinfo? (tchaphek)


Attachments (Terms of Use)

  None (edit)
Description Tejas Chaphekar 2017-06-07 04:41:31 EDT
Description of problem:
 Whenever oc delete pvc command is run, pvc gets deleted successfully, but the deletion of pv is failing

Version-Release number of selected component (if applicable):
 cns-3.6

How reproducible:

Everytime

Steps to Reproduce:
1. Create a SC
2. Create a PV, PVC
3. Delete a PVC

Actual results:

The volumes corresponding to PVC should be deleted

Expected results:

PVC is getting deleted but the PV goes into failed state

Additional info:

[root ~]# oc get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
storage-claim1   Bound     pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim2   Bound     pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim3   Bound     pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           fast           1h
[root ~]# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                            STORAGECLASS   REASON    AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim1   fast                     1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim2   fast                     1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           Delete          Bound     storage-project/storage-claim3   fast                     1h
[root ~]# oc delete pvc storage-claim3
persistentvolumeclaim "storage-claim3" deleted
[root ~]# oc get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
storage-claim1   Bound     pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim2   Bound     pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
[root ~]# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                            STORAGECLASS   REASON    AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim1   fast                     1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim2   fast                     1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           Delete          Failed    storage-project/storage-claim3   fast                     1h
[root ~]# oc describe pv pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Name:		pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Labels:		<none>
Annotations:	pv.beta.kubernetes.io/gid=2002
		pv.kubernetes.io/bound-by-controller=yes
		pv.kubernetes.io/provisioned-by=kubernetes.io/glusterfs
StorageClass:	fast
Status:		Failed
Claim:		storage-project/storage-claim3
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	2Gi
Message:	Volume has no class annotation
Source:
    Type:		Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:	glusterfs-dynamic-storage-claim3
    Path:		vol_6c6b19af3c63069854d33f57fce9c032
    ReadOnly:		false
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  33s		33s		1	persistent-volume-controller			Warning		VolumeFailedDelete	Volume has no class annotation
Comment 4 Mohamed Ashiq 2017-06-08 07:40:23 EDT
Fix is in kubernetes upstream but not yet in openshift upstream.

https://github.com/kubernetes/kubernetes/pull/44035/files

Issue is because annotations missing in the dynamically created pv's. 

Refer:
https://github.com/kubernetes/kubernetes/issues/43929

workaround:

oc edit pv <pv-name>

Add the below line in annotation section:

volume.beta.kubernetes.io/storage-class: "<strageclass-name>"

This will be fixed in openshift. I will keep track of the fix.
Comment 5 Humble Chirammal 2017-06-15 01:41:59 EDT
Tejas, Can you please check whether this issue is fixed in latest OCP builds ?
Comment 6 Humble Chirammal 2017-06-15 01:58:34 EDT
(In reply to Humble Chirammal from comment #5)
> Tejas, Can you please check whether this issue is fixed in latest OCP builds
> ?

Also please let me know the OCP build version used in your setup.
Comment 7 Jan Safranek 2017-06-15 06:11:23 EDT
This indeed looks like a bug in OpenShift, current 3.6 does not have https://github.com/kubernetes/kubernetes/pull/43982
Comment 8 Humble Chirammal 2017-06-15 06:16:13 EDT
Thanks Jan for update and filing PR # https://github.com/openshift/origin/pull/14667
Comment 9 Jan Safranek 2017-06-15 06:19:09 EDT
Origin PR: https://github.com/openshift/origin/pull/14667
Comment 10 Humble Chirammal 2017-06-16 01:40:22 EDT
Above PR is merged, so this issue should be fixed from next OCP 3.6 builds.
Comment 11 hchen 2017-06-16 12:13:10 EDT
*** Bug 1461688 has been marked as a duplicate of this bug. ***
Comment 13 Jianwei Hou 2017-06-29 04:52:40 EDT
Verified in v3.6.126.1

PV and volume are auto deleted after PVC is deleted.
Comment 15 errata-xmlrpc 2017-08-10 01:26:47 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.