Bug 1463632 - Volumes not getting deleted in Heketi-Backend [NEEDINFO]
Volumes not getting deleted in Heketi-Backend
Status: CLOSED DUPLICATE of bug 1462138
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: kubernetes (Show other bugs)
cns-3.6
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Humble Chirammal
Tejas Chaphekar
:
Depends On: 1459461
Blocks: 1445448
  Show dependency treegraph
 
Reported: 2017-06-21 07:34 EDT by Prasanth
Modified: 2017-08-08 00:59 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1459461
Environment:
Last Closed: 2017-06-22 07:32:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
hchiramm: needinfo? (pprakash)


Attachments (Terms of Use)

  None (edit)
Description Prasanth 2017-06-21 07:34:30 EDT
+++ This bug was initially created as a clone of Bug #1459461 +++

Description of problem:
 Whenever oc delete pvc command is run, pvc gets deleted successfully, but the deletion of pv is failing

Version-Release number of selected component (if applicable):
 cns-3.6

How reproducible:

Everytime

Steps to Reproduce:
1. Create a SC
2. Create a PV, PVC
3. Delete a PVC

Actual results:

The volumes corresponding to PVC should be deleted

Expected results:

PVC is getting deleted but the PV goes into failed state

Additional info:

[root ~]# oc get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
storage-claim1   Bound     pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim2   Bound     pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim3   Bound     pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           fast           1h
[root ~]# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                            STORAGECLASS   REASON    AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim1   fast                     1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim2   fast                     1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           Delete          Bound     storage-project/storage-claim3   fast                     1h
[root ~]# oc delete pvc storage-claim3
persistentvolumeclaim "storage-claim3" deleted
[root ~]# oc get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
storage-claim1   Bound     pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
storage-claim2   Bound     pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           fast           1h
[root ~]# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                            STORAGECLASS   REASON    AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim1   fast                     1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9   1Gi        RWO           Delete          Bound     storage-project/storage-claim2   fast                     1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9   2Gi        RWO           Delete          Failed    storage-project/storage-claim3   fast                     1h
[root ~]# oc describe pv pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Name:		pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Labels:		<none>
Annotations:	pv.beta.kubernetes.io/gid=2002
		pv.kubernetes.io/bound-by-controller=yes
		pv.kubernetes.io/provisioned-by=kubernetes.io/glusterfs
StorageClass:	fast
Status:		Failed
Claim:		storage-project/storage-claim3
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	2Gi
Message:	Volume has no class annotation
Source:
    Type:		Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:	glusterfs-dynamic-storage-claim3
    Path:		vol_6c6b19af3c63069854d33f57fce9c032
    ReadOnly:		false
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  33s		33s		1	persistent-volume-controller			Warning		VolumeFailedDelete	Volume has no class annotation

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-07 04:41:38 EDT ---

This bug is automatically being proposed for the current release of Container-Native Storage under active development, by setting the release flag 'cns‑3.6.0' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Atin Mukherjee on 2017-06-07 05:24:16 EDT ---

If this is done with brick multiplexing being enabled then I guess this is a duplicate of BZ 1444749 which now has been fixed in RHGS through BZ 1457219. CNS team needs to rebuild the image with glusterfs-3.8.4-27.

--- Additional comment from Mohamed Ashiq on 2017-06-08 06:52:41 EDT ---

(In reply to Atin Mukherjee from comment #2)
> If this is done with brick multiplexing being enabled then I guess this is a
> duplicate of BZ 1444749 which now has been fixed in RHGS through BZ 1457219.
> CNS team needs to rebuild the image with glusterfs-3.8.4-27.

This is not related to brick multiplexing. This because of annotations missing in the dynamically created pv's. 

Refer:
https://github.com/kubernetes/kubernetes/issues/43929

workaround:

oc edit pv <pv-name>

Add the below line in annotation section:

volume.beta.kubernetes.io/storage-class: "<strageclass-name>"

This will be fixed in openshift. I will keep track of the fix.

--- Additional comment from Mohamed Ashiq on 2017-06-08 07:40:23 EDT ---

Fix is in kubernetes upstream but not yet in openshift upstream.

https://github.com/kubernetes/kubernetes/pull/44035/files

Issue is because annotations missing in the dynamically created pv's. 

Refer:
https://github.com/kubernetes/kubernetes/issues/43929

workaround:

oc edit pv <pv-name>

Add the below line in annotation section:

volume.beta.kubernetes.io/storage-class: "<strageclass-name>"

This will be fixed in openshift. I will keep track of the fix.

--- Additional comment from Humble Chirammal on 2017-06-15 01:41:59 EDT ---

Tejas, Can you please check whether this issue is fixed in latest OCP builds ?

--- Additional comment from Humble Chirammal on 2017-06-15 01:58:34 EDT ---

(In reply to Humble Chirammal from comment #5)
> Tejas, Can you please check whether this issue is fixed in latest OCP builds
> ?

Also please let me know the OCP build version used in your setup.

--- Additional comment from Jan Safranek on 2017-06-15 06:11:23 EDT ---

This indeed looks like a bug in OpenShift, current 3.6 does not have https://github.com/kubernetes/kubernetes/pull/43982

--- Additional comment from Humble Chirammal on 2017-06-15 06:16:13 EDT ---

Thanks Jan for update and filing PR # https://github.com/openshift/origin/pull/14667

--- Additional comment from Jan Safranek on 2017-06-15 06:19:09 EDT ---

Origin PR: https://github.com/openshift/origin/pull/14667

--- Additional comment from Humble Chirammal on 2017-06-16 01:40:22 EDT ---

Above PR is merged, so this issue should be fixed from next OCP 3.6 builds.

--- Additional comment from  on 2017-06-16 12:13:10 EDT ---
Comment 4 Humble Chirammal 2017-06-22 07:29:11 EDT
This bug has been fixed in latest OCP version, iic, krk has verified it.
Comment 5 Humble Chirammal 2017-06-22 07:32:17 EDT
(In reply to Humble Chirammal from comment #4)
> This bug has been fixed in latest OCP version, iic, krk has verified it.

https://bugzilla.redhat.com/show_bug.cgi?id=1444749#c38 . I am closing this bug for now and please feel free to reopen if accordingly.

*** This bug has been marked as a duplicate of bug 1444749 ***
Comment 6 Humble Chirammal 2017-06-22 07:45:47 EDT

*** This bug has been marked as a duplicate of bug 1462138 ***

Note You need to log in before you can comment on or make changes to this bug.