+++ This bug was initially created as a clone of Bug #1459461 +++
Description of problem:
Whenever oc delete pvc command is run, pvc gets deleted successfully, but the deletion of pv is failing
Version-Release number of selected component (if applicable):
cns-3.6
How reproducible:
Everytime
Steps to Reproduce:
1. Create a SC
2. Create a PV, PVC
3. Delete a PVC
Actual results:
The volumes corresponding to PVC should be deleted
Expected results:
PVC is getting deleted but the PV goes into failed state
Additional info:
[root ~]# oc get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
storage-claim1 Bound pvc-c9cd9320-4b38-11e7-ba89-005056848cc9 1Gi RWO fast 1h
storage-claim2 Bound pvc-d017751e-4b38-11e7-ba89-005056848cc9 1Gi RWO fast 1h
storage-claim3 Bound pvc-d65c5b26-4b38-11e7-ba89-005056848cc9 2Gi RWO fast 1h
[root ~]# oc get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9 1Gi RWO Delete Bound storage-project/storage-claim1 fast 1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9 1Gi RWO Delete Bound storage-project/storage-claim2 fast 1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9 2Gi RWO Delete Bound storage-project/storage-claim3 fast 1h
[root ~]# oc delete pvc storage-claim3
persistentvolumeclaim "storage-claim3" deleted
[root ~]# oc get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
storage-claim1 Bound pvc-c9cd9320-4b38-11e7-ba89-005056848cc9 1Gi RWO fast 1h
storage-claim2 Bound pvc-d017751e-4b38-11e7-ba89-005056848cc9 1Gi RWO fast 1h
[root ~]# oc get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-c9cd9320-4b38-11e7-ba89-005056848cc9 1Gi RWO Delete Bound storage-project/storage-claim1 fast 1h
pvc-d017751e-4b38-11e7-ba89-005056848cc9 1Gi RWO Delete Bound storage-project/storage-claim2 fast 1h
pvc-d65c5b26-4b38-11e7-ba89-005056848cc9 2Gi RWO Delete Failed storage-project/storage-claim3 fast 1h
[root ~]# oc describe pv pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Name: pvc-d65c5b26-4b38-11e7-ba89-005056848cc9
Labels: <none>
Annotations: pv.beta.kubernetes.io/gid=2002
pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/glusterfs
StorageClass: fast
Status: Failed
Claim: storage-project/storage-claim3
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 2Gi
Message: Volume has no class annotation
Source:
Type: Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
EndpointsName: glusterfs-dynamic-storage-claim3
Path: vol_6c6b19af3c63069854d33f57fce9c032
ReadOnly: false
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
33s 33s 1 persistent-volume-controller Warning VolumeFailedDelete Volume has no class annotation
--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-07 04:41:38 EDT ---
This bug is automatically being proposed for the current release of Container-Native Storage under active development, by setting the release flag 'cns‑3.6.0' to '?'.
If this bug should be proposed for a different release, please manually change the proposed release flag.
--- Additional comment from Atin Mukherjee on 2017-06-07 05:24:16 EDT ---
If this is done with brick multiplexing being enabled then I guess this is a duplicate of BZ 1444749 which now has been fixed in RHGS through BZ 1457219. CNS team needs to rebuild the image with glusterfs-3.8.4-27.
--- Additional comment from Mohamed Ashiq on 2017-06-08 06:52:41 EDT ---
(In reply to Atin Mukherjee from comment #2)
> If this is done with brick multiplexing being enabled then I guess this is a
> duplicate of BZ 1444749 which now has been fixed in RHGS through BZ 1457219.
> CNS team needs to rebuild the image with glusterfs-3.8.4-27.
This is not related to brick multiplexing. This because of annotations missing in the dynamically created pv's.
Refer:
https://github.com/kubernetes/kubernetes/issues/43929
workaround:
oc edit pv <pv-name>
Add the below line in annotation section:
volume.beta.kubernetes.io/storage-class: "<strageclass-name>"
This will be fixed in openshift. I will keep track of the fix.
--- Additional comment from Mohamed Ashiq on 2017-06-08 07:40:23 EDT ---
Fix is in kubernetes upstream but not yet in openshift upstream.
https://github.com/kubernetes/kubernetes/pull/44035/files
Issue is because annotations missing in the dynamically created pv's.
Refer:
https://github.com/kubernetes/kubernetes/issues/43929
workaround:
oc edit pv <pv-name>
Add the below line in annotation section:
volume.beta.kubernetes.io/storage-class: "<strageclass-name>"
This will be fixed in openshift. I will keep track of the fix.
--- Additional comment from Humble Chirammal on 2017-06-15 01:41:59 EDT ---
Tejas, Can you please check whether this issue is fixed in latest OCP builds ?
--- Additional comment from Humble Chirammal on 2017-06-15 01:58:34 EDT ---
(In reply to Humble Chirammal from comment #5)
> Tejas, Can you please check whether this issue is fixed in latest OCP builds
> ?
Also please let me know the OCP build version used in your setup.
--- Additional comment from Jan Safranek on 2017-06-15 06:11:23 EDT ---
This indeed looks like a bug in OpenShift, current 3.6 does not have https://github.com/kubernetes/kubernetes/pull/43982
--- Additional comment from Humble Chirammal on 2017-06-15 06:16:13 EDT ---
Thanks Jan for update and filing PR # https://github.com/openshift/origin/pull/14667
--- Additional comment from Jan Safranek on 2017-06-15 06:19:09 EDT ---
Origin PR: https://github.com/openshift/origin/pull/14667
--- Additional comment from Humble Chirammal on 2017-06-16 01:40:22 EDT ---
Above PR is merged, so this issue should be fixed from next OCP 3.6 builds.
--- Additional comment from on 2017-06-16 12:13:10 EDT ---
(In reply to Humble Chirammal from comment #4)
> This bug has been fixed in latest OCP version, iic, krk has verified it.https://bugzilla.redhat.com/show_bug.cgi?id=1444749#c38 . I am closing this bug for now and please feel free to reopen if accordingly.
*** This bug has been marked as a duplicate of bug 1444749 ***