Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1293850 - Failed to delete dynamically provisioned PV when PVC is deleted
Failed to delete dynamically provisioned PV when PVC is deleted
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.1.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Jan Safranek
Liang Xia
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-23 04:32 EST by chaoyang
Modified: 2016-05-12 12:26 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 12:26:24 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:1064 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 3.2 security, bug fix, and enhancement update 2016-05-12 16:19:17 EDT

  None (edit)
Description chaoyang 2015-12-23 04:32:16 EST
Description of problem:
Failed to delete dynamically provisioned PV after deleting pod and PVC


Version-Release number of selected component (if applicable):
openshift v3.1.0.4
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

How reproducible:
80%

Steps to Reproduce:
1.Create a pvc 
{
  "kind": "PersistentVolumeClaim",
  "apiVersion": "v1",
  "metadata": {
    "name": "claim1",
    "annotations": {
      "volume.alpha.kubernetes.io/storage-class": "foo"
    }
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "3Gi"
      }
    }
  }
}

2.After pv and pvc is bound, creat a pod
kind: Pod
apiVersion: v1
metadata:
  name: mypod
  labels:
    name: frontendhttp
spec:
  containers:
    - name: myfrontend
      image: jhou/hello-openshift
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
      - mountPath: "/tmp"
        name: aws
  volumes:
    - name: aws
      persistentVolumeClaim:
        claimName: claim1
3.Delete pod and pvc, check pv status
PV is failed
4.Check the volume status from provider console
from aws web console, this volume is 'avaible'

Actual results:
pv is in failed status
pv-aws-0pcyt   <none>       3Gi        RWO           Failed      default/claim1             35m


Expected results:
pv should be sucessfully released and deleted. The volume should also be deleted from its provider

Additional info:
1.root@ip-172-18-6-204 ec2-user]# oc describe pv pv-aws-0pcyt
Name:		pv-aws-0pcyt
Labels:		<none>
Status:		Failed
Claim:		default/claim1
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	3Gi
Message:	Deletion error: error delete EBS volumes: VolumeInUse: Volume vol-18c790e5 is currently attached to i-5417fbe7
		status code: 400, request id: 
Source:
    Type:	AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:	aws://us-east-1d/vol-18c790e5
    FSType:	ext4
    Partition:	0
    ReadOnly:	false
2. This issue is also reproducible for Cinder
Comment 1 Jan Safranek 2016-01-04 09:01:29 EST
I think we hit a race here - the pod is still running (or it's being slowly deleted) at the point when volume controller deletes the volume. We need some way how to retry the volume deletion if the first attempt did not succeed. Or synchronize pod and pvc deletion.
Comment 2 Bradley Childs 2016-01-04 20:20:15 EST
> Expected results:
> pv should be sucessfully released and deleted. The volume should also be deleted from its provider

Just to clarify the use case... The PV should be deleted (bug) but not the the actual volume & data. If you have a PV pointing at an AWS volume the PV should delete but not the physical AWS volume and data.
Comment 3 chaoyang 2016-01-05 02:51:06 EST
Thanks.
Will update the test case results
Comment 4 Jan Safranek 2016-01-05 04:07:08 EST
(In reply to Bradley Childs from comment #2)
> > Expected results:
> > pv should be sucessfully released and deleted. The volume should also be deleted from its provider
> 
> Just to clarify the use case... The PV should be deleted (bug) but not the
> the actual volume & data. If you have a PV pointing at an AWS volume the PV
> should delete but not the physical AWS volume and data.

No, dynamically created AWS EBS volumes _should_ be deleted when user deletes appropriate claim that created id. IMO that's the point of dynamic provisioning - create and _delete_ volumes on demand.
Comment 5 Jianwei Hou 2016-01-05 06:07:08 EST
(In reply to Jan Safranek from comment #4)
> (In reply to Bradley Childs from comment #2)
> > > Expected results:
> > > pv should be sucessfully released and deleted. The volume should also be deleted from its provider
> > 
> > Just to clarify the use case... The PV should be deleted (bug) but not the
> > the actual volume & data. If you have a PV pointing at an AWS volume the PV
> > should delete but not the physical AWS volume and data.
> 
> No, dynamically created AWS EBS volumes _should_ be deleted when user
> deletes appropriate claim that created id. IMO that's the point of dynamic
> provisioning - create and _delete_ volumes on demand.

Yes, I think so. When I was testing cinder for this feature, if pvc is deleted, the pv will be deleted too, and the physical cinder volume will be deleted from openstack as well.
Comment 6 Bradley Childs 2016-01-05 09:50:52 EST
Yes this was my mistake- the volume is deleted when it's not set to retain.  The default / unspecified value should be to retain though.
Comment 7 Mark Turansky 2016-01-11 10:33:23 EST
Reassigning to Jan for EBS testing.
Comment 8 Jan Safranek 2016-01-12 03:35:00 EST
Kubernetes PR: https://github.com/kubernetes/kubernetes/pull/19365
Comment 9 Jan Safranek 2016-02-09 09:39:23 EST
Origin PR merged
Comment 10 chaoyang 2016-02-15 22:01:33 EST
Verification is passed on
oc v1.1.2-274-g6187dc3
kubernetes v1.2.0-origin
Comment 12 errata-xmlrpc 2016-05-12 12:26:24 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064

Note You need to log in before you can comment on or make changes to this bug.