Bug 1327531 - Dynamically provisioned PV and volume are left behind under rare conditions
Summary: Dynamically provisioned PV and volume are left behind under rare conditions
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Bradley Childs
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-15 10:23 UTC by Jianwei Hou
Modified: 2016-09-27 09:38 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-27 09:38:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC

Description Jianwei Hou 2016-04-15 10:23:41 UTC
Description of problem:
Create a PVC which dynamically provisions a PV, under some rare conditions, the PV will end up in failed status, at this time delete the PVC, the PV is left behind.

Version-Release number of selected component (if applicable):
openshift v3.2.0.15
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

How reproducible:
Always

Steps to Reproduce:
1. Use up openstack's all available volume storage

2. Create a PVC requesting provisioned PV: oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/cinder/dynamic-provisioning/pvc.json

3. PV was created in 'Failed' status with error: {"overLimit": {"message": "VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed Gigabytes quota. Requested 3G, quota is 100G and 100G has been consumed.", "code": 413}}

4. Delete the PVC

5. oc get pv

Actual results:
After step 5: The PV was not deleted.

NAME              CAPACITY   ACCESSMODES   STATUS    CLAIM           REASON    AGE
pv-cinder-axm2y   3Gi        RWO           Failed    jhou1/cinderc             29m

# oc describe pv pv-cinder-axm2y
Name:		pv-cinder-axm2y
Labels:		<none>
Status:		Failed
Claim:		jhou1/cinderc
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	3Gi
Message:	Expected HTTP response code [200 201] when accessing [POST http://<hidden>/volumes], but got 413 instead
{"overLimit": {"message": "VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed Gigabytes quota. Requested 3G, quota is 100G and 100G has been consumed.", "code": 413}}
Source:


If PV is provisioned but due to some reason it was not successfully attached to the node. When pod and PVC is deleted, PV and volume are also left behind. This was discovered in https://bugzilla.redhat.com/show_bug.cgi?id=1313210#c12

Expected results:
It is better to also clean up dirty PV and volume when the PVC is deleted.

Additional info:

Comment 1 Jan Safranek 2016-06-03 14:35:29 UTC
This should be fixed by controller refactoring in Kubernetes 1.3 - when provisioning fails, no PV in is created, therefore it can't enter Failed state.

Comment 2 Troy Dawson 2016-09-01 15:57:39 UTC
This is in OSE v3.3.0.28 or newer.

Comment 4 Jianwei Hou 2016-09-02 05:38:31 UTC
Verified on 
openshift v3.3.0.28
kubernetes v1.3.0+507d3a7

Now PV won't be created if provision fails.

Comment 6 errata-xmlrpc 2016-09-27 09:38:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933


Note You need to log in before you can comment on or make changes to this bug.