Cause: Volume recycler was being called incorrectly.
Consequence: Volumes that were not being recycled when persistentVolumeReclaimPolicy was set to 'Recycle'
Fix: The recycler is now called correctly.
Result: Volumes are properly recycled when persistentVolumeReclaimPolicy is set to 'Recycle'
Description of problem:
On deletion of a pod and a pvc, the pv's status does not become 'available' when it's reclaim policy is 'recycle', it can not be bound to another pvc.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Create an nfs pv, create pvc and pod
"accessModes": [ "ReadWriteOnce" ],
"accessModes": [ "ReadWriteOnce" ],
2. Delete the pod and delete the PVC, the PV's status become 'released'
3. Wait for a moment at get the PV status again, it remains 'released'
4. Go to the NFS volume, verify it's data are deleted
5. Create a PVC again using the same json template
6. Get the PV status
After step 4:
Data on NFS volume are not deleted when the pod and pvc are deleted
After step 5:
The PVC can not bound the PV, PV is not available
After step 6:
The PV's status is 'Failed'
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON
gluster <none> 5368709120 RWO Bound jhou/glusterc
iscsi <none> 5368709120 RWO Bound jhou/iscsic
local type=local 5368709120 RWO Bound jhou/localc
nfs <none> 5368709120 RWO Failed jhou/nfsc
The data on nfs volume should be scrubbed, the PV should become available and can be bound to new PVC
The Kube version is too old for the recycler: kubernetes v0.17.1-804-g496be63
what is the output for "describe pod nfs"? What does that container's "docker inspect" look like? Any errors in the kubelet log?
Cut and pasted from an email from Jan Safranek who is working with gluster:
To follow up on current development, docker-1.6.2-14.el7 adds a new
boolean to SELinux policy. This should allow docker container to access
gluster volume (optionally with -P for persistent change):
setsebool virt_sandbox_use_fusefs 1
dwalsh told me docker-1.6.2-14.el7 is heading to RHEL 7.1.3, i.e. to be
released in couple of days (2015-Jun-23?) in RHEL7 extras channel.
So what we need is just to document the boolean and encourage customers
to check docker package version (and wait for 1.6.2-14).
Docker build: https://brewweb.devel.redhat.com/buildinfo?buildID=441835
*** This bug has been marked as a duplicate of bug 1231936 ***
sorry. closed the wrong one :)
I am looking into this one.
What does the container log say?
Also, what image does "docker ps -a" list for the scrubber pod? The scrubber pod will be named something like "pv-scrubber-nfs-ulzou".
I successfully tested the recycler code using a HostPath PV (have to enable HostPath via a SCC), so the code is working as expected.
NFS has specific permission issues to work around. See this: https://ci.openshift.redhat.com/openshift-docs-master-testing/latest/admin_guide/persistent_storage_nfs.html#selinux-and-nfs-export-settings
If your OpenShift installation is pulling only from approved docker registries, it could be recycling fails because it is pulling the wrong image.
Alternatively, if your install allows you to pull from other public registries, the currently used image (gcr.io/google_containers/busybox) would work if your NFS permissions are correct.
Success after PR 3351 above is applied.
Given a persistentVolumeReclaimPolicy is 'Recycle', after the pod and PVC are deleted, the PV became available again. The data on the nfs volume are scrubbed.
Also verified hostPath mode, works as expected. I'll mark it as verified when it's ON_QA.
Move it to verified according to test result in comment 9
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.