OpenShift ships its own recycler pod which deletes all files on a persistent volume, typically a NFS share. Since it is a NFS share, it can be mounted to another pods that can create files while the recycler removes them. Typically, user deletes a namespace with both pods and PVCs and dying MySQL pod there may prevent some files from being recycled (see bug #1392338).

As result, the recycler may finish with the NFS share not empty. The recycler should check that the NFS share is empty and return an error if not. This way, Kubernetes will retry recycling after a while, hoping that the dying pod is already terminated.

Version-Release number of selected component (if applicable):
openshift v3.3.1.3
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

How reproducible:
~50%

Steps to Reproduce:
1. create 10 separate NFS shares, 10 PVs and 10 claims
2. create 10 pods, each using one claim. The pods randomly create files on the its PVs.
3. delete the pods and claims.

Actual results:
Recycler is started before the pods are killed. Therefore recycler will try to remove files from a NFS share, while the dying pod is creating new ones.
-> recycler finished with non-empty NFS shares and reports success

Expected results:
recycler should report error and it should be restarted by Kubernetes in a while to remove remaining files.

Additional info:
Of course, this is not 100% safe, a dying pod may create a file just *after* the recycler pod checks that the NFS share is empty. I'll fill a separate bug to prevent recycler to run if there is a pod that uses the same PV. However, a check in recycler should be super simple and easy to implement and it should help in 99.9% of cases.