Bug 1415624

Clone for 3.4.x backport

+++ This bug was initially created as a clone of Bug #1392338 +++

Description of problem:
Version-Release number of selected component (if applicable):
openshift v3.3.1.3
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

How reproducible:
1. Create PV in OS namespace.
2. Create new project 
3. Create new MYSQL persistent instances 
4. Delete project.

Steps to Reproduce:
1. Create some NFS persistent volumes on NFS server and in OS (using Recycle). (at least 8)
2. Create new project, add new MYSQL persistent instances (8 in my case)
3. The pods with MYSQL will be deployed and get assigned to a persistent volume using a pvc.
4. Once all pods are up and ready, delete the project, either using web interface or oc delete.
5. Watch PV status => oc get pv -w
6. Some PV get status "AVAILABLE" before the recycler complete the data scrubbing + sometimes the recycler pod didnt start and flag the volume as AVAILABLE while data are still present on the NFS volume.


Actual results:
Some volumes get correctly scrubbed.
Some volumes are not at all scrubbed and their status is changed from Bound to Recycle to Available.

The oc get po -w in openshift-infra namespace list strange pod creation when failing to recycle :
recycler-for-volume1 0/1 Completed
recycler-for-volume1 0/1 Terminating
recycler-for-volume1 0/1 Terminating

in oc get pv -w:
volume1 10Gi RWO,RWX Released projecta/dbmysql 
volume1 10Gi RWO,RWX Released 
volume1 10Gi RWO,RWX Available

=> NFS share still has folder and data.

Expected results:
A volume cannot be in status Available before the recycler successfully delete all data on the NFS volume.

Additional info:
There is no permission issue related here as some volume the data scrubbing is correctly done, based on my observation the issues are related to the recycler pod creation and the PV status update. I feel like the recycler pod creation is not always started correctly like "Pending" -> "Container Creating" -> "Running" -> "Completed" -> "Terminating" 

For some reason sometimes the creation is no done and we have only Completed + terminating in the logs or simply no recycler creation.

--- Additional comment from Jan Safranek on 2016-11-28 07:37:39 EST ---

Origin pull request: https://github.com/openshift/origin/pull/12041