Bug 1296457

Summary: Sometimes Persistent Volume can not become available after it is created
Product: OpenShift Container Platform Reporter: Jianwei Hou <jhou>
Component: StorageAssignee: Mark Turansky <mturansk>
Status: CLOSED ERRATA QA Contact: Liang Xia <lxia>
Severity: low Docs Contact:
Priority: low    
Version: 3.1.0CC: akostadi, aos-bugs, bleanhar, jhou, jokerman, pruan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-26 19:20:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jianwei Hou 2016-01-07 10:14:51 UTC
Description of problem:
Create a persistent volume, monitor the master log, the log reports the Persistent Volume has become available. Run 'oc get pv', the Persistent Volume is still 'Pending'.

Version-Release number of selected component (if applicable):
[root@openshift-139 ~]# oc version
oc v3.1.1.0
kubernetes v1.1.0-origin-1107-g4c8e6f4


How reproducible:
Sometimes

Steps to Reproduce:
1. Create a Persistent Volume
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "accbed"
  },
  "spec": {
    "capacity": {
        "storage": "5Gi"
    },
    "accessModes": [ "ReadWriteOnce" ],
    "nfs": {
        "path": "/jhou",
        "server": "10.66.79.133"
    },
    "persistentVolumeReclaimPolicy": "Default"
  }
}

2. oc get pv
3. monitor master log, in my test env, log appended to /var/log/messge
4. oc get pv

Actual results:
After step 2:
[root@openshift-139 ~]# oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM     REASON    AGE
accbed    <none>    5Gi        RWO           Pending                       4m

After step 3:
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.324882   91129 persistentvolume_claim_binder_controller.go:196] Synchronizing PersistentVolume[accbed], current phase: Pending
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.325116   91129 persistentvolume_claim_binder_controller.go:256] PersistentVolume[accbed] is available
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.325145   91129 persistentvolume_claim_binder_controller.go:313] PersistentVolume[accbed] changing phase from Pending to Available
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.333521   91129 jwt.go:190] Could not retrieve token openshift-infra/pv-binder-controller-token-aqv0r for service account openshift-infra/pv-binder-controller: secrets "pv-binder-controller-token-aqv0r" not found


After step 4:
[root@openshift-139 ~]# date
Thu Jan  7 18:08:32 CST 2016
[root@openshift-139 ~]# oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM     REASON    AGE
accbed    <none>    5Gi        RWO           Pending                       4m


Expected results:
The PV should become available

Additional info:

Comment 1 Aleksandar Kostadinov 2016-01-08 22:20:44 UTC
Not sure if issues are related but with origin:
oc v1.1-686-g07abc42
kubernetes v1.1.0-origin-1107-g4c8e6f4

I'm observing that if PV is created after PVC, the PVC never gets Bound. Also if I have 2 PVCs with only one bound to PV, then removing the Bound PV does not cause second PVC to become Bound.

In short, if a PV cannot be bound on creation it stays in Pending mode forever, no matter that appropriate PV is created/freed.

Comment 2 Mark Turansky 2016-01-11 17:21:37 UTC
I have no been able to recreate this.

Is it still an issue with the latest OpenShift version?  Are there any more specific instructions?

Comment 3 Jianwei Hou 2016-01-13 05:22:44 UTC
I've been trying it several times, and I can't reproduce it manually. My only clue is that the pv-binder-controller-token under openshift-infra is deleted somehow. Maybe it's done by the automation testing. I'll keep an eye on it.

Comment 5 errata-xmlrpc 2016-01-26 19:20:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070