Bug 1296457 - Sometimes Persistent Volume can not become available after it is created
Summary: Sometimes Persistent Volume can not become available after it is created
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.1.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Mark Turansky
QA Contact: Liang Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-07 10:14 UTC by Jianwei Hou
Modified: 2016-01-26 19:20 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-01-26 19:20:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:0070 0 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 3.1.1 bug fix and enhancement update 2016-01-27 00:12:41 UTC

Description Jianwei Hou 2016-01-07 10:14:51 UTC
Description of problem:
Create a persistent volume, monitor the master log, the log reports the Persistent Volume has become available. Run 'oc get pv', the Persistent Volume is still 'Pending'.

Version-Release number of selected component (if applicable):
[root@openshift-139 ~]# oc version
oc v3.1.1.0
kubernetes v1.1.0-origin-1107-g4c8e6f4


How reproducible:
Sometimes

Steps to Reproduce:
1. Create a Persistent Volume
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "accbed"
  },
  "spec": {
    "capacity": {
        "storage": "5Gi"
    },
    "accessModes": [ "ReadWriteOnce" ],
    "nfs": {
        "path": "/jhou",
        "server": "10.66.79.133"
    },
    "persistentVolumeReclaimPolicy": "Default"
  }
}

2. oc get pv
3. monitor master log, in my test env, log appended to /var/log/messge
4. oc get pv

Actual results:
After step 2:
[root@openshift-139 ~]# oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM     REASON    AGE
accbed    <none>    5Gi        RWO           Pending                       4m

After step 3:
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.324882   91129 persistentvolume_claim_binder_controller.go:196] Synchronizing PersistentVolume[accbed], current phase: Pending
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.325116   91129 persistentvolume_claim_binder_controller.go:256] PersistentVolume[accbed] is available
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.325145   91129 persistentvolume_claim_binder_controller.go:313] PersistentVolume[accbed] changing phase from Pending to Available
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.333521   91129 jwt.go:190] Could not retrieve token openshift-infra/pv-binder-controller-token-aqv0r for service account openshift-infra/pv-binder-controller: secrets "pv-binder-controller-token-aqv0r" not found


After step 4:
[root@openshift-139 ~]# date
Thu Jan  7 18:08:32 CST 2016
[root@openshift-139 ~]# oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM     REASON    AGE
accbed    <none>    5Gi        RWO           Pending                       4m


Expected results:
The PV should become available

Additional info:

Comment 1 Aleksandar Kostadinov 2016-01-08 22:20:44 UTC
Not sure if issues are related but with origin:
oc v1.1-686-g07abc42
kubernetes v1.1.0-origin-1107-g4c8e6f4

I'm observing that if PV is created after PVC, the PVC never gets Bound. Also if I have 2 PVCs with only one bound to PV, then removing the Bound PV does not cause second PVC to become Bound.

In short, if a PV cannot be bound on creation it stays in Pending mode forever, no matter that appropriate PV is created/freed.

Comment 2 Mark Turansky 2016-01-11 17:21:37 UTC
I have no been able to recreate this.

Is it still an issue with the latest OpenShift version?  Are there any more specific instructions?

Comment 3 Jianwei Hou 2016-01-13 05:22:44 UTC
I've been trying it several times, and I can't reproduce it manually. My only clue is that the pv-binder-controller-token under openshift-infra is deleted somehow. Maybe it's done by the automation testing. I'll keep an eye on it.

Comment 5 errata-xmlrpc 2016-01-26 19:20:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070


Note You need to log in before you can comment on or make changes to this bug.