Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1296457 - Sometimes Persistent Volume can not become available after it is created
Sometimes Persistent Volume can not become available after it is created
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.1.0
Unspecified Unspecified
low Severity low
: ---
: ---
Assigned To: Mark Turansky
Liang Xia
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-07 05:14 EST by Jianwei Hou
Modified: 2016-01-26 14:20 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-26 14:20:50 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:0070 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 3.1.1 bug fix and enhancement update 2016-01-26 19:12:41 EST

  None (edit)
Description Jianwei Hou 2016-01-07 05:14:51 EST
Description of problem:
Create a persistent volume, monitor the master log, the log reports the Persistent Volume has become available. Run 'oc get pv', the Persistent Volume is still 'Pending'.

Version-Release number of selected component (if applicable):
[root@openshift-139 ~]# oc version
oc v3.1.1.0
kubernetes v1.1.0-origin-1107-g4c8e6f4


How reproducible:
Sometimes

Steps to Reproduce:
1. Create a Persistent Volume
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "accbed"
  },
  "spec": {
    "capacity": {
        "storage": "5Gi"
    },
    "accessModes": [ "ReadWriteOnce" ],
    "nfs": {
        "path": "/jhou",
        "server": "10.66.79.133"
    },
    "persistentVolumeReclaimPolicy": "Default"
  }
}

2. oc get pv
3. monitor master log, in my test env, log appended to /var/log/messge
4. oc get pv

Actual results:
After step 2:
[root@openshift-139 ~]# oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM     REASON    AGE
accbed    <none>    5Gi        RWO           Pending                       4m

After step 3:
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.324882   91129 persistentvolume_claim_binder_controller.go:196] Synchronizing PersistentVolume[accbed], current phase: Pending
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.325116   91129 persistentvolume_claim_binder_controller.go:256] PersistentVolume[accbed] is available
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.325145   91129 persistentvolume_claim_binder_controller.go:313] PersistentVolume[accbed] changing phase from Pending to Available
Jan  7 18:03:36 openshift-139 atomic-openshift-master: I0107 18:03:36.333521   91129 jwt.go:190] Could not retrieve token openshift-infra/pv-binder-controller-token-aqv0r for service account openshift-infra/pv-binder-controller: secrets "pv-binder-controller-token-aqv0r" not found


After step 4:
[root@openshift-139 ~]# date
Thu Jan  7 18:08:32 CST 2016
[root@openshift-139 ~]# oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM     REASON    AGE
accbed    <none>    5Gi        RWO           Pending                       4m


Expected results:
The PV should become available

Additional info:
Comment 1 Aleksandar Kostadinov 2016-01-08 17:20:44 EST
Not sure if issues are related but with origin:
oc v1.1-686-g07abc42
kubernetes v1.1.0-origin-1107-g4c8e6f4

I'm observing that if PV is created after PVC, the PVC never gets Bound. Also if I have 2 PVCs with only one bound to PV, then removing the Bound PV does not cause second PVC to become Bound.

In short, if a PV cannot be bound on creation it stays in Pending mode forever, no matter that appropriate PV is created/freed.
Comment 2 Mark Turansky 2016-01-11 12:21:37 EST
I have no been able to recreate this.

Is it still an issue with the latest OpenShift version?  Are there any more specific instructions?
Comment 3 Jianwei Hou 2016-01-13 00:22:44 EST
I've been trying it several times, and I can't reproduce it manually. My only clue is that the pv-binder-controller-token under openshift-infra is deleted somehow. Maybe it's done by the automation testing. I'll keep an eye on it.
Comment 5 errata-xmlrpc 2016-01-26 14:20:50 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070

Note You need to log in before you can comment on or make changes to this bug.