Red Hat Bugzilla – Bug 1388316
Failed to provision GlusterFS PV/volume with StorageClass using secret + namespace
Last modified: 2017-03-08 13:43 EST
Description of problem: Create a StorageClass and configure its parameters to use secret + namespace, the PV creation fails with error: Failed to provision volume with StorageClass "glusterprovisioner1": failed to get secret from ["jhou"/"heketi-secret"] But the secret is present in the target namespace: Version-Release number of selected component (if applicable): openshift v3.4.0.15+9c963ec kubernetes v1.4.0+776c994 etcd 3.1.0-alpha.1 How reproducible: Always Steps to Reproduce: 1. Create a secret in a target namespace 2. Update the StorageClass's parameters field, use secretName and secretNamespace(not restuserkey) 3. Create the StorageClass 4. Create a PVC that uses this StorageClass as provisioner Actual results: No PV provisioned. # oc describe pvc glusterc Name: glusterc Namespace: jhou StorageClass: glusterprovisioner1 Status: Pending Volume: Labels: <none> Capacity: Access Modes: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 6m 14s 27 {persistentvolume-controller } Warning ProvisioningFailed Failed to provision volume with StorageClass "glusterprovisioner1": failed to get secret from ["jhou"/"heketi-secret"] # oc get secrets heketi-secret -n jhou NAME TYPE DATA AGE heketi-secret Opaque 1 4m # oc get storageclass -o yaml apiVersion: v1 items: - apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: creationTimestamp: 2016-10-25T05:51:05Z name: glusterprovisioner1 resourceVersion: "8078" selfLink: /apis/storage.k8s.io/v1beta1/storageclasses/glusterprovisioner1 uid: 04dbdd54-9a77-11e6-9420-0e4f3633a564 parameters: resturl: http://<hidden> restuser: admin secretName: heketi-secret secretNamespace: jhou provisioner: kubernetes.io/glusterfs kind: List metadata: {} Expected results: Could provision PV/volume successfully Additional info: This feature was tested upstream, test version: Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.1.444+1d323adade5fc6", GitCommit:"1d323adade5fc6ecd898908d2c6cb391c8748835", GitTreeState:"clean", BuildDate:"2016-10-18T02:54:24Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.1.444+1d323adade5fc6", GitCommit:"1d323adade5fc6ecd898908d2c6cb391c8748835", GitTreeState:"clean", BuildDate:"2016-10-18T02:54:24Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
The process changed. The new
Looks like both Ceph ( https://bugzilla.redhat.com/show_bug.cgi?id=1388368 ) and GlusterFS fails with an error to fetch secrets in Openshift. iic, this was tested in kubernetes upstream and passed. I am not sure whether some special setup is required for openshift. I am adding Jan and Huamin for their thoughts.
Gluster provisioner seems not to have access to secrets. * I filled https://github.com/kubernetes/kubernetes/pull/35615 to get nicer message in oc describe pvc: Failed to provision volume with StorageClass "glusterprovisioner1": failed to get secret default/heketi-secret: User "system:serviceaccount:openshift-infra:pv-binder-controller" cannot get secrets in project "default"] (this not really *required* in Origin in 3.4) * I filled https://github.com/openshift/origin/pull/11581 with system:serviceaccount:openshift-infra:pv-binder-controller permission updates
*** Bug 1388368 has been marked as a duplicate of this bug. ***
(In reply to Jan Safranek from comment #3) > Gluster provisioner seems not to have access to secrets. > > * I filled https://github.com/kubernetes/kubernetes/pull/35615 to get nicer > message in oc describe pvc: > Failed to provision volume with StorageClass "glusterprovisioner1": failed > to get secret default/heketi-secret: User > "system:serviceaccount:openshift-infra:pv-binder-controller" cannot get > secrets in project "default"] > > (this not really *required* in Origin in 3.4) > > * I filled https://github.com/openshift/origin/pull/11581 with > system:serviceaccount:openshift-infra:pv-binder-controller permission updates Thanks Jan!!
This has been merged into ose and is in OSE v3.4.0.23 or newer.
This is still reproduced on: openshift v3.4.0.23+24b1a58 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 # oc describe pvc glusterc Name: glusterc Namespace: jhou StorageClass: glusterprovisioner1 Status: Pending Volume: Labels: <none> Capacity: Access Modes: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 30s 10s 3 {persistentvolume-controller } Warning ProvisioningFailed Failed to provision volume with StorageClass "glusterprovisioner1": failed to get secret from ["jhou"/"heketi-secret"] # oc get secret/heketi-secret -n jhou NAME TYPE DATA AGE heketi-secret Opaque 1 14m
The secret type must match the provisioner, not be Opaque
As Jordan wrote, the secret type must be either "kubernetes.io/glusterfs" or "kubernetes.io/rbd" respectively. Relevant docs change is here: https://github.com/kubernetes/kubernetes.github.io/pull/1594/files Sorry, I should have let you know earlier.
@jliggitt @jsafrane Thank you! With secret type being "kubernetes.io/glusterfs", the issue is gone! This bug can be verified now. Could you please change its status to ON_QA?
moving to on_qa per comment.
This is fixed according to comment 15.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066