Bug 1388316 - Failed to provision GlusterFS PV/volume with StorageClass using secret + namespace
Summary: Failed to provision GlusterFS PV/volume with StorageClass using secret + name...
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Humble Chirammal
QA Contact: Jianwei Hou
URL:
Whiteboard:
Keywords:
: 1388368 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-25 06:05 UTC by Jianwei Hou
Modified: 2017-03-08 18:43 UTC (History)
8 users (show)

(edit)
undefined
Clone Of:
(edit)
Last Closed: 2017-01-18 12:44:38 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0066 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4 RPM Release Advisory 2017-01-18 17:23:26 UTC

Description Jianwei Hou 2016-10-25 06:05:15 UTC
Description of problem:
Create a StorageClass and configure its parameters to use secret + namespace, the PV creation fails with error: Failed to provision volume with StorageClass
"glusterprovisioner1": failed to get secret from ["jhou"/"heketi-secret"]

But the secret is present in the target namespace:

Version-Release number of selected component (if applicable):
openshift v3.4.0.15+9c963ec
kubernetes v1.4.0+776c994
etcd 3.1.0-alpha.1

How reproducible:
Always

Steps to Reproduce:
1. Create a secret in a target namespace
2. Update the StorageClass's parameters field, use secretName and secretNamespace(not restuserkey)
3. Create the StorageClass
4. Create a PVC that uses this StorageClass as provisioner

Actual results:
No PV provisioned.

# oc describe pvc glusterc            
Name:		glusterc
Namespace:	jhou
StorageClass:	glusterprovisioner1
Status:		Pending
Volume:		
Labels:		<none>
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  6m		14s		27	{persistentvolume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass
"glusterprovisioner1": failed to get secret from ["jhou"/"heketi-secret"]

# oc get secrets heketi-secret -n jhou
NAME            TYPE      DATA      AGE
heketi-secret   Opaque    1         4m

# oc get storageclass -o yaml
apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1beta1
  kind: StorageClass
  metadata:
    creationTimestamp: 2016-10-25T05:51:05Z
    name: glusterprovisioner1
    resourceVersion: "8078"
    selfLink: /apis/storage.k8s.io/v1beta1/storageclasses/glusterprovisioner1
    uid: 04dbdd54-9a77-11e6-9420-0e4f3633a564
  parameters:
    resturl: http://<hidden>
    restuser: admin
    secretName: heketi-secret
    secretNamespace: jhou
  provisioner: kubernetes.io/glusterfs
kind: List
metadata: {}


Expected results:
Could provision PV/volume successfully

Additional info:
This feature was tested upstream, test version:
Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.1.444+1d323adade5fc6", GitCommit:"1d323adade5fc6ecd898908d2c6cb391c8748835", GitTreeState:"clean", BuildDate:"2016-10-18T02:54:24Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.1.444+1d323adade5fc6", GitCommit:"1d323adade5fc6ecd898908d2c6cb391c8748835", GitTreeState:"clean", BuildDate:"2016-10-18T02:54:24Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Comment 1 hchen 2016-10-25 14:12:34 UTC
The process changed. The new

Comment 2 Humble Chirammal 2016-10-26 10:40:57 UTC
Looks like both Ceph ( https://bugzilla.redhat.com/show_bug.cgi?id=1388368 ) and GlusterFS fails with an error to fetch secrets in Openshift. iic, this was tested in kubernetes upstream and passed. I am not sure whether some special setup is required for openshift. I am adding Jan and Huamin for their thoughts.

Comment 3 Jan Safranek 2016-10-26 11:24:14 UTC
Gluster provisioner seems not to have access to secrets.

* I filled https://github.com/kubernetes/kubernetes/pull/35615 to get nicer message in oc describe pvc:
Failed to provision volume with StorageClass "glusterprovisioner1": failed to get secret default/heketi-secret: User "system:serviceaccount:openshift-infra:pv-binder-controller" cannot get secrets in project "default"]

(this not really *required* in Origin in 3.4)

* I filled https://github.com/openshift/origin/pull/11581 with system:serviceaccount:openshift-infra:pv-binder-controller permission updates

Comment 4 Jan Safranek 2016-10-26 11:25:03 UTC
*** Bug 1388368 has been marked as a duplicate of this bug. ***

Comment 5 Humble Chirammal 2016-10-26 13:00:59 UTC
(In reply to Jan Safranek from comment #3)
> Gluster provisioner seems not to have access to secrets.
> 
> * I filled https://github.com/kubernetes/kubernetes/pull/35615 to get nicer
> message in oc describe pvc:
> Failed to provision volume with StorageClass "glusterprovisioner1": failed
> to get secret default/heketi-secret: User
> "system:serviceaccount:openshift-infra:pv-binder-controller" cannot get
> secrets in project "default"]
> 
> (this not really *required* in Origin in 3.4)
> 
> * I filled https://github.com/openshift/origin/pull/11581 with
> system:serviceaccount:openshift-infra:pv-binder-controller permission updates

Thanks Jan!!

Comment 7 Troy Dawson 2016-11-07 19:33:24 UTC
This has been merged into ose and is in OSE v3.4.0.23 or newer.

Comment 9 Jianwei Hou 2016-11-08 05:57:38 UTC
This is still reproduced on:
openshift v3.4.0.23+24b1a58
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

# oc describe pvc glusterc
Name:		glusterc
Namespace:	jhou
StorageClass:	glusterprovisioner1
Status:		Pending
Volume:		
Labels:		<none>
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  30s		10s		3	{persistentvolume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "glusterprovisioner1": failed to get secret from ["jhou"/"heketi-secret"]

# oc get secret/heketi-secret -n jhou
NAME            TYPE      DATA      AGE
heketi-secret   Opaque    1         14m

Comment 13 Jordan Liggitt 2016-11-09 15:35:52 UTC
The secret type must match the provisioner, not be Opaque

Comment 14 Jan Safranek 2016-11-09 16:28:51 UTC
As Jordan wrote, the secret type must be either "kubernetes.io/glusterfs" or "kubernetes.io/rbd" respectively. Relevant docs change is here: https://github.com/kubernetes/kubernetes.github.io/pull/1594/files

Sorry, I should have let you know earlier.

Comment 15 Jianwei Hou 2016-11-10 05:45:02 UTC
@jliggitt @jsafrane Thank you! With secret type being  "kubernetes.io/glusterfs", the issue is gone! This bug can be verified now. Could you please change its status to ON_QA?

Comment 16 Bradley Childs 2016-11-10 18:44:56 UTC
moving to on_qa per comment.

Comment 17 Jianwei Hou 2016-11-11 00:58:24 UTC
This is fixed according to comment 15.

Comment 19 errata-xmlrpc 2017-01-18 12:44:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066


Note You need to log in before you can comment on or make changes to this bug.