Bug 1766856

Summary:	Local volume feature broken with RBAC permission issue
Product:	OpenShift Container Platform	Reporter:	Liang Xia <lxia>
Component:	Storage	Assignee:	Hemant Kumar <hekumar>
Status:	CLOSED ERRATA	QA Contact:	Liang Xia <lxia>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.3.0	CC:	aos-bugs, aos-storage-staff, hekumar, lbednar
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	4.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1774090 (view as bug list)		Environment:
Last Closed:	2020-01-23 11:09:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1768701
Bug Blocks:	1774090

Description Liang Xia 2019-10-30 06:21:26 UTC

Description of problem:
Deploy local volume on OCP 4.3 cluster by following https://github.com/openshift/local-storage-operator/blob/master/docs/deploy-with-olm.md, 
the local-storage-operator is throwing RBAC permission issue.

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-10-28-222147

How reproducible:
Always

Steps to Reproduce:
1. Following https://github.com/openshift/local-storage-operator/blob/master/docs/deploy-with-olm.md
2. After CR is created, check all in namespace local-storage
3.

Actual results:
There are no storage class and PV created, even no provisioner and disk-maker pod exist/running.
$ oc get all
NAME                                          READY   STATUS    RESTARTS   AGE
pod/local-storage-operator-7759795457-9mh7r   1/1     Running   0          15h

NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/local-storage-operator   ClusterIP   172.30.166.181   <none>        60000/TCP   15h

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/local-storage-operator   1/1     1            1           15h

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/local-storage-operator-7759795457   1         1         1       15h


Expected results:
The local volume is functioning.


Additional info:
$ oc logs pod/local-storage-operator-7759795457-9mh7r
I1029 11:19:53.682709       1 main.go:18] Go Version: go1.10.8
I1029 11:19:53.683050       1 main.go:19] Go OS/Arch: linux/amd64
I1029 11:19:53.683074       1 main.go:20] operator-sdk Version: 0.0.7
time="2019-10-29T11:19:53Z" level=info msg="Metrics service local-storage-operator created"
I1029 11:19:53.861347       1 main.go:36] Watching local.storage.openshift.io/v1, LocalVolume
I1029 11:19:53.861356       1 main.go:41] Watching local.storage.openshift.io/v1, LocalVolume, local-storage, 180000000000
E1029 22:07:53.986662       1 memcache.go:147] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E1029 22:08:43.372004       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=95079, ErrCode=NO_ERROR, debug=""
W1029 22:08:43.797110       1 reflector.go:341] github.com/openshift/local-storage-operator/vendor/github.com/operator-framework/operator-sdk/pkg/sdk/informer.go:91: watch of *unstructured.Unstructured ended with: unexpected object: &{map[apiVersion:v1 metadata:map[] status:Failure message:too old resource version: 143222 (345197) reason:Gone code:410 kind:Status]}
E1029 22:08:53.957243       1 memcache.go:147] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E1029 22:12:31.513069       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=589, ErrCode=NO_ERROR, debug=""
W1029 22:12:31.814700       1 reflector.go:341] github.com/openshift/local-storage-operator/vendor/github.com/operator-framework/operator-sdk/pkg/sdk/informer.go:91: watch of *unstructured.Unstructured ended with: unexpected object: &{map[metadata:map[] status:Failure message:too old resource version: 345197 (345967) reason:Gone code:410 kind:Status apiVersion:v1]}
E1030 02:56:16.995443       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=41547, ErrCode=NO_ERROR, debug=""
W1030 02:56:17.111693       1 reflector.go:341] github.com/openshift/local-storage-operator/vendor/github.com/operator-framework/operator-sdk/pkg/sdk/informer.go:91: watch of *unstructured.Unstructured ended with: unexpected object: &{map[status:Failure message:too old resource version: 345967 (346702) reason:Gone code:410 kind:Status apiVersion:v1 metadata:map[]]}
W1030 02:58:12.819994       1 reflector.go:341] github.com/openshift/local-storage-operator/vendor/github.com/operator-framework/operator-sdk/pkg/sdk/informer.go:91: watch of *unstructured.Unstructured ended with: unexpected object: &{map[code:410 kind:Status apiVersion:v1 metadata:map[] status:Failure message:too old resource version: 346702 (434280) reason:Gone]}
I1030 03:07:37.704347       1 api_updater.go:75] Updating localvolume local-storage/local-disks
E1030 03:07:37.871331       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:37Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:38.005282       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:38Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:38.187667       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:38Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:38.978857       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:38Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:39.785121       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:39Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:40.581366       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:40Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:41.380715       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:41Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:42.179644       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:42Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:42.982532       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-10-30T03:07:42Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"
E1030 03:07:43.784049       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}

Comment 6 Liang Xia 2019-11-11 07:06:57 UTC

Tested with 4.3.0-0.nightly-2019-11-10-185106 using quay.io/hekumar as app registry, no RBAC issue now.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2019-11-10-185106   True        False         5h4m    Cluster version is 4.3.0-0.nightly-2019-11-10-185106



$ oc get all -n local-storage
NAME                                         READY   STATUS    RESTARTS   AGE
pod/local-disks-local-diskmaker-b442v        1/1     Running   0          3m58s
pod/local-disks-local-diskmaker-zxmz8        1/1     Running   0          3m58s
pod/local-disks-local-provisioner-df62f      1/1     Running   0          3m58s
pod/local-disks-local-provisioner-kgb6d      1/1     Running   0          3m58s
pod/local-storage-operator-b6b7c94b6-jwjzh   1/1     Running   0          37m

NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
service/local-storage-operator   ClusterIP   172.30.117.98   <none>        60000/TCP   37m

NAME                                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/local-disks-local-diskmaker     2         2         2       2            2           <none>          3m59s
daemonset.apps/local-disks-local-provisioner   2         2         2       2            2           <none>          3m59s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/local-storage-operator   1/1     1            1           37m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/local-storage-operator-b6b7c94b6   1         1         1       37m



$ oc get sc,pv
NAME                                             PROVISIONER                    AGE
storageclass.storage.k8s.io/local-sc             kubernetes.io/no-provisioner   4m44s
storageclass.storage.k8s.io/standard (default)   kubernetes.io/gce-pd           5h16m

NAME                                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
persistentvolume/local-pv-cc199c3e   10Gi       RWO            Delete           Available           local-sc                4m20s



$ pwd ; git diff catalog-create-subscribe.yaml
/home/lxia/github.com/local-storage-operator/examples/olm
diff --git a/examples/olm/catalog-create-subscribe.yaml b/examples/olm/catalog-create-subscribe.yaml
index 68c088c..1ca7fb2 100644
--- a/examples/olm/catalog-create-subscribe.yaml
+++ b/examples/olm/catalog-create-subscribe.yaml
@@ -19,7 +19,7 @@ metadata:
   namespace: openshift-marketplace
 spec:
   type: appregistry
-  endpoint: https://quay.io/cnr
+  endpoint: https://quay.io/hekumar
   registryNamespace: hekumar
   displayName: "Red Hat Storage operators"
   publisher: "Red Hat Storage"

Comment 8 errata-xmlrpc 2020-01-23 11:09:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062