Bug 1774090

Summary: Local volume feature broken with RBAC permission issue
Product: OpenShift Container Platform Reporter: Fabio Bertinatto <fbertina>
Component: StorageAssignee: Hemant Kumar <hekumar>
Status: CLOSED ERRATA QA Contact: Liang Xia <lxia>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.2.zCC: aos-bugs, aos-storage-staff, hekumar, lbednar, lxia
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.2.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1766856 Environment:
Last Closed: 2019-12-11 22:36:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1766856    
Bug Blocks:    

Comment 2 Liang Xia 2019-11-29 03:13:15 UTC
As stated in https://github.com/openshift/local-storage-operator/pull/56,
This will allow 4.2 manifests to work on OCP-4.3

So tested with,
4.3.0-0.nightly-2019-11-27-011055 (cluster version)
4.2.8-201911190952 (local-storage-operator version)

And the issue is reproduced.

$ oc logs pod/local-storage-operator-77df666d5-sjc46 | head -10
I1129 02:57:49.798886       1 main.go:18] Go Version: go1.11.13
I1129 02:57:49.799540       1 main.go:19] Go OS/Arch: linux/amd64
I1129 02:57:49.799558       1 main.go:20] operator-sdk Version: 0.0.7
time="2019-11-29T02:57:49Z" level=info msg="Metrics service local-storage-operator created"
I1129 02:57:49.980410       1 main.go:36] Watching local.storage.openshift.io/v1, LocalVolume
I1129 02:57:49.980419       1 main.go:41] Watching local.storage.openshift.io/v1, LocalVolume, local-storage, 180000000000
I1129 03:07:48.916620       1 api_updater.go:75] Updating localvolume local-storage/local-disks
E1129 03:07:49.058092       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-11-29T03:07:49Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"

We will try again on a latest 4.3 nightly build.

Comment 3 Liang Xia 2019-11-29 05:50:09 UTC
Unfortunately, still reproduced.

4.3.0-0.nightly-2019-11-29-013902 (cluster version)
4.2.8-201911190952 (local-storage-operator version)

Moving back to investigate.

$ oc logs pod/local-storage-operator-76df8f8c56-jsjdr | head -10
I1129 05:27:27.825563       1 main.go:18] Go Version: go1.11.13
I1129 05:27:27.825786       1 main.go:19] Go OS/Arch: linux/amd64
I1129 05:27:27.825796       1 main.go:20] operator-sdk Version: 0.0.7
time="2019-11-29T05:27:28Z" level=info msg="Metrics service local-storage-operator created"
I1129 05:27:28.071743       1 main.go:36] Watching local.storage.openshift.io/v1, LocalVolume
I1129 05:27:28.071755       1 main.go:41] Watching local.storage.openshift.io/v1, LocalVolume, local-storage, 180000000000
I1129 05:37:42.176661       1 api_updater.go:75] Updating localvolume local-storage/local-disks
E1129 05:37:42.371737       1 controller.go:135] error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io "local-storage-provisioner-pv-binding" is forbidden: user "system:serviceaccount:local-storage:local-storage-operator" (groups=["system:serviceaccounts" "system:serviceaccounts:local-storage" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["events.k8s.io"], Resources:["events"], Verbs:["create" "patch" "update"]}
time="2019-11-29T05:37:42Z" level=error msg="error syncing key (local-storage/local-disks): error applying pv cluster role binding local-storage-provisioner-pv-binding: clusterrolebindings.rbac.authorization.k8s.io \"local-storage-provisioner-pv-binding\" is forbidden: user \"system:serviceaccount:local-storage:local-storage-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:local-storage\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"events.k8s.io\"], Resources:[\"events\"], Verbs:[\"create\" \"patch\" \"update\"]}"

Comment 8 errata-xmlrpc 2019-12-11 22:36:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4093