Bug 1477956 - Creating a rolebinding doesn't find the local role due to missing policybinding [NEEDINFO]
Creating a rolebinding doesn't find the local role due to missing policybinding
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth (Show other bugs)
3.6.1
Unspecified Unspecified
unspecified Severity medium
: ---
: 3.7.0
Assigned To: Jordan Liggitt
Chuan Yu
:
: 1477955 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-03 06:12 EDT by Erik Bernoth
Modified: 2018-02-16 04:54 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Reason: Result: RoleBinding objects can now be created without first creating a PolicyBinding object
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-28 17:06:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
mmariyan: needinfo?


Attachments (Terms of Use)
for reproducing bug, workaround commented out (714 bytes, text/plain)
2017-08-03 06:13 EDT, Erik Bernoth
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-28 21:34:54 EST

  None (edit)
Description Erik Bernoth 2017-08-03 06:12:34 EDT
Description of problem:
When creating a SA, (local) Role and a (local) RoleBinding, creation of the RoleBinding fails since it is not able to find the PolicyBinding.

Version-Release number of selected component (if applicable):
-

How reproducible:
Create Resources in an empty project.

Steps to Reproduce:
1. Switch to fresh project "myproject" with user "system:admin"
2. cat rb.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: operator
---
apiVersion: v1
kind: Role
metadata:
  name: xrole
rules:
- apiGroups:
  - ""
  - "batch"
  - "api"
  - "apps"
  resources:
  - "pods"
  - "jobs"
  - "statefulsets"
  verbs:
  - "get"
  - "list"
  - "create"
  - "delete"
  - "watch"
---
apiVersion: v1
kind: RoleBinding
metadata:
  name: xbinding
subjects:
- kind: ServiceAccount
  name: operator
roleRef:
  kind: Role
  name: xrole
  namespace: myproject
userNames:
- system:serviceaccount:myproject:operator
groupNames: null

3. oc create -f rb.yaml

Actual results:
serviceaccount "operator" created
role "xrole" created
Error from server (NotFound): error when creating "rb.yaml": policybinding "myproject:default" not found

Expected results:
PolicyBinding is generated
RoleBinding is created

Additional info:
It works if the PolicyBinding is created before the RoleBinding:
---
apiVersion: v1
kind: PolicyBinding
metadata:
  name: myproject:default
policyRef:
  name: default
  namespace: myproject
roleBindings: null
Comment 1 Erik Bernoth 2017-08-03 06:13 EDT
Created attachment 1308677 [details]
for reproducing bug, workaround commented out
Comment 2 Felix Meulenkamp 2017-08-03 06:19:35 EDT
I also saw this behavior on a "oc cluster up" for Version Origin 1.3.0, OCP 3.4.1, OCP 3.5.5 and Origin 3.6.0.

I would expect that the Policy and PolicyBindings are created automatically.
Comment 3 Eric Paris 2017-08-03 11:51:02 EDT
*** Bug 1477955 has been marked as a duplicate of this bug. ***
Comment 4 Jordan Liggitt 2017-08-03 12:00:58 EDT
This is working as designed, though the requirement to pre-create the policybinding is removed in 3.6

Prior to 3.6, you had to create the policybinding before you could bind to a local role.
Comment 5 Chuan Yu 2017-08-04 03:54:34 EDT
Verified as Jordan mentioned.

# openshift version
openshift v3.5.5.31.6

# openshift version
openshift v3.6.173.0.5
Comment 7 Erik Bernoth 2017-08-30 09:36:38 EDT
I'm trying the same with:

oc version
oc v3.6.173.0.5
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ld3140.wdf.sap.corp:8443
openshift v3.6.173.0.5
kubernetes v1.6.1+5115d708d7



One modification from the previous code examples is the apiVersion, which is not "v1" anymore but "rbac..../v1beta1" (see bottom for current version). This is a desired result for SAP and we are very happy about it, since that is the same apiVersion that kubernetes 1.6 uses. I also checked with "v1" and that doesn't even work for "Role" anymore.

Also I find that there is a :default policybinding now, which is nice. Now running our rbac.yaml succeeds without modification.



!!!!!!!!!!!!!!!!!!!!!
!!
!! HOWEVER!
!!
!!!!!!!!!!!!!!!!!!!!!

After setting up the pod it complains that it still doesn't have the necessary rights!

logs:
E0830 13:30:54.246623       1 reflector.go:201] github.wdf.sap.corp/velocity/k8s-operator/pkg/vora/controller.go:121: Failed to list *v1.Pod: User "system:serviceaccount:d069407:vora-k8s-operator" cannot list pods in project "d069407"
E0830 13:30:54.247118       1 reflector.go:201] github.wdf.sap.corp/velocity/k8s-operator/pkg/vora/controller.go:120: Failed to list *v1beta1.StatefulSet: User "system:serviceaccount:d069407:vora-k8s-operator" cannot list statefulsets.apps in project "d069407"
E0830 13:30:54.248065       1 reflector.go:201] github.wdf.sap.corp/velocity/k8s-operator/pkg/vora/controller.go:119: Failed to list *v1.Job: User "system:serviceaccount:d069407:vora-k8s-operator" cannot list jobs.batch in project "d069407"
E0830 13:30:55.251094       1 reflector.go:201] github.wdf.sap.corp/velocity/k8s-operator/pkg/vora/controller.go:121: Failed to list *v1.Pod: User "system:serviceaccount:d069407:vora-k8s-operator" cannot list pods in project "d069407"
E0830 13:30:55.251345       1 reflector.go:201] github.wdf.sap.corp/velocity/k8s-operator/pkg/vora/controller.go:120: Failed to list *v1beta1.StatefulSet: User "system:serviceaccount:d069407:vora-k8s-operator" cannot list statefulsets.apps in project "d069407


I currently don't have a workaround for this. Please advise.


the file:
$ cat rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vora-k8s-operator
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: vora-k8s-operator
rules:
- apiGroups: ["", "batch", "api", "apps"]
  resources: ["pods", "jobs", "statefulsets"]
  verbs: ["get", "list", "create", "delete", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name:  vora-k8s-operator-binding
subjects:
- kind: ServiceAccount
  name: vora-k8s-operator
roleRef:
  kind: Role
  name: vora-k8s-operator
  apiGroup: rbac.authorization.k8s.io
Comment 8 Jordan Liggitt 2017-08-30 09:47:53 EDT
The rbac.authorization.k8s.io/v1beta1 APIs may not be used to control authorization in 3.6.x

The Openshift v1 Role/RoleBinding must still be used in 3.6.x

In 3.7, the Openshift and RBAC APIs will be unified and either may be used
Comment 9 Erik Bernoth 2017-08-30 09:53:29 EDT
Hi Jordan,

thanks for the quick reply. If that's true how do you explain that I get an error message that the API doesn't know v1.Role but no error mesage when I use the kubernetes api Version?
Comment 10 Jordan Liggitt 2017-08-30 09:59:09 EDT
> If that's true how do you explain that I get an error message that the API doesn't know v1.Role but no error mesage when I use the kubernetes api Version?

It is the OpenShift v1 API, located under /oapi/v1/...

If you are using `kubectl create` (not `oc create`), it will not know about the OpenShift API.

You can use the "groupified" version of the OpenShift API instead, located under /apis/authorization.openshift.io/v1/..., which will work with kubectl:

kind: Role
apiVersion: authorization.openshift.io/v1
...
Comment 11 Erik Bernoth 2017-08-30 10:44:26 EDT
Looks like progress. Now I get the good old error message that it can't find the rolebinding.

I switched the order so the policybinding is created before the rolebinding. But the policybinding still has an error:
Error from server (BadRequest): error when creating "rbac-openshift.yaml": PolicyBinding in version "v1" cannot be handled as a PolicyBinding: parsing time "" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "2006"


If I enter one of the two strings presented instead of an empty string, it also can't parse it. What can I use as meaningful default for lastModified?

https://docs.openshift.com/container-platform/3.5/rest_api/openshift_v1.html#v1-policybinding

$  cat rbac-openshift.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vora-k8s-operator
---
kind: Role
apiVersion: authorization.openshift.io/v1
metadata:
  name: vora-k8s-operator
rules:
- apiGroups: ["", "batch", "api", "apps"]
  resources: ["pods", "jobs", "statefulsets"]
  verbs: ["get", "list", "create", "delete", "watch"]
---
apiVersion: authorization.openshift.io/v1
kind: PolicyBinding
metadata:
  name: d069407:default
policyRef:
  name: default
  namespace: d069407
roleBindings: null
lastModified: "2006-01-02T15:04:05"
---
kind: RoleBinding
apiVersion: authorization.openshift.io/v1
metadata:
  name:  vora-k8s-operator-binding
subjects:
- kind: ServiceAccount
  name: vora-k8s-operator
roleRef:
  kind: Role
  name: vora-k8s-operator
userNames: null
groupNames: null
Comment 12 Jordan Liggitt 2017-08-30 23:12:01 EDT
can you omit lastModified entirely?
Comment 13 Erik Bernoth 2017-08-31 03:23:02 EDT
Hi Jordan,
omitting is not possible. See the link, it's a required argument (although probably unreasonable). Can you check in the source code what they are actually checking for as a format? Then I will just supply a value they can live with and be done with it.
Comment 14 Erik Bernoth 2017-08-31 06:40:23 EDT
I found a good format for the lastModified and applied it. Now I can create the policybinding again, but the rolebinding is still not found:

$ kubectl create -f rbac-openshift.yaml
serviceaccount "vora-k8s-operator" created
role "vora-k8s-operator" created
policybinding "d069407:default" created
Error from server (NotFound): error when creating "rbac-openshift.yaml": role.authorization.openshift.io "vora-k8s-operator" not found

$ cat rbac-openshift.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vora-k8s-operator
---
kind: Role
apiVersion: authorization.openshift.io/v1
metadata:
  name: vora-k8s-operator
rules:
- apiGroups: ["", "batch", "api", "apps"]
  resources: ["pods", "jobs", "statefulsets"]
  verbs: ["get", "list", "create", "delete", "watch"]
---
apiVersion: authorization.openshift.io/v1
kind: PolicyBinding
metadata:
  name: d069407:default
policyRef:
  name: default
  namespace: d069407
roleBindings: null
lastModified: "2015-08-22T12:57:06Z"
---
kind: RoleBinding
apiVersion: authorization.openshift.io/v1
metadata:
  name:  vora-k8s-operator-binding
subjects:
- kind: ServiceAccount
  name: vora-k8s-operator
roleRef:
  kind: Role
  name: vora-k8s-operator
userNames: null
groupNames: null
Comment 15 Erik Bernoth 2017-08-31 10:21:26 EDT
Is someone still working on this?
Comment 16 Jordan Liggitt 2017-08-31 14:42:26 EDT
This works for me:


$ cat rbac-openshift.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vora-k8s-operator
---
kind: Role
apiVersion: authorization.openshift.io/v1
metadata:
  name: vora-k8s-operator
rules:
- apiGroups: ["", "batch", "api", "apps"]
  resources: ["pods", "jobs", "statefulsets"]
  verbs: ["get", "list", "create", "delete", "watch"]
---
kind: RoleBinding
apiVersion: authorization.openshift.io/v1
metadata:
  name:  vora-k8s-operator-binding
subjects:
- kind: ServiceAccount
  name: vora-k8s-operator
roleRef:
  kind: Role
  name: vora-k8s-operator
  namespace: d069407
userNames: null
groupNames: null



$ oc create -f rbac-openshift.yaml

serviceaccount "vora-k8s-operator" created
role "vora-k8s-operator" created
rolebinding "vora-k8s-operator-binding" created
Comment 17 Sherry Yu 2017-08-31 18:27:29 EDT
(In reply to Jordan Liggitt from comment #16)
> This works for me:
> 
> 
> $ cat rbac-openshift.yaml
> 
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: vora-k8s-operator
> ---
> kind: Role
> apiVersion: authorization.openshift.io/v1
> metadata:
>   name: vora-k8s-operator
> rules:
> - apiGroups: ["", "batch", "api", "apps"]
>   resources: ["pods", "jobs", "statefulsets"]
>   verbs: ["get", "list", "create", "delete", "watch"]
> ---
> kind: RoleBinding
> apiVersion: authorization.openshift.io/v1
> metadata:
>   name:  vora-k8s-operator-binding
> subjects:
> - kind: ServiceAccount
>   name: vora-k8s-operator
> roleRef:
>   kind: Role
>   name: vora-k8s-operator
>   namespace: d069407
> userNames: null
> groupNames: null
> 
> 
> 
> $ oc create -f rbac-openshift.yaml
> 
> serviceaccount "vora-k8s-operator" created
> role "vora-k8s-operator" created
> rolebinding "vora-k8s-operator-binding" created

This has been tried and worked, thanks for that! But when the SAP team continues there may be other questions coming up, and a prompt response is highly needed. The deadline to have OpenShift fully tested is end of Friday Sept 1. On Friday in German time zone when SAP continues the work, someone standing by in the same time zone would be greatly appreciated.
Comment 18 Erik Bernoth 2017-09-01 14:21:35 EDT
Hi, in case someone provided someone to look into this issue but got no response the whole day: Sorry! Due to the time pressure and because the priorities changing this topic moved from highest to almost lowest priorities in the morning.

The final goal of a successful validation could be achieved though. And because yesterday night nobody could know that this is not important, the help is still appreciated a lot!

In the evening (German POV) we have been continuing with this topic together with Tommy Hughes, Sherry Yu, and Markus Koch in bluejeans sessions. We aim to present the results here before closing the issue after we've got it working, or earlier if a new question arises.


TLDR: time pressure is reduced for now, and we continue working on it, keeping this ticket updated when we have something of interest. Thanks!
Comment 21 errata-xmlrpc 2017-11-28 17:06:30 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.