Bug 1767806 - Jenkins service account permission error after upgraded kubernetes plugin to 1.19.2 in jenkins
Summary: Jenkins service account permission error after upgraded kubernetes plugin to ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Jenkins
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.0
Assignee: Akram Ben Aissi
QA Contact: Jitendar Singh
URL:
Whiteboard:
: 1839322 (view as bug list)
Depends On:
Blocks: 1843945 1846957
TreeView+ depends on / blocked
 
Reported: 2019-11-01 13:08 UTC by Abhishek
Modified: 2020-07-13 17:12 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1843941 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:12:05 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift jenkins pull 1044 0 None closed Bug 1767806: bump jenkins to 2.222.1 and update k8s plugins 2020-10-14 07:35:19 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:12:27 UTC

Description Abhishek 2019-11-01 13:08:18 UTC
Description of problem:Jenkins service account permission issue after upgraded kubernetes plugin to 1.19.2  in jenkins

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/<test>/persistentvolumeclaims. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. persistentvolumeclaims "pvc-maven-7xv1q" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>.

Do we have any tested integration document for plugin vs openshift/jenkins?

Openshift version: 4.1.18

Comment 1 Akram Ben Aissi 2019-11-07 09:52:22 UTC
Hi Abishek,

the current release of the Jenkins image uses kubernetes plugin 1.18.2 and we do not plan to migrate immediately unless there a security issue with this version.
Could you elaborate on the root problem? and see why you require 1.19.2 ?

Comment 2 Christian Koep 2019-11-11 08:53:47 UTC
Hi Akram, Abishek,

the reason why one might want to use the Kubernetes plugin in version >1.19.2 is the "dynamic pvc workspace volume" feature. [1]

This feature is currently not usable, because the Jenkins ServiceAccount is configured to have the `edit` ClusterRole if I remember correctly. That being said, the Plugin tries to create PVCs like so: [2]

~~~
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test
  ownerReferences:
  - apiVersion: v1
    kind: Pod
    blockOwnerDeletion: true
    controller: true
    name: foo
    uid: bar
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: glusterfs-storage
~~~

Which fails with an error message along the lines of:

~~~
Error from server (Forbidden): error when creating "pvc.yaml": persistentvolumeclaims "test" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: no RBAC policy matched, <nil>
~~~

So I suppose the fix for this issue would be to add the correct permissions to the `ServiceAccount`, which I was unable to do so far. Here is what I tried to add:

rules:
- apiGroups: ["extensions"]
  resources: ["deployments/finalizers"]
  verbs: ["update"]

and 

- apiGroups: ["extensions"]
  resources: ["pods/finalizers"]
  verbs: ["update"]

However, none of the above apparently does the trick.

Any hints would be appreciated.

Kind regards,
Christian
---

[1] - https://github.com/jenkinsci/kubernetes-plugin/pull/600
[2] - https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes/workspace/DynamicPVCWorkspaceVolume.java#L77-L99

Comment 6 scphantm 2019-12-03 16:30:49 UTC
I just encountered this exact thing with openshift 3.11 and plugin 1.19.3.  exactly the same issue.

Comment 7 Stephen Cuppett 2019-12-13 12:01:52 UTC
Targeting 4.4.0 for investigation on the current master branch.

Comment 8 Vibhav Bobade 2020-01-17 17:54:18 UTC
Hello Abhishek,

Can you try creating the ClusterRole below and binding it to the Jenkins Service Account with the command in a similar fashion after that
-------------------------------------------------
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: finalizer-role
  labels:
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
rules:
  - apiGroups: ["*"]
    resources: ["pods/finalizers","deployments/finalizers","persistentvolumeclaims/finalizers","persistentvolumes/finalizers"]
    verbs: ["*"]
-------------------------------------------------
$ oc create rolebinding jenkins_finalizer --clusterrole=finalizer-role --serviceaccount=jenkins-test:jenkins
-------------------------------------------------

I noticed that you have given the finalizer config only on the pods and not the pvc, whereas the error is happening at the PVC level. 
Let me know if the above works.

Regards,
Vibhav Bobade

Comment 9 Akram Ben Aissi 2020-03-12 20:40:07 UTC
We do not plan to upgrade kubernetes plugin to 1.19.3 yet.
So we are posptoning this one to 4.5.

Comment 12 Jitendar Singh 2020-04-14 11:07:08 UTC
verfied

Comment 13 Akram Ben Aissi 2020-06-15 09:20:14 UTC
*** Bug 1839322 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2020-07-13 17:12:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.