Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1496981

Summary: Rolebinding incorrect for management-admin CloudForms SA
Product: OpenShift Container Platform Reporter: nate stephany <nstephan>
Component: InstallerAssignee: Tim Bielawa <tbielawa>
Status: CLOSED DUPLICATE QA Contact: Johnny Liu <jialiu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: aos-bugs, jokerman, mmccomas, nstephan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-16 15:51:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description nate stephany 2017-09-28 21:04:24 UTC
Description of problem:
After deploying OCP 3.6, the rolebinding for the management-infra project is incorrect and doesn't allow CloudForms to connect to OpenShift


Version-Release number of selected component (if applicable): 3.6


How reproducible: 100%


Steps to Reproduce:
1. Install OpenShift
2. oc get rolebinding -n management-infra
3.

Actual results: role is given to management-admin user

[root@rhel7-workstation openshift]# oc get rolebinding
NAME                   ROLE                    USERS              GROUPS                                    SERVICE ACCOUNTS   SUBJECTS
admin                  /admin                  management-admin
system:deployer        /system:deployer                                                                     deployer
system:image-builder   /system:image-builder                                                                builder
system:image-puller    /system:image-puller                       system:serviceaccounts:management-infra


Expected results: role should be given to management-admin service account. Easily fixed with the following command:

[root@rhel7-workstation openshift]# oc adm policy add-role-to-user admin -z management-admin -n management-infra
role "admin" added: "management-admin"
[root@rhel7-workstation openshift]# oc get rolebinding -n management-infra
NAME                   ROLE                    USERS              GROUPS                                    SERVICE ACCOUNTS   SUBJECTS
admin                  /admin                  management-admin                                             management-admin
system:deployer        /system:deployer                                                                     deployer
system:image-builder   /system:image-builder                                                                builder
system:image-puller    /system:image-puller                       system:serviceaccounts:management-infra


Additional info: Looks like the installer sets this up, so might just need to update /usr/share/ansible/openshift-ansible/roles/openshift_manageiq/vars/main.yml to change the first user var to the service account

Actual error from cloudforms evm.log:
[----] E, [2017-09-28T13:43:20.761132 #4315:f7d140] ERROR -- : Q-task_id([9e7efebe-a48d-11e7-bebd-0217bf1a0264]) MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::Scanning::Job#start) pod creation for [management-infra/manageiq-img-scan-9e7ef] failed: [HTTP status code 403, User "system:serviceaccount:managem
ent-infra:management-admin" cannot create pods in project "management-infra"]

Comment 1 Tim Bielawa 2017-10-13 16:08:05 UTC
How to reproduce this? I have the latest manageIQ running on OCP 3.7 and I do not see this error in my evm logs.

Is there some action you're taking in the web UI that triggers this error?

Comment 2 nate stephany 2017-10-13 17:27:33 UTC
Kick of a smartstate analysis of a container image. I have not tried 3.7...this was from 3.6. Is the rolebinding output in your 3.7 different and already includes the management-admin service account instead of a regular user? Since we just add the token for the management-admin service account when adding openshift as a provider to CF/MiQ, the regular user doesn't really work.

Comment 3 Tim Bielawa 2017-10-13 19:19:06 UTC
Using the bearer token from the system account created by the openshift_manageiq role to integrate OCP as a container provider in manageiq, I am not running into issues. I am receiving messages indicating success 

{"@timestamp":"2017-10-13T18:44:14.572743 ","hostname":"manageiq-0","pid":182,"tid":"2acf1da6b140","level":"info","message":"Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::Scanning::Job#start) Creating pod [management-infra/manageiq-img-scan-a05f8] to analyze docker image [docker.io/manageiq/postgresql@sha256:4a0a1fd27bcc1d8fa3be8298e488878b61f35786dbf257cb24c1766c16fefd37] [{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"name\":\"manageiq-img-scan-a05f8\",\"namespace\":\"management-infra\",\"labels\":{\"name\":\"manageiq-img-scan-a05f8\",\"manageiq.org\":\"true\"},\"annotations\":{\"manageiq.org/hostname\":\"manageiq-0\",\"manageiq.org/guid\":\"416aa9d7-f36d-409b-b7d0-6b3d59a9f05c\",\"manageiq.org/image\":\"docker.io/manageiq/postgresql@sha256:4a0a1fd27bcc1d8fa3be8298e488878b61f35786dbf257cb24c1766c16fefd37\",\"manageiq.org/jobid\":\"a05f8beb-764d-4ba4-9718-ec5c61ba71be\"}},\"spec\":{\"restartPolicy\":\"Never\",\"containers\":[{\"name\":\"image-inspector\",\"image\":\"registry.access.redhat.com/openshift3/image-inspector:2.1\",\"imagePullPolicy\":\"Always\",\"command\":[\"/usr/bin/image-inspector\",\"--chroot\",\"--image=docker.io/manageiq/postgresql@sha256:4a0a1fd27bcc1d8fa3be8298e488878b61f35786dbf257cb24c1766c16fefd37\",\"--scan-type=openscap\",\"--serve=0.0.0.0:8080\",\"--dockercfg=/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-lt72f/.dockercfg\"],\"ports\":[{\"containerPort\":8080}],\"securityContext\":{\"privileged\":true},\"volumeMounts\":[{\"mountPath\":\"/var/run/docker.sock\",\"name\":\"docker-socket\"},{\"name\":\"inspector-admin-secret\",\"mountPath\":\"/var/run/secrets/kubernetes.io/inspector-admin-secret-inspector-admin-dockercfg-lt72f\",\"readOnly\":true}],\"env\":[],\"readinessProbe\":{\"initialDelaySeconds\":15,\"periodSeconds\":5,\"httpGet\":{\"path\":\"/healthz\",\"port\":8080}}}],\"volumes\":[{\"name\":\"docker-socket\",\"hostPath\":{\"path\":\"/var/run/docker.sock\"}},{\"name\":\"inspector-admin-secret\",\"secret\":{\"secretName\":\"inspector-admin-dockercfg-lt72f\"}}]}}]"}


Did you add your OCP cluster to your MIQ installation using the token from the management-admin account in the management-infra namespace?

> $ oc serviceaccounts get-token -n management-infra management-admin

...


> Is the rolebinding output in your 3.7 different and already includes the management-admin service account instead of a regular user?

It looks like in https://github.com/openshift/openshift-ansible/commit/c3d6c7a4233a9c0a8bd2e361927e86f9ee3431ad (Tue Sep 12 13:50:27 2017) we added role: admin to management-admin

> + - resource_kind: role
> +   resource_name: admin
> +   user: system:serviceaccount:management-infra:management-admin


Looks like you are right here, we just need to backport that change to the 3.6 release series and you will be good to go.

Comment 4 Tim Bielawa 2017-10-13 19:28:03 UTC
Oh, nevermind that last comment about backporting. This is already present in the 3.6 file:

$ grep -A2 'resource_kind: role' roles/openshift_manageiq/vars/main.yml- resource_kind: role
  resource_name: admin
  user: management-admin
- resource_kind: role
  resource_name: admin
  user: system:serviceaccount:management-infra:management-admin


On my 3.7 cluster I see the following role bindings automatically:


[root@m01 ~]# oc get rolebinding -n management-infra

NAME                   ROLE                    USERS              GROUPS                                    SERVICE ACCOUNTS   SUBJECTS
admin                  /admin                  management-admin                                             management-admin   
system:deployer        /system:deployer                                                                     deployer           
system:image-builder   /system:image-builder                                                                builder            
system:image-puller    /system:image-puller                       system:serviceaccounts:management-infra                      


These should be present as well in an OCP 3.6 installation because the release-3.6 branch is including that same file (there is 0 diff between that file on the release-3.6 branch and the master branch).

Do you know what exact version of the installer you used which introduced this error? I see the BZ was opened September 28, but that change to the role bindings was already present as of September 11th.

Comment 5 nate stephany 2017-10-16 14:57:13 UTC
Looks like this is my current version and the latest available when I do a yum update on the system:

openshift-ansible-roles-3.6.173.0.21-2.git.0.44a4038.el7.noarch

This system (my "installer" host) was originally 3.5 and I upgraded to 3.6 on Aug 25:

[root@rhel7-workstation inventories]# cat /var/log/yum.log | grep -i openshift-ansible-roles
May 30 09:18:11 Installed: openshift-ansible-roles-3.5.71-1.git.0.128c2db.el7.noarch
Jun 23 08:13:24 Updated: openshift-ansible-roles-3.5.78-1.git.0.f7be576.el7.noarch
Jun 23 11:38:23 Erased: openshift-ansible-roles-3.5.78-1.git.0.f7be576.el7.noarch
Jun 23 11:38:52 Installed: openshift-ansible-roles-3.5.78-1.git.0.f7be576.el7.noarch
Jul 06 14:55:48 Updated: openshift-ansible-roles-3.5.91-1.git.0.28b3ddb.el7.noarch
Aug 25 16:53:30 Updated: openshift-ansible-roles-3.6.173.0.5-3.git.0.522a92a.el7.noarch
Sep 20 20:55:28 Updated: openshift-ansible-roles-3.6.173.0.21-2.git.0.44a4038.el7.noarch



This is the current content of the same var file for the role:

[root@rhel7-workstation inventories]# cat /usr/share/ansible/openshift-ansible/roles/openshift_manageiq/vars/main.yml | head -n 8
---
manage_iq_tasks:
- resource_kind: role
  resource_name: admin
  user: management-admin

Comment 6 nate stephany 2017-10-16 15:22:24 UTC
Also, as far as I can remember, this was a problem for me from the beginning with 3.6.

Comment 7 Tim Bielawa 2017-10-16 15:51:09 UTC

*** This bug has been marked as a duplicate of bug 1487648 ***

Comment 8 Tim Bielawa 2017-10-16 16:04:47 UTC
(In reply to nate stephany from comment #6)
> Also, as far as I can remember, this was a problem for me from the beginning
> with 3.6.

The fix for this is included in the referenced duplicate bug. Once the errata ships a new version of openshift-ansible-roles will be available through the repositories. You will be able to upgrade the package and the missing SCCs will be available.