Bug 1561247

Summary: [starter-us-east-1b] Unable to apply auto-approver resource definitions: no kind \"ClusterRoleBinding\" is registered for version \"rbac.authorization.k8s.io/v1\"
Product: OpenShift Container Platform Reporter: Justin Pierce <jupierce>
Component: InstallerAssignee: Russell Teague <rteague>
Status: CLOSED ERRATA QA Contact: Gaoyun Pei <gpei>
Severity: high Docs Contact:
Priority: high    
Version: 3.9.0CC: aos-bugs, jokerman, jupierce, mmccomas
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The master admin.kubeconfig was added to the oc command to allow the operation to have the proper authorization and access to the necessary resources.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-17 06:43:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1566238    
Bug Blocks:    
Attachments:
Description Flags
oc apply error none

Description Justin Pierce 2018-03-28 00:16:00 UTC
Created attachment 1413938 [details]
oc apply error

Description of problem:
During an upgrade from 3.7 to 3.9, the following error was reported. Retries encounter the same issue. See attached listing. 


Version-Release number of selected component (if applicable):
v3.9.14

Comment 1 Scott Dodson 2018-03-28 21:29:41 UTC
Which playbook was executed to invoke this role? openshift-ansible doesn't include this in any playbooks other than gce provisioning.

Comment 3 Scott Dodson 2018-03-29 01:08:36 UTC
Correction, this is called via the control plane upgrade.

https://github.com/openshift/openshift-ansible/blob/master/playbooks/openshift-master/private/upgrade.yml#L83-L85

The problem likely happened because the oc command doesn't utilize the admin kubeconfig and on that particular host root's kubeconfig was in another project at the time. We need to sweep the codebase again for this problem.

Comment 4 Scott Dodson 2018-03-29 13:28:04 UTC
We're also likely running this code on each master where as I believe it only needs to be run once as it's cluster scoped.

Comment 5 Russell Teague 2018-03-29 19:24:02 UTC
master: https://github.com/openshift/openshift-ansible/pull/7713

Comment 6 Russell Teague 2018-04-03 19:41:30 UTC
release-3.9: https://github.com/openshift/openshift-ansible/pull/7758

Comment 7 Russell Teague 2018-04-11 12:25:12 UTC
Commit is in build openshift-ansible-3.9.20-1.git.0.f99fb43.el7

Comment 10 Gaoyun Pei 2018-04-23 10:03:01 UTC
Couldn't reproduce this issue with openshift-ansible-3.9.14-1.git.0.ca2cfc3.el7.noarch.rpm when upgrading 3.7 to 3.9.

openshift-ansible wouldn't run the step unless we have openshift_master_bootstrap_enabled=true set in ansible inventory file, tried an upgrade from 3.7 to 3.9 using openshift-ansible-3.9.24-1.git.0.d0289ea.el7.noarch  with openshift_master_bootstrap_enabled=true set, no such error happened.

[root@gpei-preserved ~]# grep -A 2 "Create auto-approver on cluster" logs/0423_upgrade
TASK [openshift_bootstrap_autoapprover : Create auto-approver on cluster] ***************************************************************************************************
changed: [qe-gpei-37master-1.0423-t8x.qe.rhcloud.com] => {"changed": true, "cmd": ["oc", "apply", "-f", "/tmp/openshift-approver/", "--config=/etc/origin/master/admin.kubeconfig"], "delta": "0:00:00.323191", "end": "2018-04-23 04:27:42.663713", "failed": false, "rc": 0, "start": "2018-04-23 04:27:42.340522", "stderr": "", "stderr_lines": [], "stdout": "clusterrolebinding \"bootstrap-autoapprover\" created\nclusterrole \"system:node-bootstrap-autoapprover\" created\nserviceaccount \"bootstrap-autoapprover\" created\nstatefulset \"bootstrap-autoapprover\" created", "stdout_lines": ["clusterrolebinding \"bootstrap-autoapprover\" created", "clusterrole \"system:node-bootstrap-autoapprover\" created", "serviceaccount \"bootstrap-autoapprover\" created", "statefulset \"bootstrap-autoapprover\" created"]}

--
TASK [openshift_bootstrap_autoapprover : Create auto-approver on cluster] ***************************************************************************************************
changed: [qe-gpei-37master-1.0423-t8x.qe.rhcloud.com] => {"changed": true, "cmd": ["oc", "apply", "-f", "/tmp/openshift-approver/", "--config=/etc/origin/master/admin.kubeconfig"], "delta": "0:00:00.386440", "end": "2018-04-23 04:38:44.751348", "failed": false, "rc": 0, "start": "2018-04-23 04:38:44.364908", "stderr": "", "stderr_lines": [], "stdout": "clusterrolebinding \"bootstrap-autoapprover\" configured\nclusterrole \"system:node-bootstrap-autoapprover\" configured\nserviceaccount \"bootstrap-autoapprover\" unchanged\nstatefulset \"bootstrap-autoapprover\" configured", "stdout_lines": ["clusterrolebinding \"bootstrap-autoapprover\" configured", "clusterrole \"system:node-bootstrap-autoapprover\" configured", "serviceaccount \"bootstrap-autoapprover\" unchanged", "statefulset \"bootstrap-autoapprover\" configured"]}

Move this bug to verified.

Comment 13 errata-xmlrpc 2018-05-17 06:43:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566