Bug 1493213

Summary: Builds fail with "authentication required" after upgrade
Product: OpenShift Container Platform Reporter: Steven Walter <stwalter>
Component: apiserver-authAssignee: Simo Sorce <ssorce>
Status: CLOSED ERRATA QA Contact: Chuan Yu <chuyu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: aos-bugs, bingli, cheimes, jialiu, jkaur, jokerman, mkhan, mmariyan, mmccomas, plarsen, ssorce, xxia
Target Milestone: ---   
Target Release: 3.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
During upgrades reconciliation happens only for cluster roles automatically. But this role needs to be adjusted in 3.6 due to enablement of API groups in this realease. The ansible upgrade code has been changed to take care of this role upgrade.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-07 07:11:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Steven Walter 2017-09-19 15:27:45 UTC
Description of problem:
After upgrade from 3.5 to 3.6, builds fail with unauthorized: authentication required, when trying to pull from registry. When manually pulling with a user account (admin) to the node that should run the build, re-starting the build works. Service accounts do not appear to be able to pull the images from the registry.

Version-Release number of selected component (if applicable):
3.6

How reproducible:
Unconfirmed


Actual results:
pulling image error : unauthorized: authentication required
error: build error: unable to get 172.30.109.30:5000/openshift/a24-tc8-jdk8@sha256:269d6a4e72

[root]# docker login -u serviceaccount -p $(oc sa get-token deployer) 172.30.109.30:5000
Login Succeeded
[root]# docker pull  172.30.109.30:5000/openshift/a24-tc8-jdk8@sha256:acc6fc10a8ea90b49e56f3259162cf4a8e57cfa98ead03a81ace9b90a5e4b947
Trying to pull repository 172.30.109.30:5000/openshift/a24-tc8-jdk8 ...
unauthorized: authentication required
[root]# docker login -u serviceaccount -p $(oc sa get-token builder) 172.30.109.30:5000
Login Succeeded
[root]# docker pull  172.30.109.30:5000/openshift/a24-tc8-jdk8@sha256:acc6fc10a8ea90b49e56f3259162cf4a8e57cfa98ead03a81ace9b90a5e4b947
Trying to pull repository 172.30.109.30:5000/openshift/a24-tc8-jdk8 ...
unauthorized: authentication required


Expected results:
[root@master-1 cloud-user]# docker login -u serviceaccount -p $( oc sa get-token default ) 172.30.155.150:5000
Login Succeeded
[root@master-1 cloud-user]# docker pull 172.30.155.150:5000/openshift/python 
Using default tag: latest
Trying to pull repository 172.30.155.150:5000/openshift/python ... 
sha256:3c9b3aa7da699a02a9a3285b2c4f816fd4405f580f0120e2fbddb976c9299d22: Pulling from 172.30.155.150:5000/openshift/python
d55ab3b04d8b: Downloading [==============================================>    ] 67.04 MB/72.16 MB
b94f985aad49: Download complete 
6d71013e372d: Downloading [==================================================>] 68.66 MB/68.66 MB
3398045bac98: Download complete

Comment 6 Steven Walter 2017-09-20 15:32:09 UTC
Customer was able to get builds working using workaround from github issue:

oc adm policy add-role-to-group system:image-puller system:serviceaccounts -n openshift

Comment 9 Simo Sorce 2017-10-03 12:43:00 UTC
Upstream issue here :https://github.com/openshift/origin/pull/16465
Initial PR to deal with the issue here: https://github.com/openshift/openshift-ansible/pull/5617

Comment 10 Mo 2017-10-10 20:20:08 UTC
*** Bug 1500225 has been marked as a duplicate of this bug. ***

Comment 11 Simo Sorce 2017-10-11 13:50:10 UTC
A fix has been commit to openshift ansible for release 3.6:
https://github.com/openshift/openshift-ansible/pull/5649

Comment 13 Chuan Yu 2017-10-25 05:57:05 UTC
Verified.

# openshift version
openshift v3.6.173.0.59
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

openshift-ansible-3.6.173.0.59-1

Comment 18 errata-xmlrpc 2017-12-07 07:11:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389

Comment 19 Simo Sorce 2018-01-26 14:25:40 UTC
*** Bug 1538261 has been marked as a duplicate of this bug. ***

Comment 20 Ryan Howe 2018-02-28 16:50:32 UTC
*** Bug 1550162 has been marked as a duplicate of this bug. ***