Bug 1463630

Summary: [trello pyDUR5Bx] oc cluster up fails with --service-catalog flag on Windows
Product: OpenShift Container Platform Reporter: Dongbo Yan <dyan>
Component: SecurityAssignee: Mo <mkhan>
Status: CLOSED ERRATA QA Contact: Xiaoli Tian <xtian>
Severity: low Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, deads, dyan, jialiu, jokerman, mmccomas, smunilla
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:28:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dongbo Yan 2017-06-21 11:30:17 UTC
Description of problem:
oc cluster up fails with --service-catalog flag on Windows

Version-Release number of selected component (if applicable):
oc v3.6.116
kubernetes v1.6.1+5115d708d7

docker@openshift3:~$ docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5

How reproducible:


Steps to Reproduce:
1.Use docker-machine tool to create a docker-machine VM on Windows OS
2.Execute oc cluster up with --service-catalog flag
3.

Actual results:
C:\Windows\system32>oc cluster up --image=brew-pulp.../openshift3/ose --version=v3.6 --service-catalog --docker-machine=openshift3.5
Starting OpenShift using brew-pulp.../openshift3/ose:v3.6 ...
Pulling image brew-pulp.../openshift3/ose:v3.6
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for brew-pulp.../openshift3/ose:v3.6 image ...
   Pulling image brew-pulp.../openshift3/ose:v3.6
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using Docker shared volumes for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using docker-machine IP 192.168.99.100 as the host IP
   Using 192.168.99.100 as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Adding default OAuthClient redirect URIs ... OK
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Installing service catalog ... FAIL
   Error: cannot instantiate service catalog template
   Caused By:
     Error: cannot create objects from template openshift/service-catalog
     Caused By:
       Error: role.authorization.openshift.io "extension-apiserver-authentication-reader" not found

Expected results:
Execute command successfully without error

Additional info:

Comment 1 Ben Parees 2017-06-22 04:13:49 UTC
This seems like a possible bug in the windows client.  On the same machine were you are hitting this error, can you start up oc cluster up without the --service-catalog flag, and then run:

$ oc login -u system:admin
$ oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/service-catalog/service-catalog.yaml --loglevel=5


?

Comment 2 Dongbo Yan 2017-06-22 11:16:54 UTC
Hi, ben, please check

--> Creating resources ...
    clusterrole "servicecatalog-serviceclass-viewer" created
    clusterrolebinding "servicecatalog-serviceclass-viewer-binding" created
    serviceaccount "service-catalog-controller" created
    serviceaccount "service-catalog-apiserver" created
    clusterrole "namespace-viewer" created
    clusterrolebinding "service-catalog-namespace-viewer-binding" created
    clusterrolebinding "service-catalog-controller-namespace-viewer-binding" created
    clusterrole "service-catalog-controller" created
    clusterrolebinding "service-catalog-controller-binding" created
    role "endpoint-accessor" created
    error: role.authorization.openshift.io "endpoint-accessor" not found
    role "extension-apiserver-authentication-reader" created
    rolebinding "extension-apiserver-authentication-reader-binding" created
    clusterrolebinding "system:auth-delegator-binding" created
    deployment "apiserver" created
    service "apiserver" created
    deployment "controller-manager" created
    service "controller-manager" created
--> Failed

Comment 3 Ben Parees 2017-06-22 20:44:52 UTC
ultimate this issue is related to code that's in the "brew-pulp.../openshift3/ose:v3.6" image you were using during the test.  Can you check if the same image was being used for both the windows and other tests?  I'm wondering if your windows environment had an old version of the image around.

Comment 4 Dongbo Yan 2017-06-23 02:05:58 UTC
I check the ose image on Macbook and Windows, they are the same version.

brew-pulp.../openshift3/ose                     v3.6                bbac4caf7fb5        43 hours ago        994 MB

Comment 5 Ben Parees 2017-06-23 04:02:20 UTC
this looks like it may be a race condition, i hit it once on my linux machine while trying to do something else.  will keep investigating.  Would be curious if you are hitting it consistently though.

Comment 6 Dongbo Yan 2017-06-23 10:27:43 UTC
Test with new oc client
oc v3.6.121
kubernetes v1.6.1+5115d708d7

could execute command with service-catalog flag successfully, but it is instable.
Sometimes cannot see installed templates in service-catalog, and sometimes can see all templates

Comment 7 Ben Parees 2017-06-23 18:13:25 UTC
This appears to be a race condition, our template creates the role and then the rolebinding:

https://github.com/openshift/origin/blob/master/examples/service-catalog/service-catalog.yaml#L115-L138

we can see from the output that the role was created successfully, but then immediately after it can't be found (some sort of caching issue).

    role "endpoint-accessor" created
    error: role.authorization.openshift.io "endpoint-accessor" not found

David tracked down what's happening here and has details.

Comment 8 David Eads 2017-06-23 18:15:20 UTC
Looks like the non-cached rule resolver in the virtual storage is actually using a cache.

Comment 9 Dongbo Yan 2017-06-28 03:04:49 UTC
Since this issue does not reproduce all the time, modify severity to low

Comment 10 Mo 2017-06-28 19:21:39 UTC
Fixed in https://github.com/openshift/origin/pull/14896

Comment 11 Dongbo Yan 2017-07-06 07:18:07 UTC
Verified
oc v3.6.135
kubernetes v1.6.1+5115d708d7

could move to verified

Comment 14 errata-xmlrpc 2017-08-10 05:28:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716