Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1621909

Summary: API endpoint, /version/openshift, missing with latest oc cluster up
Product: OpenShift Container Platform Reporter: John Matthews <jmatthew>
Component: Service BrokerAssignee: Jason Montleon <jmontleo>
Status: CLOSED CURRENTRELEASE QA Contact: Zhang Cheng <chezhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, deads, dzager, fabian, jiazha, jmatthew, jmontleo, jokerman, jvallejo, mmccomas, spahl, tsanders, xxia, zitang
Target Milestone: ---   
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: 1621724 Environment:
Last Closed: 2018-12-21 15:23:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1607913, 1624320    
Bug Blocks:    

Description John Matthews 2018-08-23 20:41:50 UTC
+++ This bug was initially created as a clone of Bug #1621724 +++

Description of problem:
The /version/openshift endpoint appears to be gone when running oc cluster up with oc version v3.11.0-alpha.0+cd7d048-1070.

Version-Release number of selected component (if applicable):
oc v3.11.0-alpha.0+cd7d048-1070

How reproducible:
Always

Steps to Reproduce:
1. Run oc cluster up
2. Run oc version
3. See the server openshift version is missing:

oc v3.11.0-alpha.0+cd7d048-1070
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://localhost:8443
kubernetes v1.11.0+d4cacc0

Actual results:
oc version is missing

Expected results:
oc version is reported correctly

Additional info:
This breaks the ansible kubernetes modules and openshift dynamic client

I saw a suggestion that it may have moved to /version/oversion but this returns a 404 as well.

My understanding is this is a change intended for 4.0, however I don't see a release-3.11 branch in the openshift/origin github repo so my concern is that this is going to make its way into 3.11 upstream resulting different behavior upstream and downstream, and poitentially downstream where it would break functionality we rely on.

oc --loglevel=6  version 
I0823 17:52:20.859731   25868 loader.go:359] Config loaded from file /etc/origin/node/node.kubeconfig
I0823 17:52:20.860519   25868 loader.go:359] Config loaded from file /etc/origin/node/node.kubeconfig
oc v3.11.0-alpha.0+cd7d048-1070
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO
I0823 17:52:20.878096   25868 round_trippers.go:405] GET https://localhost:8443/version 200 OK in 16 milliseconds
I0823 17:52:20.879394   25868 round_trippers.go:405] GET https://localhost:8443/version/openshift?timeout=32s 404 Not Found in 0 milliseconds

Server https://localhost:8443
kubernetes v1.11.0+d4cacc0

curl -i -k -XGET -H "Authorization: Bearer bW-FUYm9LaMjBbrMxzL7vu_XCbDlJNbC1hPfJNFQa_M" https://172.17.0.1:8443/version/openshift
HTTP/1.1 404 Not Found

On a downstream 3.11 install this works and it worked prior to yesterday with upstream:
oc version
oc v3.11.0-0.20.0
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.121.108.nip.io:8443
openshift v3.11.0-0.20.0
kubernetes v1.11.0+d4cacc0
[root@192 ~]# oc --loglevel=6 version
I0823 17:54:18.693103   28616 loader.go:359] Config loaded from file /root/.kube/config
I0823 17:54:18.694573   28616 loader.go:359] Config loaded from file /root/.kube/config
oc v3.11.0-0.20.0
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO
I0823 17:54:18.705382   28616 round_trippers.go:405] GET https://192.168.121.108.nip.io:8443/version 200 OK in 9 milliseconds
I0823 17:54:18.706514   28616 round_trippers.go:405] GET https://192.168.121.108.nip.io:8443/version/openshift?timeout=32s 200 OK in 0 milliseconds

Server https://192.168.121.108.nip.io:8443
openshift v3.11.0-0.20.0
kubernetes v1.11.0+d4cacc0

--- Additional comment from David Eads on 2018-08-23 15:24:40 EDT ---

`oc cluster up` is not a cluster that we support.  It produces something to play with as a preview and for openshift developers to work against, but the topology bears no relationship to the openshift-ansible created cluster.

`oc cluster up` sets up a kube-apiserver and aggregates in an openshift-apiserver.  kube-apiserver's have no openshift version. It's not a concept that makes sense because the level of openshift being aggregated can be anything from 3.8 to this release +1.  Yesterday's build happened to include an inaccurate endpoint that reported an incorrect version and we removed it as we collapsed fully onto the upstream kube-apiserver.

The openshift-ansible created cluster will still report the /version/openshift for 3.11.

Comment 1 John Matthews 2018-08-23 20:53:45 UTC
Summary of the issue.

A recent change was introduced to 3.11 that removes the endpoint '/version/openshift' from clusters brought up by 'oc cluster up'.

Clusters brought up from openshift-ansible will continue to have /version/openshift.

APBs are relying on, /version/openshift, to determine if the cluster is an openshift or k8s cluster.  With this recent change APBs will be broken on oc cluster up with 3.11, this would also include minishift.


Also worth noting, OCP 4.0 release will likely remove /version/openshift as well from openshift-ansible derived clusters, so long term looks best that APBs are updated to avoid relying on /version/openshift.

For background info.

APBs are currently relying on the below to determine different of openshift/k8s
  https://github.com/openshift/openshift-restclient-python/blob/master/openshift/dynamic/client.py#L64)

We will update the APBs to leverage a different mechanism for determining openshift/k8s. Likely using https://github.com/ansible/ansible/blob/a8d4bf86421d151d8df7132e8e87d04b6662f45a/lib/ansible/plugins/lookup/k8s.py

Comment 3 Zhang Cheng 2018-08-29 08:54:40 UTC
The testing and bug verification will focus on two sides.
1. oc cluster up should work well with asb, and provision/deprovision/bind/undind/update a serviceinstance should succeed.
2. All updated(rebuild) APB images should work well in openshift ansible env, and no regression issue.

Comment 5 Zhang Cheng 2018-08-31 08:30:44 UTC
I was trying to double check our changes in oc cluster up env. But "cannot setup a cluster while enalbe service-catalog and automation-service-broker by oc cluster up", refer to bug https://bugzilla.redhat.com/show_bug.cgi?id=1624320

Comment 6 Zhang Cheng 2018-09-03 05:48:14 UTC
John,

Verification in oc cluster up env is blocking by another bug https://bugzilla.redhat.com/show_bug.cgi?id=1624320 (target release is 3.11.z), could we change the target release to 3.11.z either?

Furthermore, regression test in openshift ansible env is on going. I will add comments for test result later.

Comment 7 Zhang Cheng 2018-09-05 02:03:52 UTC
Jason, could you help to provide response for comment 6? Thanks.

Q: Verification in oc cluster up env is blocking by another bug https://bugzilla.redhat.com/show_bug.cgi?id=1624320 (target release is 3.11.z), could we change the target release to 3.11.z either?

Comment 8 John Matthews 2018-09-05 13:57:51 UTC
Aligning to 3.11.z as oc cluster up env is currently broken and blocked by bug 1624320 which is blocked by bug 1607913

The root issue looks to be cert signing is broken with oc cluster up, tracked by bug 1607913

An upstream PR is being worked to address here:
  https://github.com/openshift/origin/pull/20760

Comment 9 Jason Montleon 2018-09-06 12:54:45 UTC
I think this should be testable now. https://github.com/openshift/origin/pull/20868 was merged for 3.11 and appears to fix the cert problems.

Comment 10 Zhang Cheng 2018-09-07 02:17:34 UTC
Relate code was merged in v3.11.0-0.29.0 in downstream. The latest puddle of downstream is v3.11.0-0.28.0
Waiting image ready for test.

Comment 11 Zhang Cheng 2018-09-10 05:48:06 UTC
Double checked with latest oc, test Failed although https://github.com/openshift/origin/pull/20868 was merged in v3.11.0-0.29.0 and later, the original issue still exist.

$ oc version
oc v3.11.0-0.32.0
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://127.0.0.1:8443
kubernetes v1.11.0+d4cacc0


Command:
$ oc cluster up --enable='*,service-catalog,template-service-broker' --image='brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-${component}:${version}'

Comment 12 Jason Montleon 2018-09-10 13:53:32 UTC
Can you explain which original image still exist? This BZ was opened for the /version/openshift api going away and the APB's needing to be updated, but you pointed at a BZ for the cert signer operator.

As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1624320 we're not seeing problems with oc cluster up anymore. Can you review my two comments there and trying running oc cluster up again. If that or something else doesn't work can you please provide more detail?

Comment 13 Zhang Cheng 2018-09-11 03:27:57 UTC
We are waiting for clarification from developer, and will double check later. Please refer to comment 7, comment 8, comment 9 and comment 10 in BZ https://bugzilla.redhat.com/show_bug.cgi?id=1624320

Comment 14 Zhang Cheng 2018-09-12 03:39:10 UTC
Moving status to VERIFIED since ASB/APBs work well both in oc cluster up and openshift ansible env.

oc v3.11.0-0.32.0

Comment 15 Luke Meyer 2018-12-21 15:23:17 UTC
Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content.