Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1607913

Summary: web-console returns a blank page and tls handshake errors after oc cluster up --enable=web-console
Product: OpenShift Container Platform Reporter: Jason Montleon <jmontleo>
Component: ocAssignee: Matt Rogers <mrogers>
Status: CLOSED CURRENTRELEASE QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.11.0CC: aos-bugs, deads, jaboyd, jmatthew, jokerman, mfojtik, mmccomas, mrogers, spadgett, xxia
Target Milestone: ---   
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-21 15:16:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1621909, 1624257, 1624320    

Description Jason Montleon 2018-07-24 14:31:15 UTC
Description of problem:
After running 'oc cluster up --enable=web-console' the web console returns a blank page instead of a login.

The logs show tls handshake errors.

Version-Release number of selected component (if applicable):
oc v3.11.0-alpha.0+c1054f4-508
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.17.0.1:8443
openshift v3.11.0-alpha.0+c1054f4-508
kubernetes v1.11.0+d4cacc0

How reproducible:
seems always. 

Steps to Reproduce:
1.oc cluster up --routing-suffix=172.17.0.1.nip.io --public-hostname=172.17.0.1 --base-dir=/tmp/openshift.local.clusterup --tag=latest --image=docker.io/openshift/origin-\\${component}:\\${version} --enable=service-catalog,template-service-broker,router,registry,web-console,persistent-volumes,sample-templates,rhel-imagestreams,automation-service-broker
2. Browse to https://172.17.0.1:8443/console/

Actual results:
Web console serves a blank page, can't login. Logs show tls handshake errors

Expected results:
Console works

Additional info:
If the automation service broker is installed I see the same errors when the service catalog tries to communicate with it.

I tried the 3.11 tag and saw the same thing.

Log:
oc logs -f -n openshift-web-console                 webconsole-75c9658999-jz896
W0719 14:36:58.673138       1 start.go:93] Warning: config.clusterInfo.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console, web console start will continue.
W0719 14:36:58.673295       1 start.go:93] Warning: config.clusterInfo.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console, web console start will continue.
I0719 14:36:59.008432       1 start.go:208] OpenShift Web Console Version: v3.11.0-alpha.0+93d5ebe-3
I0719 14:36:59.009145       1 serve.go:89] Serving securely on 0.0.0.0:8443
I0719 15:00:26.110568       1 logs.go:41] http: TLS handshake error from 10.13.57.76:60884: remote error: tls: bad certificate

Comment 1 Juan Vallejo 2018-07-24 20:30:26 UTC
David, could this perhaps be a webconsole component trying to communicate over http with the api server?

Comment 2 Juan Vallejo 2018-07-25 13:55:44 UTC
Related github issue: https://github.com/openshift/origin/issues/20365

Comment 3 Samuel Padgett 2018-07-25 13:58:20 UTC
This is the only place the console pod talks to the API server:

https://github.com/openshift/origin-web-console-server/blob/16717b7a44cb2089412b80a80f615220b5f290b7/pkg/assets/apiserver/asset_apiserver.go#L129-L138

Comment 4 Jason Montleon 2018-07-25 14:42:28 UTC
In response to the question on the linked/closed issue, "@jmontleon Any chance you've previously clustered up with a different public hostname? It looks like the console can't discovery the OAuth endpoints because the certificate is not trusted.":

I typically use the public hostname in step one from the original comment. I won't say I've never used anything else, but I'm also pretty careful to wipe out /tmp/openshift.local.clusterup (base-dir specified above) in between runs of oc cluster up so any old configuration shouldn't play a role.

10.13.57.76:60884 is my hosts address and I'll see one line for each time I attempt to connect, typically from a different port and I believe this is firefox.

If I browse directly to the pod at https://172.17.0.10:8443/console I get a login page, though things break down when it redirects me after attempting to login.

So I don't think the web consoles back end communication is broken. Rather it seems to be something to do with communicating with the client and I'm wondering if this came about from a recently merged PR to use the service cert signing project: https://github.com/openshift/origin/commit/2c44605dc4db2f7d9b37f82f9350d03509207d31#diff-6e5712ecacc2343a0baf01e32d2f20fc

Comment 5 Jason Montleon 2018-07-25 15:50:48 UTC
oc logs -f -n kube-system                           master-api-localhost 

I see this when I try to refresh the page:

I0725 15:49:09.103353       1 logs.go:49] http: proxy error: x509: certificate signed by unknown authority
I0725 15:49:09.995597       1 logs.go:49] http: proxy error: x509: certificate signed by unknown authority

Comment 6 Samuel Padgett 2018-07-25 15:53:40 UTC
Looks like the proxy doesn't like the console certificate, which uses a service serving cert secret.

Comment 7 Juan Vallejo 2018-08-01 14:59:57 UTC
Reassigning to David as this deals with certs.

Comment 8 Simo Sorce 2018-08-13 09:54:04 UTC
I can repro the error spti by the web console after a simple oc cluster up

Matt, can you take this ?

Comment 9 John Matthews 2018-08-23 17:31:53 UTC
Hi Matt,

Do we know if this BZ will be addressed in 3.11.

As far as I know the web-console is not usable from oc cluster up with 3.11 right now.

Comment 10 Matt Rogers 2018-08-24 21:12:58 UTC
Hi John, yes, I've been working on a fix. 

The problem stems from the kube-apiserver only trusting the service-signer.crt CA (which is created on-disk at the beginning of cluster up) for the webconsole proxy, which is different than the CA that is created by the service-serving-cert-signer operator and used to sign the webconsole server certificate. My fix reads the service-serving-cert-signer's CA-bundle configmap to dynamically update the proxy transport. 

So far this approach works, however I still need to make sure that the update will only happen on error and the CA-bundle configmap has an update.

Comment 11 Jason Montleon 2018-08-31 21:51:04 UTC
Is removing the cert signer work until it is functional not an option? Given that the web-console, service catalog, and automation-service-broker are all affected it seems like this should be addressed for 3.11.0?

I've had things seemingly much more functional by reverting two PR's and making a few other small changes.

c2ed5ecbff3cba228d04d0588cd965d7470fce3b
2c44605dc4db2f7d9b37f82f9350d03509207d31

Comment 12 David Eads 2018-09-05 14:00:43 UTC
We could remove it, but doing so would effectively remove the openshift-apiserver, so it's probably not a good solution.

There is a pull open and under review for 3.11.

Comment 13 John Matthews 2018-09-06 15:46:08 UTC
Assume we should align this BZ to 3.11 as the below PR fixed the issue
https://github.com/openshift/origin/pull/20868

Comment 14 Xingxing Xia 2018-09-14 03:25:42 UTC
https://github.com/openshift/origin/pull/20868 is merged in > v3.11.0-0.29.0. Will test.

Comment 15 Xingxing Xia 2018-09-14 14:30:19 UTC
Verified in oc v3.11.6. Web console returns login and pages well. Please move to ON_QA.
But found another bug 1628968 reported separately

Comment 16 Luke Meyer 2018-12-21 15:16:36 UTC
Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content.