Bug 1861789 - Failed to decode HTML response from the OIDC server with error "invalid character '<' looking for beginning of value"
Summary: Failed to decode HTML response from the OIDC server with error "invalid chara...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oauth-apiserver
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.6.0
Assignee: Standa Laznicka
QA Contact: pmali
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-29 14:36 UTC by aghadge
Modified: 2020-10-27 16:21 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Some OIDC servers ignore "Accept: application/json" when requesting a flow that they don't support. Consequence: Such OIDC servers may respond with an HTML page that the operator fails to parse as it's expecting json. It may be hard for a user to understand what the error is. Fix: Improve the error message by adding the URL of the page the authentication operator tried to reach. Result: It should now be more clear why the authentication operator fails in these cases.
Clone Of:
Environment:
Last Closed: 2020-10-27 16:21:20 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github goharbor harbor issues 9457 0 None closed err: oidc: failed to decode provider discovery object 2021-02-16 20:22:04 UTC
Github openshift cluster-authentication-operator pull 313 0 None closed Bug 1861789: improve password-grant discovery message on non-JSON server response 2021-02-16 20:22:05 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:21:39 UTC

Description aghadge 2020-07-29 14:36:41 UTC
Description of problem:
----------------------
Failure to decode error response from OIDC server if the response is in html format instead of json.

Version-Release number of selected component (if applicable):
------------------------------------------------------------
Red Hat Openshift Container Platform 4.5

How reproducible:
----------------
Always

Steps to Reproduce:
------------------
- In one of the customer cluster the authentication was failing with below error and "cluster-authentication-operator" was in degraded state

E0716 10:07:07.949783       1 oauth.go:75] failed to honor IDP v1.IdentityProvider{Name:"PingAuthorization", MappingMethod:"claim", IdentityProviderConfig:v1.IdentityProviderConfig{Type:"OpenID", BasicAuth:(*v1.BasicAuthIdentityProvider)(nil), GitHub:(*v1.GitHubIdentityProvider)(nil), GitLab:(*v1.GitLabIdentityProvider)(nil), Google:(*v1.GoogleIdentityProvider)(nil), HTPasswd:(*v1.HTPasswdIdentityProvider)(nil), Keystone:(*v1.KeystoneIdentityProvider)(nil), LDAP:(*v1.LDAPIdentityProvider)(nil), OpenID:(*v1.OpenIDIdentityProvider)(0xc0002e0840), RequestHeader:(*v1.RequestHeaderIdentityProvider)(nil)}}: error attempting password grant flow: failed to decode response from the OIDC server: invalid character '<' looking for beginning of value

- Later it was identified that the issue was happening because the Token Request was blocked by customer's corporate firewall.

- Here the initial error message "error attempting password grant flow" indicates issue with the password grant flow.

		// openshift CR validating in kube-apiserver does not allow
		// challenge-redirecting IdPs to be configured with OIDC so it is safe
		// to allow challenge-issuing flow if it's available on the OIDC side
		challengeFlowsAllowed, err := c.checkOIDCPasswordGrantFlow(
			ctx,
			urls.Token,
			openIDConfig.ClientID,
			openIDConfig.CA,
			openIDConfig.ClientSecret,
		)
		if err != nil {
			return nil, fmt.Errorf("error attempting password grant flow: %v", err) <<


- But the later response i.e. "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value" is failing to give clear indication of the issue

- Customer mentioned that the error response returned by the firewall was in html and not in json.

- Maybe because of that the actual error response was not printed and instead it prints the error "invalid character '<' looking for beginning of value"

- I also came across below link which somewhere indicates that the error is from "go-oidc" pkg which only expects json response instead of the html response.

https://github.com/goharbor/harbor/issues/9457
 
Actual results:
--------------
- In this case firewall blocking the token request was the actual error but instead of firewall error we got the response as "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value"
- This makes hard to identify and troubleshoot the issue

Expected results:
----------------
- Instead of the response as "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value", we should get the actual firewall error in response. 


Additional info:
---------------

- Is it mandatory to have the error response in json format only to print it clearly in the logs?

- Is there any configuration which can allow to print the html responses also.

Comment 1 Standa Laznicka 2020-08-04 07:49:20 UTC
Changing to low severity, only happens in misconfigured environments

Comment 6 Standa Laznicka 2020-08-06 13:25:33 UTC
Note that the fix consists of adding the URL when the parsing of the HTTP response fails so that the URL access can be attempted from inside the pod/wherever necessary, we won't be adding the full HTTP response's body to our error messages.

Comment 9 errata-xmlrpc 2020-10-27 16:21:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.