Cause:
Some OIDC servers ignore "Accept: application/json" when requesting a flow that they don't support.
Consequence:
Such OIDC servers may respond with an HTML page that the operator fails to parse as it's expecting json. It may be hard for a user to understand what the error is.
Fix:
Improve the error message by adding the URL of the page the authentication operator tried to reach.
Result:
It should now be more clear why the authentication operator fails in these cases.
Description of problem:
----------------------
Failure to decode error response from OIDC server if the response is in html format instead of json.
Version-Release number of selected component (if applicable):
------------------------------------------------------------
Red Hat Openshift Container Platform 4.5
How reproducible:
----------------
Always
Steps to Reproduce:
------------------
- In one of the customer cluster the authentication was failing with below error and "cluster-authentication-operator" was in degraded state
E0716 10:07:07.949783 1 oauth.go:75] failed to honor IDP v1.IdentityProvider{Name:"PingAuthorization", MappingMethod:"claim", IdentityProviderConfig:v1.IdentityProviderConfig{Type:"OpenID", BasicAuth:(*v1.BasicAuthIdentityProvider)(nil), GitHub:(*v1.GitHubIdentityProvider)(nil), GitLab:(*v1.GitLabIdentityProvider)(nil), Google:(*v1.GoogleIdentityProvider)(nil), HTPasswd:(*v1.HTPasswdIdentityProvider)(nil), Keystone:(*v1.KeystoneIdentityProvider)(nil), LDAP:(*v1.LDAPIdentityProvider)(nil), OpenID:(*v1.OpenIDIdentityProvider)(0xc0002e0840), RequestHeader:(*v1.RequestHeaderIdentityProvider)(nil)}}: error attempting password grant flow: failed to decode response from the OIDC server: invalid character '<' looking for beginning of value
- Later it was identified that the issue was happening because the Token Request was blocked by customer's corporate firewall.
- Here the initial error message "error attempting password grant flow" indicates issue with the password grant flow.
// openshift CR validating in kube-apiserver does not allow
// challenge-redirecting IdPs to be configured with OIDC so it is safe
// to allow challenge-issuing flow if it's available on the OIDC side
challengeFlowsAllowed, err := c.checkOIDCPasswordGrantFlow(
ctx,
urls.Token,
openIDConfig.ClientID,
openIDConfig.CA,
openIDConfig.ClientSecret,
)
if err != nil {
return nil, fmt.Errorf("error attempting password grant flow: %v", err) <<
- But the later response i.e. "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value" is failing to give clear indication of the issue
- Customer mentioned that the error response returned by the firewall was in html and not in json.
- Maybe because of that the actual error response was not printed and instead it prints the error "invalid character '<' looking for beginning of value"
- I also came across below link which somewhere indicates that the error is from "go-oidc" pkg which only expects json response instead of the html response.
https://github.com/goharbor/harbor/issues/9457
Actual results:
--------------
- In this case firewall blocking the token request was the actual error but instead of firewall error we got the response as "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value"
- This makes hard to identify and troubleshoot the issue
Expected results:
----------------
- Instead of the response as "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value", we should get the actual firewall error in response.
Additional info:
---------------
- Is it mandatory to have the error response in json format only to print it clearly in the logs?
- Is there any configuration which can allow to print the html responses also.
Note that the fix consists of adding the URL when the parsing of the HTTP response fails so that the URL access can be attempted from inside the pod/wherever necessary, we won't be adding the full HTTP response's body to our error messages.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2020:4196
Description of problem: ---------------------- Failure to decode error response from OIDC server if the response is in html format instead of json. Version-Release number of selected component (if applicable): ------------------------------------------------------------ Red Hat Openshift Container Platform 4.5 How reproducible: ---------------- Always Steps to Reproduce: ------------------ - In one of the customer cluster the authentication was failing with below error and "cluster-authentication-operator" was in degraded state E0716 10:07:07.949783 1 oauth.go:75] failed to honor IDP v1.IdentityProvider{Name:"PingAuthorization", MappingMethod:"claim", IdentityProviderConfig:v1.IdentityProviderConfig{Type:"OpenID", BasicAuth:(*v1.BasicAuthIdentityProvider)(nil), GitHub:(*v1.GitHubIdentityProvider)(nil), GitLab:(*v1.GitLabIdentityProvider)(nil), Google:(*v1.GoogleIdentityProvider)(nil), HTPasswd:(*v1.HTPasswdIdentityProvider)(nil), Keystone:(*v1.KeystoneIdentityProvider)(nil), LDAP:(*v1.LDAPIdentityProvider)(nil), OpenID:(*v1.OpenIDIdentityProvider)(0xc0002e0840), RequestHeader:(*v1.RequestHeaderIdentityProvider)(nil)}}: error attempting password grant flow: failed to decode response from the OIDC server: invalid character '<' looking for beginning of value - Later it was identified that the issue was happening because the Token Request was blocked by customer's corporate firewall. - Here the initial error message "error attempting password grant flow" indicates issue with the password grant flow. // openshift CR validating in kube-apiserver does not allow // challenge-redirecting IdPs to be configured with OIDC so it is safe // to allow challenge-issuing flow if it's available on the OIDC side challengeFlowsAllowed, err := c.checkOIDCPasswordGrantFlow( ctx, urls.Token, openIDConfig.ClientID, openIDConfig.CA, openIDConfig.ClientSecret, ) if err != nil { return nil, fmt.Errorf("error attempting password grant flow: %v", err) << - But the later response i.e. "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value" is failing to give clear indication of the issue - Customer mentioned that the error response returned by the firewall was in html and not in json. - Maybe because of that the actual error response was not printed and instead it prints the error "invalid character '<' looking for beginning of value" - I also came across below link which somewhere indicates that the error is from "go-oidc" pkg which only expects json response instead of the html response. https://github.com/goharbor/harbor/issues/9457 Actual results: -------------- - In this case firewall blocking the token request was the actual error but instead of firewall error we got the response as "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value" - This makes hard to identify and troubleshoot the issue Expected results: ---------------- - Instead of the response as "failed to decode response from the OIDC server: invalid character '<' looking for beginning of value", we should get the actual firewall error in response. Additional info: --------------- - Is it mandatory to have the error response in json format only to print it clearly in the logs? - Is there any configuration which can allow to print the html responses also.