Bug 1777137 - OpenShift Authentication pods crash when badly formatted certificate is provided
Summary: OpenShift Authentication pods crash when badly formatted certificate is provided
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.2.z
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.6.0
Assignee: Standa Laznicka
QA Contact: pmali
Whiteboard: 4.5
Depends On:
TreeView+ depends on / blocked
Reported: 2019-11-27 01:50 UTC by Robert Bost
Modified: 2020-10-27 15:55 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Lack of validation of CMs/secrets might cause the authentication operator to rollout a new deployment of oauth-server with invalid/missing files. Consequence: oauth-server pods won't be able to run properly and might crash. Fix: Make the authentication operator validate the input CMs and secrets. Result: New deployment should not get rolled out when the CMs and secrets referenced in the oauth/cluster configuration contain invalid data.
Clone Of:
Last Closed: 2020-10-27 15:54:20 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-authentication-operator pull 222 0 None closed Bug 1777137: add observation of idp config and validation of its cm/secrets 2021-02-05 12:15:40 UTC
Github openshift cluster-authentication-operator pull 312 0 None closed Bug 1777137: don't post new observed config on resources validation 2021-02-05 12:15:40 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:55:08 UTC

Description Robert Bost 2019-11-27 01:50:29 UTC
Description of problem:

The following error was seen in oauth pod logs after configuring a new idp client keypair. I would expect the authentication-operator to check formatting before proprogating to Pods:

panic: Error building BasicAuthPasswordIdentityProvider client: error loading x509 keypair from cert file /var/config/user/idp/0/secret/v4-0-config-user-idp-0-tls-client-cert/tls.crt and key file /var/config/user/idp/0/secret/v4-0-config-user-idp-0-tls-client-key/tls.key: tls: failed to find any PEM data in certificate input

goroutine 1 [running]:
github.com/openshift/oauth-server/pkg/oauthserver.(*OAuthServerConfig).buildHandlerChainForOAuth(0xc000461500, 0x1be6b00, 0xc0003a03e0, 0xc0002a8c40, 0x16b57a0, 0xc0005e9638)
	/go/src/github.com/openshift/oauth-server/pkg/oauthserver/oauth_apiserver.go:303 +0xee
github.com/openshift/oauth-server/vendor/k8s.io/apiserver/pkg/server.completedConfig.New.func1(0x1be6b00, 0xc0003a03e0, 0x1be6b00, 0xc0003a03e0)
	/go/src/github.com/openshift/oauth-server/vendor/k8s.io/apiserver/pkg/server/config.go:444 +0x45
github.com/openshift/oauth-server/vendor/k8s.io/apiserver/pkg/server.NewAPIServerHandler(0x18dc66b, 0xf, 0x1c1b7c0, 0xc0004bbec0, 0xc0003a0380, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/oauth-server/vendor/k8s.io/apiserver/pkg/server/handler.go:96 +0x2fc
github.com/openshift/oauth-server/vendor/k8s.io/apiserver/pkg/server.completedConfig.New(0xc0002a8c40, 0x0, 0x0, 0x18dc66b, 0xf, 0x1c378a0, 0x2bc4f80, 0xc0002a8c40, 0x0, 0x0)
	/go/src/github.com/openshift/oauth-server/vendor/k8s.io/apiserver/pkg/server/config.go:446 +0x124
github.com/openshift/oauth-server/pkg/oauthserver.completedOAuthConfig.New(0xc0003a0360, 0xc000461508, 0x1c378a0, 0x2bc4f80, 0x4, 0x1c00dc0, 0xc000610e00)
	/go/src/github.com/openshift/oauth-server/pkg/oauthserver/oauth_apiserver.go:286 +0x70
github.com/openshift/oauth-server/pkg/cmd/oauth-server.RunOsinServer(0xc000258600, 0xc00009c720, 0xd07, 0xf07)
	/go/src/github.com/openshift/oauth-server/pkg/cmd/oauth-server/server.go:40 +0x8a
github.com/openshift/oauth-server/pkg/cmd/oauth-server.(*OsinServer).RunOsinServer(0xc00048ce80, 0xc00009c720, 0x5ac320, 0x16a1ce0)
	/go/src/github.com/openshift/oauth-server/pkg/cmd/oauth-server/cmd.go:91 +0x35c
github.com/openshift/oauth-server/pkg/cmd/oauth-server.NewOsinServer.func1(0xc0001fc500, 0xc0004a4a40, 0x0, 0x2)
	/go/src/github.com/openshift/oauth-server/pkg/cmd/oauth-server/cmd.go:39 +0xf4
github.com/openshift/oauth-server/vendor/github.com/spf13/cobra.(*Command).execute(0xc0001fc500, 0xc0004a49e0, 0x2, 0x2, 0xc0001fc500, 0xc0004a49e0)
	/go/src/github.com/openshift/oauth-server/vendor/github.com/spf13/cobra/command.go:760 +0x2ae
github.com/openshift/oauth-server/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0001fc280, 0xc0001fc280, 0x0, 0x0)
	/go/src/github.com/openshift/oauth-server/vendor/github.com/spf13/cobra/command.go:846 +0x2ec
	/go/src/github.com/openshift/oauth-server/cmd/oauth-server/main.go:41 +0x2cf

Version-Release number of selected component (if applicable): 4.2

How reproducible: always

Steps to Reproduce:
1. Create a secret with badly formatted certificate (i.e. missing header or footer lines
2. Operator will proceed and oauth-server pods will crash

Actual results: Crashing pods

Expected results: Operator to validate certificate

Comment 1 Standa Laznicka 2019-11-27 08:55:45 UTC
We are looking into improving input validation in 4.4 as a part of stabilization for the authentication operator. Until then, I'd suggest formatting your certificates properly.

Comment 13 errata-xmlrpc 2020-10-27 15:54:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.