Bug 1739262 - Authentication attempts are failing with Request Header Authentication Provider
Summary: Authentication attempts are failing with Request Header Authentication Provider
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.1.0
Hardware: Unspecified
OS: Linux
Target Milestone: ---
: 4.1.z
Assignee: Standa Laznicka
QA Contact: scheng
Depends On: 1741822 1744995 1750809 1764558
TreeView+ depends on / blocked
Reported: 2019-08-08 22:01 UTC by Anand Paladugu
Modified: 2023-03-24 15:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1764558 (view as bug list)
Last Closed: 2019-12-17 07:17:12 UTC
Target Upstream Version:

Attachments (Terms of Use)
Login attempt via oc cli (5.99 KB, text/plain)
2019-08-08 22:03 UTC, Anand Paladugu
no flags Details
auth CR definition, .pem file, attempt to auth against oauth-openshift (11.83 KB, text/plain)
2019-08-08 22:06 UTC, Anand Paladugu
no flags Details

System ID Private Priority Status Summary Last Updated
Github openshift origin pull 24124 0 'None' closed Bug 1739262: RequestHeaders IdP: fix TLS handshake [4.1] 2020-08-12 17:44:17 UTC
Red Hat Product Errata RHBA-2019:4084 0 None None None 2019-12-17 07:17:15 UTC

Description Anand Paladugu 2019-08-08 22:01:15 UTC
Description of problem:

Authentication attempts are failing with Request Header Authentication Provider

Version-Release number of selected component (if applicable):

How reproducible:

Always in the customer environment.

Steps to Reproduce:
1. Configure Request Header authentication provider as documented in ocp 4.1 DOCS.  Auth proxy (httpd) conf is provided in Additional info below.
2. Try to login from console or via oc cli as the SSO user

Actual results:

Authentication attempts via UI are failing and must-gather logs do not show any specific reason. must-gather attached along with logs from the httpd proxy.

OC Cli login attempts result "unexpected response: 500".  cli-login-linux.txt is attached.

If cusomer tries to authenticate directly with oauth-openshift providing the creds and certificates. request-header-config.txt is attached.

Expected results:

Successful authentication via UI and CLI.

Additional info:

Customer claims that the same procedure worked in 3.11

Customers httpd auth proxy config is as below:

SSLProxyCACertificateFile /pki/authproxyca.crt

SSLCACertificateFile /pki/smartcard.crt
  SSLVerifyClient require
  SSLVerifyDepth 4
  SSLOptions +ExportCertData

Here is the new file

LoadModule request_module modules/mod_request.so
ServerName auth.apps.amlosdev.aws.faa.gov

LogLevel trace6

<VirtualHost *:8443>

  DocumentRoot /var/www/html
  SSLEngine on
  SSLCertificateFile /pki/web.crt
  SSLCertificateKeyFile /pki/web.key
  SSLCACertificateFile /pki/ca.crt

  SSLProxyEngine on
  SSLProxyCACertificateFile /pki/ca.crt
  # It's critical to enforce client certificates on the Master.  Otherwise
  # requests could spoof the X-Remote-User header by accessing the Master's
  # /oauth/authorize endpoint directly.
  SSLProxyMachineCertificateFile /pki/authproxy.pem

  # Send all requests to the console
  RewriteEngine              On
  RewriteRule     ^/console(.*)$     https://%{HTTP_HOST}/console$1 [R,L]

  # In order to using the challenging-proxy an X-Csrf-Token must be present.
  RewriteCond %{REQUEST_URI} ^/challenging-proxy
  RewriteCond %{HTTP:X-Csrf-Token} ^$ [NC]
  RewriteRule ^.* - [F,L]

  <Location /challenging-proxy/oauth/authorize>
      ProxyPass https://oauth-openshift.apps.amlosdev.aws.faa.gov:443/oauth/authorize
      AuthName "SSO Login"
      # For Kerberos
      AuthType GSSAPI
      Require valid-user
      RequestHeader set X-Remote-User %{REMOTE_USER}s

      GssapiCredStore keytab:/krb5/auth-proxy.keytab
      GssapiBasicAuth On
      GssapiImpersonate  On
      GssapiConnectionBound On

    <Location /login-proxy/oauth/authorize>
    ProxyPass https://oauth-openshift.apps.amlosdev.aws.faa.gov:443/oauth/authorize

      AuthName "SSO Login"
      AuthType GSSAPI
      Require valid-user
      RequestHeader set X-Remote-User %{REMOTE_USER}s
      GssapiUseSessions On
      Session On
      SessionCookieName os_gssapi_session path=/private;httponly;secure;

      GssapiCredStore keytab:/krb5/auth-proxy.keytab
      GssapiBasicAuth On
      GssapiImpersonate On
      GssapiConnectionBound On

      ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/login-proxy/oauth/authorize"><body>Kerberos authentication did not pass.</body></html>'


RequestHeader unset X-Remote-User

Comment 1 Anand Paladugu 2019-08-08 22:03:23 UTC
Created attachment 1601949 [details]
Login attempt via oc cli

Comment 2 Anand Paladugu 2019-08-08 22:06:12 UTC
Created attachment 1601950 [details]
auth CR definition,   .pem file,  attempt to auth against oauth-openshift

Comment 3 Anand Paladugu 2019-08-08 22:07:25 UTC
I am unable to upload the must-gather to bz due to size.  Let me know if you have a prefered location to access the file

Comment 6 Anand Paladugu 2019-08-09 14:02:15 UTC
Thanks Standa.  So ca.crt should be the pem bundle containing both the client cert and the signer cert.   Is "openshift-proxy-signer" the right signer to use in this case ?

Comment 7 Anand Paladugu 2019-08-09 14:19:52 UTC
Also can that single data item be named "ca-bundle.crt" like in other config maps or does it have to be named "ca.crt" ?

Comment 8 Anand Paladugu 2019-08-09 17:19:00 UTC
Provided the update below to customer and waiting for this respoonce.


Engineering provided an update that the only data field used in ca-config-map is the ca.crt.

        // ca is an optional reference to a config map by name containing the PEM-encoded CA bundle.
        // It is used as a trust anchor to validate the TLS certificate presented by the remote server.
        // The key "ca.crt" is used to locate the data.
        // If specified and the config map or expected key is not found, the identity provider is not honored.
        // If the specified ca data is not valid, the identity provider is not honored.
        // If empty, the default system roots are used.
        // The namespace for this config map is openshift-config.
        // +optional
        CA ConfigMapNameReference `json:"ca"`

Can you include all certs in the bundle in ca.crt field, remove all other data items in that configmap and test again ?

If the issue is resolved then I can have this information reflected in the docs by opening a doc bug.

If the issue is still persisting, then please upload a new must-gather.


Comment 9 Anand Paladugu 2019-08-12 15:02:53 UTC
Customer removed other entries from the config map, verified that the ca in the config map signed the client cert in the auth proxy, and re-ran the tests.  Same results.  New must gather is uploaded to the case..

Comment 12 Anand Paladugu 2019-08-14 01:39:58 UTC
Not sure if you were planning to have a call tomorrow AM.  I would love to , but let me know if the following is correct just in case if we dont speak to each other.

/pki/authproxyca.crt  is the cert (self signed) with CN=openshift-proxy-signer@auth, Will ask customer to include this in the ca.crt field of ca-config-map.

/pki/authproxy.pem  has CN=system:proxy  but is also issued by openshift-proxy-signer@auth.  Customer is using this in the httpd conf as below.

 SSLProxyMachineCertificateFile /pki/authproxy.pem 

and he has also used this cert in testing to authenticate directly against ocp oauth server (example below). Is this the right certificate to use in proxy config ?

curl -k -v -L --cert /pki/authproxy.pem https://oauth-openshift.apps.amlosdev.aws.faa.gov/oauth/token/request -H "X-Remote-User: test"



Comment 14 Standa Laznicka 2019-08-14 08:31:50 UTC
/pki/authproxyca.crt is a self-signed CA certificate and is the only certificate that should appear in the config map.

/pki/authproxy.pem is the cert/key bundle issued by the above CA. I saw the config in the case.

Since the `ca-config-map` does not contain the CA cert in /pki/authproxyca.crt, oauth-server rejects the curl request (you get 401), because it does not trust your certificate.

Comment 15 Anand Paladugu 2019-08-14 12:29:00 UTC
Ok. Thanks

Comment 17 Standa Laznicka 2019-08-16 16:09:45 UTC
Engineering found the cause of the bug, we will deliver a fix through the usual release channels.

Comment 18 Anand Paladugu 2019-08-16 19:31:46 UTC
Ok. Thanks.  Let me know the approx timeline, so I can update the customer.

Comment 21 Anand Paladugu 2019-09-10 13:01:24 UTC
@Standa.  any update on this Bug?  Both issues it depends are in verified state.

Comment 23 Standa Laznicka 2019-09-10 14:48:20 UTC
We found a different issue further down that would still prevent the use of Request Header IdP. The fix has been approved and will be merged shortly, it will then go through the common release chain.

Comment 24 Standa Laznicka 2019-09-30 11:41:30 UTC

Both bugs were fixed and their respective bugzillas are in verified state. Customer's issue should go away once the fix to https://bugzilla.redhat.com/show_bug.cgi?id=1750809 is released.

Comment 26 Anand Paladugu 2019-10-21 17:03:00 UTC

"https://bugzilla.redhat.com/show_bug.cgi?id=1750809" is resolved, customer upgraded to the version with updates, but he is still seeing the issue.

Can you review the attached logs (in case) at your earliest?   I have upped the severity to urgent as the customer has been waiting for some time and may escalate.

Comment 27 Standa Laznicka 2019-10-22 15:04:59 UTC
I was able to figure out what happens. The technical details are a bit complicated, but the outcome is that the oauth-server does not announce all the CAs it will accept client certificates from, which causes the Apache proxy not to even try to use its client certificate which would actually validate the request as valid.

I am not sure there is a workaround that would make the Apache server force-use of its client cert, or whether we can think of a way around this before it's fixed.

Comment 28 Anand Paladugu 2019-10-22 18:56:29 UTC
I am assuming customers would be ok for a workaround if it's not a huge change for them and if it's something that can be automated.

Why is it that we only accept certs from certain CA's?  So is this a middle ground between accepting certs of public trusted CA's Vs Private CA's that OCP is taking ?

Comment 29 Standa Laznicka 2019-10-24 08:15:22 UTC
I'm yet to figure out whether a workaround is possible. I think I know of a way to use "unsupportedConfigOverrides", but will investigate it first.

"Why is it that we only accept certs from certain CA's?"
- this is how TLS works and is dependent on the server configuration, which was wrong and is being fixed in https://github.com/openshift/oauth-server/pull/19. I even created a reproducer for the issue using the customer config ported to my cluster to make sure that this is the last fix we need.

"So is this a middle ground between accepting certs of public trusted CA's Vs Private CA's that OCP is taking ?"
- No. This was a misconfiguration we did because we did not realize the changes in 4.x oauth-server would cause the side-effect they did.

If you want specific technical details about the issue, I tried to explain it as clear as I could in the description of https://github.com/openshift/oauth-server/pull/19

Comment 30 Standa Laznicka 2019-10-24 13:47:23 UTC
There is a possible workaround:

1. call `oc get deployment -n openshift-authentication oauth-openshift -o yaml | grep -e "mountPath:.*v4-0-config-user-idp-[0-9]*-ca" | cut -d: -f2`

2. append /ca.crt behind the path from the step 1.
3. oc edit authentication.operator
  - modify spec so that it looks somehow like this:

  managementState: Managed
      clientCA: THE_PATH
replacing THE_PATH with what you've got in step 2, in my case that would be `/var/config/user/idp/0/configmap/v4-0-config-user-idp-0-ca/ca.crt`

4. save the object, wait for the auth pods to come back up

This should make `oc login` work. As a trade-off, it will break the cluster-admin being directly able to query the /metrics endpoint of the oauth-server, but I don't think that's commonly being done, and from my testing, prometheus is still capable of scraping the endpoint, so it is still possible to get the data somewhere else.

Comment 31 Standa Laznicka 2019-10-24 13:50:01 UTC
Note that the presence of `unsupportedConfigOverrides` will prevent you from upgrading, so you'll have to remove this field before an upgrade is attempted.

Comment 32 Anand Paladugu 2019-10-27 22:38:49 UTC
Thanks @Standa.   Customer may object to the inability to upgrade.  If they remove the setting before the upgrade and restore if after upgrade,  would that break anything?

Comment 33 Standa Laznicka 2019-10-29 09:04:39 UTC
No, even the upgrade should work fine, only the IdP's users will be unavailable for that period. The need to do this workaround will of course be removed once we manage to fix the bug, hopefully we'll get the fix into 4.1 rather sooner.

Comment 37 errata-xmlrpc 2019-12-17 07:17:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.