Description of problem: Authentication attempts are failing with Request Header Authentication Provider Version-Release number of selected component (if applicable): 4.1.0 How reproducible: Always in the customer environment. Steps to Reproduce: 1. Configure Request Header authentication provider as documented in ocp 4.1 DOCS. Auth proxy (httpd) conf is provided in Additional info below. 2. Try to login from console or via oc cli as the SSO user 3. Actual results: Authentication attempts via UI are failing and must-gather logs do not show any specific reason. must-gather attached along with logs from the httpd proxy. OC Cli login attempts result "unexpected response: 500". cli-login-linux.txt is attached. If cusomer tries to authenticate directly with oauth-openshift providing the creds and certificates. request-header-config.txt is attached. Expected results: Successful authentication via UI and CLI. Additional info: Customer claims that the same procedure worked in 3.11 Customers httpd auth proxy config is as below: ############# SSLProxyCACertificateFile /pki/authproxyca.crt SSLCACertificateFile /pki/smartcard.crt SSLVerifyClient require SSLVerifyDepth 4 SSLOptions +ExportCertData ############# Here is the new file ############################################################ LoadModule request_module modules/mod_request.so ServerName auth.apps.amlosdev.aws.faa.gov LogLevel trace6 LISTEN 8443 <VirtualHost *:8443> DocumentRoot /var/www/html SSLEngine on SSLCertificateFile /pki/web.crt SSLCertificateKeyFile /pki/web.key SSLCACertificateFile /pki/ca.crt SSLProxyEngine on SSLProxyCACertificateFile /pki/ca.crt # It's critical to enforce client certificates on the Master. Otherwise # requests could spoof the X-Remote-User header by accessing the Master's # /oauth/authorize endpoint directly. SSLProxyMachineCertificateFile /pki/authproxy.pem # Send all requests to the console RewriteEngine On RewriteRule ^/console(.*)$ https://%{HTTP_HOST}/console$1 [R,L] # In order to using the challenging-proxy an X-Csrf-Token must be present. RewriteCond %{REQUEST_URI} ^/challenging-proxy RewriteCond %{HTTP:X-Csrf-Token} ^$ [NC] RewriteRule ^.* - [F,L] <Location /challenging-proxy/oauth/authorize> ProxyPass https://oauth-openshift.apps.amlosdev.aws.faa.gov:443/oauth/authorize AuthName "SSO Login" # For Kerberos AuthType GSSAPI Require valid-user RequestHeader set X-Remote-User %{REMOTE_USER}s GssapiCredStore keytab:/krb5/auth-proxy.keytab GssapiBasicAuth On GssapiImpersonate On GssapiConnectionBound On </Location> <Location /login-proxy/oauth/authorize> ProxyPass https://oauth-openshift.apps.amlosdev.aws.faa.gov:443/oauth/authorize AuthName "SSO Login" AuthType GSSAPI Require valid-user RequestHeader set X-Remote-User %{REMOTE_USER}s GssapiUseSessions On Session On SessionCookieName os_gssapi_session path=/private;httponly;secure; GssapiCredStore keytab:/krb5/auth-proxy.keytab GssapiBasicAuth On GssapiImpersonate On GssapiConnectionBound On ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/login-proxy/oauth/authorize"><body>Kerberos authentication did not pass.</body></html>' </Location> </VirtualHost> RequestHeader unset X-Remote-User
Created attachment 1601949 [details] Login attempt via oc cli
Created attachment 1601950 [details] auth CR definition, .pem file, attempt to auth against oauth-openshift
I am unable to upload the must-gather to bz due to size. Let me know if you have a prefered location to access the file
Thanks Standa. So ca.crt should be the pem bundle containing both the client cert and the signer cert. Is "openshift-proxy-signer" the right signer to use in this case ?
Also can that single data item be named "ca-bundle.crt" like in other config maps or does it have to be named "ca.crt" ?
Provided the update below to customer and waiting for this respoonce. ======================================== Engineering provided an update that the only data field used in ca-config-map is the ca.crt. // ca is an optional reference to a config map by name containing the PEM-encoded CA bundle. // It is used as a trust anchor to validate the TLS certificate presented by the remote server. // The key "ca.crt" is used to locate the data. // If specified and the config map or expected key is not found, the identity provider is not honored. // If the specified ca data is not valid, the identity provider is not honored. // If empty, the default system roots are used. // The namespace for this config map is openshift-config. // +optional CA ConfigMapNameReference `json:"ca"` Can you include all certs in the bundle in ca.crt field, remove all other data items in that configmap and test again ? If the issue is resolved then I can have this information reflected in the docs by opening a doc bug. If the issue is still persisting, then please upload a new must-gather. =========================================
Customer removed other entries from the config map, verified that the ca in the config map signed the client cert in the auth proxy, and re-ran the tests. Same results. New must gather is uploaded to the case..
Not sure if you were planning to have a call tomorrow AM. I would love to , but let me know if the following is correct just in case if we dont speak to each other. /pki/authproxyca.crt is the cert (self signed) with CN=openshift-proxy-signer@auth, Will ask customer to include this in the ca.crt field of ca-config-map. /pki/authproxy.pem has CN=system:proxy but is also issued by openshift-proxy-signer@auth. Customer is using this in the httpd conf as below. SSLProxyMachineCertificateFile /pki/authproxy.pem and he has also used this cert in testing to authenticate directly against ocp oauth server (example below). Is this the right certificate to use in proxy config ? curl -k -v -L --cert /pki/authproxy.pem https://oauth-openshift.apps.amlosdev.aws.faa.gov/oauth/token/request -H "X-Remote-User: test" Thanks Anand
/pki/authproxyca.crt is a self-signed CA certificate and is the only certificate that should appear in the config map. /pki/authproxy.pem is the cert/key bundle issued by the above CA. I saw the config in the case. Since the `ca-config-map` does not contain the CA cert in /pki/authproxyca.crt, oauth-server rejects the curl request (you get 401), because it does not trust your certificate.
Ok. Thanks
Engineering found the cause of the bug, we will deliver a fix through the usual release channels.
Ok. Thanks. Let me know the approx timeline, so I can update the customer.
@Standa. any update on this Bug? Both issues it depends are in verified state.
We found a different issue further down that would still prevent the use of Request Header IdP. The fix has been approved and will be merged shortly, it will then go through the common release chain.
Anand, Both bugs were fixed and their respective bugzillas are in verified state. Customer's issue should go away once the fix to https://bugzilla.redhat.com/show_bug.cgi?id=1750809 is released.
@Standa "https://bugzilla.redhat.com/show_bug.cgi?id=1750809" is resolved, customer upgraded to the version with updates, but he is still seeing the issue. Can you review the attached logs (in case) at your earliest? I have upped the severity to urgent as the customer has been waiting for some time and may escalate.
I was able to figure out what happens. The technical details are a bit complicated, but the outcome is that the oauth-server does not announce all the CAs it will accept client certificates from, which causes the Apache proxy not to even try to use its client certificate which would actually validate the request as valid. I am not sure there is a workaround that would make the Apache server force-use of its client cert, or whether we can think of a way around this before it's fixed.
I am assuming customers would be ok for a workaround if it's not a huge change for them and if it's something that can be automated. Why is it that we only accept certs from certain CA's? So is this a middle ground between accepting certs of public trusted CA's Vs Private CA's that OCP is taking ?
I'm yet to figure out whether a workaround is possible. I think I know of a way to use "unsupportedConfigOverrides", but will investigate it first. "Why is it that we only accept certs from certain CA's?" - this is how TLS works and is dependent on the server configuration, which was wrong and is being fixed in https://github.com/openshift/oauth-server/pull/19. I even created a reproducer for the issue using the customer config ported to my cluster to make sure that this is the last fix we need. "So is this a middle ground between accepting certs of public trusted CA's Vs Private CA's that OCP is taking ?" - No. This was a misconfiguration we did because we did not realize the changes in 4.x oauth-server would cause the side-effect they did. If you want specific technical details about the issue, I tried to explain it as clear as I could in the description of https://github.com/openshift/oauth-server/pull/19
There is a possible workaround: 1. call `oc get deployment -n openshift-authentication oauth-openshift -o yaml | grep -e "mountPath:.*v4-0-config-user-idp-[0-9]*-ca" | cut -d: -f2` 2. append /ca.crt behind the path from the step 1. 3. oc edit authentication.operator - modify spec so that it looks somehow like this: ``` spec: managementState: Managed unsupportedConfigOverrides: servingInfo: clientCA: THE_PATH ``` replacing THE_PATH with what you've got in step 2, in my case that would be `/var/config/user/idp/0/configmap/v4-0-config-user-idp-0-ca/ca.crt` 4. save the object, wait for the auth pods to come back up This should make `oc login` work. As a trade-off, it will break the cluster-admin being directly able to query the /metrics endpoint of the oauth-server, but I don't think that's commonly being done, and from my testing, prometheus is still capable of scraping the endpoint, so it is still possible to get the data somewhere else.
Note that the presence of `unsupportedConfigOverrides` will prevent you from upgrading, so you'll have to remove this field before an upgrade is attempted.
Thanks @Standa. Customer may object to the inability to upgrade. If they remove the setting before the upgrade and restore if after upgrade, would that break anything?
No, even the upgrade should work fine, only the IdP's users will be unavailable for that period. The need to do this workaround will of course be removed once we manage to fix the bug, hopefully we'll get the fix into 4.1 rather sooner.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:4084