Bug 1702429

Summary: LDAP OAuth config fails with "TLS handshake error ... remote error: tls: unknown certificate"
Product: OpenShift Container Platform Reporter: Naveen Malik <nmalik>
Component: apiserver-authAssignee: Standa Laznicka <slaznick>
Status: CLOSED NOTABUG QA Contact: Chuan Yu <chuyu>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.1.0CC: aos-bugs, dminnich, nagrawal, nmalik, scuppett, slaznick
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-29 17:02:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ca.crt none

Description Naveen Malik 2019-04-23 18:34:55 UTC
Description of problem:

When LDAP OAuth is configured against Red Hat external LDAP (ext-ldap.corp.redhat.com) login fails.


Version-Release number of selected component (if applicable):
OCP 4.0.0-0.11 (upgraded from 4.0.0-0.9)


How reproducible:

Configure LDAP OAuth on OCP 4.0.0-0.11 cluster.


Steps to Reproduce:
1. Create 4.0.0-0.9 cluster
2. Upgrade to 4.0.0-0.11
3. Create secret for ldap bindPassword

oc create secret generic ldap-secret --from-literal=bindPassword=REDACTED -n openshift-config

4. Configure ca configmap

oc create configmap ldap-ca-configmap --from-file=ca.crt -n openshift-config

5. Configure OAuth

cat << EOF > oauth.yaml
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: ldap-aos-sre
    challenge: true
    login: true
    mappingMethod: claim
    type: LDAP
    ldap:
      attributes:
        id:
        - memberUid
      bindDN: "uid=<SA UID>,ou=serviceaccounts,dc=redhat,dc=com"
      bindPassword:
        name: ldap-secret
      ca:
        name: ldap-ca-configmap
      insecure: false
      url: "ldaps://ext-ldap.corp.redhat.com/cn=aos-sre,ou=Groups,dc=redhat,dc=com?memberUid?sub"
  tokenConfig:
    accessTokenMaxAgeSeconds: 86400
EOF

oc apply -f oauth.yaml

6. Wait for openshift-authentication pods to provision

7. Login to console with kerberos / pin+token

Actual results:

Login Web UI:  User sees error "Invalid login or password. Please try again."

Authentication pods:  "See http: TLS handshake error from <IP ADDR>:<PORT>: remote error: tls: unknown certificate"


Expected results:

Login is successful.

Additional info:



Log from pod that got the failure:

$ oc logs openshift-authentication-686bfb47b5-pr2fg -n openshift-authentication
Command "openshift-osinserver" is deprecated, will be removed in 4.0
I0423 12:50:05.234924 1 serving.go:195] [0] "/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt" serving certificate: "openshift-authentication.openshift-authentication.svc" [serving] validServingFor=[openshift-authentication.openshift-authentication.svc,openshift-authentication.openshift-authentication.svc.cluster.local] issuer="openshift-service-serving-signer@1555965094" (2019-04-22 20:40:01 +0000 UTC to 2021-04-21 20:40:02 +0000 UTC (now=2019-04-23 12:50:05.234894094 +0000 UTC))
I0423 12:50:05.235060 1 serving.go:195] [1] "/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt" serving certificate: "openshift-service-serving-signer@1555965094" [] issuer="<self>" (2019-04-22 20:31:33 +0000 UTC to 2020-04-21 20:31:34 +0000 UTC (now=2019-04-23 12:50:05.235044729 +0000 UTC))
I0423 12:50:05.235082 1 secure_serving.go:125] Serving securely on 0.0.0.0:6443
I0423 12:50:05.235181 1 serving.go:77] Starting DynamicLoader
I0423 12:53:32.207067 1 log.go:172] http: TLS handshake error from 10.131.0.12:54024: remote error: tls: unknown certificate

Comment 2 Erica von Buelow 2019-04-23 19:14:14 UTC
Can you attach the ca.crt you're using?

Comment 3 Naveen Malik 2019-04-23 19:28:17 UTC
Created attachment 1557859 [details]
ca.crt

Comment 4 Standa Laznicka 2019-04-24 12:18:21 UTC
Tried to reproduce today, but I get the expected `AuthenticationError: LDAP Result Code 49 "Invalid Credentials"` in the logs - since I do not know the proper setup for ext-ldap.corp.redhat.com. Even though there is

"I0424 12:09:37.565808       1 log.go:172] http: TLS handshake error from 10.131.0.13:40228: remote error: tls: bad certificate"

right after start of the dynamic loader which would be worth investigating, the TLS connection to the actual LDAP server seems fine.