Bug 1626988 - LDAPS can't connect to AD - connection reset by peer
Summary: LDAPS can't connect to AD - connection reset by peer
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.11.z
Assignee: Mo
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-10 08:59 UTC by Vladislav Walek
Modified: 2022-03-13 15:32 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-10 09:03:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0024 0 None None None 2019-01-10 09:04:05 UTC

Description Vladislav Walek 2018-09-10 08:59:44 UTC
Description of problem:

customer is trying to connect to the AD. if using ldaps it fails on the connection reset by peer.
We collected the tcpdump and see that the AD server is resetting the connection when the TLS handshake starts.

- we know (from tcpdump) that during the TLS handshake between OCP and AD - the AD immediately reset the connection
- the ldap_ca.pem certificate is correct - no issue there on OCP
- from the logs you have provided the error 40 in the AD logs means in TLS -  handshake_failure
- from the logs we also see error that the Ciphers supported by client are not supported by AD


Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.7.54
Microsoft Windows Server 2012 R2 Standard Version 6.3.9600 Build 9600

How reproducible:
not reproducible

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Windows 8.1/2012r2

Cipher suite string	Allowed by SCH_USE_STRONG_CRYPTO	TLS/SSL Protocol Versions
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256	Yes	TLS 1.2
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384	Yes	TLS 1.2
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256	Yes	TLS 1.2
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384	Yes	TLS 1.2
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384	Yes	TLS 1.2
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256	Yes	TLS 1.2
TLS_DHE_RSA_WITH_AES_256_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_DHE_RSA_WITH_AES_128_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_RSA_WITH_AES_256_GCM_SHA384	Yes	TLS 1.2
TLS_RSA_WITH_AES_128_GCM_SHA256	Yes	TLS 1.2
TLS_RSA_WITH_AES_256_CBC_SHA256	Yes	TLS 1.2
TLS_RSA_WITH_AES_128_CBC_SHA256	Yes	TLS 1.2
TLS_RSA_WITH_AES_256_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_RSA_WITH_AES_128_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384	Yes	TLS 1.2
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256	Yes	TLS 1.2
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384	Yes	TLS 1.2
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384	Yes	TLS 1.2
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256	Yes	TLS 1.2
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384	Yes	TLS 1.2
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256	Yes	TLS 1.2
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256	Yes	TLS 1.2
TLS_DHE_DSS_WITH_AES_256_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_DHE_DSS_WITH_AES_128_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_RSA_WITH_3DES_EDE_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA	Yes	TLS 1.2, TLS 1.1, TLS 1.0, SSL 3.0
TLS_RSA_WITH_RC4_128_SHA	No	TLS 1.2, TLS 1.1, TLS 1.0, SSL 3.0
TLS_RSA_WITH_RC4_128_MD5	No	TLS 1.2, TLS 1.1, TLS 1.0, SSL 3.0
TLS_RSA_WITH_NULL_SHA256 
Only used when application explicitly requests.	Yes	TLS 1.2
TLS_RSA_WITH_NULL_SHA 
Only used when application explicitly requests.	Yes	TLS 1.2, TLS 1.1, TLS 1.0, SSL 3.0

Comment 2 Simo Sorce 2018-09-11 16:53:44 UTC
Sounds like there are incompatibility issues between Microsoft AD servers using TLS1.2 with certs using SHA512 as their hash.

Here there is a discussion that hints in that direction:
https://social.technet.microsoft.com/Forums/en-US/b6ffa278-4a04-4609-ac35-8390f5ba9cb6/ldap-over-ssl-on-windows-2012r2-server-dcs-tls-12-not-working?forum=winserversecurity

Two possible workarounds suggested are:
- disable TLS1.2 in the Widnows DC
- change certificates to use supported hashes

I do not have reason to believe anything is amiss on the openshift side, sounds like a bug or an unfortunate set of defaults in Windows server certificate creation.

Comment 3 Vladislav Walek 2018-09-12 14:46:35 UTC
I have reply from customer:

The 2 options are "workarounds" on AD side.

The actual reason seems that the client does not support SHA512 sigantures.
How can openshift be changed to support SHA512?

Comment 4 Simo Sorce 2018-09-12 15:01:24 UTC
No the TLS1.2 protocol standard does not support SHA512 signatures, we are not going to deviate from protocol standards, as those are implemented directly by Go crypto libraries and not Openshift itself.

Sorry but this is not we can solve on the Openshift side.

Comment 5 Mo 2018-09-12 16:00:16 UTC
I do not know if SSSD/Apache/etc support their AD configuration, but if they do, they could switch to the remote basic auth IDP [1] and have Apache handle the TLS to AD.

[1] https://docs.okd.io/latest/install_config/sssd_for_ldap_failover.html

Comment 6 Simo Sorce 2018-09-12 16:11:46 UTC
(In reply to Mo from comment #5)
> I do not know if SSSD/Apache/etc support their AD configuration, but if they
> do, they could switch to the remote basic auth IDP [1] and have Apache
> handle the TLS to AD.
> 
> [1] https://docs.okd.io/latest/install_config/sssd_for_ldap_failover.html

True, an SSSD joined to AD doesn't even care for LDAPS,a s it uses plain LDAP + GSSAPI for encryption, so it would work w/o certs... and avoid the whole issue.

Comment 8 Simo Sorce 2018-09-17 12:42:00 UTC
SHA512 is supported for signing and verifying certificates, but if you read appendinx A.6 and C you will see there are no ciphers with SHA512 in them. It's ether SHA or SHA256.

Microsoft made a Mistake and forced Windows server to use RSA/SHA512 and ECDSA/SHA512 in TLS1.2, when their certificates use SHA512 for signing.

Comment 9 Simo Sorce 2018-09-17 14:44:20 UTC
Sorry for the above I had misunderstood the issue.
After re-reading the case I figured out that you are almost certainly affected by a missing signatureAlgorithm support issue in golang crypto.

This is the Issue:
https://github.com/golang/go/issues/22422

And this is the commit that fixes the issue:
https://github.com/golang/go/commit/96cd66b266a24d8ed1f66dfa10ddb86d88b50fca

It is tagged to lang in golang 1.11 so we can properly fix this issue only once we pick up go lang 1.11 ourselves in a future version or get this fix backported.

Till then the workarounds are the best way to go.

Comment 10 Simo Sorce 2018-09-17 14:45:10 UTC
I am goign to move this to the master team as they are the ones currently dealing with rebases and decisions about pulling in new go lang versions.

Comment 11 Simo Sorce 2018-09-17 14:47:12 UTC
So just one more bit of context, the reason this fails with Microsoft and not other servers is that Microsoft implemented TLS1.2 in a stricter way than others. But their implementation is technically correct.

Comment 12 Mo 2018-09-17 14:55:39 UTC
I believe the fix was included in go1.10:

https://github.com/golang/go/blob/go1.10/src/crypto/tls/common.go#L145-L146
https://github.com/golang/go/blob/go1.10/src/crypto/tls/prf.go#L189-L190

so it should be fixed by upgrading to v3.11

Comment 29 Mo 2018-10-25 14:10:16 UTC
Per gdb, 3.11 uses go1.10.2

$ ls
atomic-openshift-3.11.16-1.git.0.b48b8f8.el7.x86_64.rpm  openshift

$ gdb openshift
GNU gdb (GDB) Fedora 8.1.1-3.fc28
...
(gdb) p 'runtime.buildVersion'
$1 = 0x5197fe3 "go1.10.2"

Comment 33 Eric Rich 2018-11-12 17:48:18 UTC
I wonder if this is not a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1643119

Comment 38 Chuan Yu 2018-12-07 04:31:26 UTC
The issue fixed in v3.11.43

Comment 40 errata-xmlrpc 2019-01-10 09:03:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0024


Note You need to log in before you can comment on or make changes to this bug.