Bug 1145846

Summary: 389-ds 1.3.3.0 does not adjust cipher suite configuration on upgrade, breaks itself and pki-server: "Cipher suite fortezza is not available in NSS 3.17" , "Cannot communicate securely with peer: no common encryption algorithm(s)."
Product: Red Hat Enterprise Linux 7 Reporter: Noriko Hosoi <nhosoi>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: high Docs Contact:
Priority: high    
Version: 7.1CC: amessina, amsharma, awilliam, edewata, extras-qa, jgalipea, jpazdziora, mreynolds, nhosoi, nkinder, rmeggins
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.3.1-4.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1139954 Environment:
Last Closed: 2015-03-05 09:36:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1139954    
Bug Blocks:    

Description Noriko Hosoi 2014-09-24 01:02:50 UTC
+++ This bug was initially created as a clone of Bug #1139954 +++

I'm testing FreeIPA in a clean test config with F21 Alpha server and client.

I installed the server with Alpha TC6 and it worked, but I ran into some other issues. I updated the server from updates-testing to see if it'd address those issues, but the updated 389-ds - 1.3.3.0-1.fc21 - seems to have a bug of its own.

When I try and start ipa.service , ns-slapd fails to start correctly:

Sep 09 20:38:40 ipa001.domain1.local ns-slapd[2454]: [09/Sep/2014:20:38:40 -0700] - SSL alert: Cipher suite fortezza is not available in NSS 3.17
Sep 09 20:38:40 ipa001.domain1.local ns-slapd[2454]: [09/Sep/2014:20:38:40 -0700] - SSL alert: Security Initialization: Failed to set SSL cipher preference information: unknown cipher fortezza (Netscape Portable Runtime error 0 - no error)
Sep 09 20:38:40 ipa001.domain1.local ns-slapd[2454]: [09/Sep/2014:20:38:40 -0700] - ERROR: SSL Initialization Failed.
Sep 09 20:39:53 ipa001.domain1.local ns-slapd[14472]: [09/Sep/2014:20:39:53 -0700] - Information: Non-Secure Port Disabled

I was able to fix this by hand-editing /etc/dirsrv/slapd-DOMAIN1-LOCAL/dse.ldif and removing the references to all fortezza suites in the nsSSL3Ciphers: lines. After that, it starts successfully:

Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert: Configured NSS Ciphers
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         TLS_RSA_WITH_3DES_EDE_CBC_SHA: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         TLS_RSA_WITH_RC4_128_MD5: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         SSL_RSA_FIPS_WITH_DES_CBC_SHA: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         TLS_RSA_WITH_DES_CBC_SHA: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         TLS_RSA_EXPORT1024_WITH_RC4_56_SHA: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         TLS_RSA_EXPORT_WITH_RC4_40_MD5: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] - SSL alert:         TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5: enabled, (WEAK CIPHER)
Sep 09 21:09:04 ipa001.domain1.local ns-slapd[1434]: [09/Sep/2014:21:09:04 -0700] SSL Initialization - SSL version range: min: TLS1.0, max: TLS1.2

It seems this check was added as part of the cipher suite hardening attempt in https://fedorahosted.org/389/ticket/47838 , but obviously it's a problem if it fails for the 'stock' cipher suite on a clean install. I don't know if the check needs to be changed, or the upgrade to 1.3.3.0 should edit the nsSSL3Ciphers: settings as part of its work?

This is a showstopper for FreeIPA, I believe, so setting high severity.

--- Additional comment from Adam Williamson (Red Hat) on 2014-09-10 02:44:06 EDT ---

It seems the bug here is that 389-ds should update the cipher suite config on upgrade, because even after working around the fortezza bug, I hit another fatal error later which also seems to be fallout from 47838: pki-server fails to start because it and 389-ds now have no ciphers in common. /var/log/dirsrv/slapd-DOMAIN1-LOCAL/access shows this when pki-server tries to connect to it:

conn-8 op=-1 fd=69 closed - Cannot communicate securely with peer: no common encryption algorithm(s).

if I edit the LDAP config by hand and throw out the nsSSL3Ciphers setting entirely (which, in 1.3.3.0, causes 389-ds to use a default list of ciphers provided by NSS), everything works and my server finally starts up.

it seems the new checks and logic for ciphers in 1.3.3.0 are just too strict for the cipher suite that was configured by 1.3.2.22, so the upgrade process really needs to adjust the suite somehow, or else upgraded servers are very likely to break.

--- Additional comment from Amita Sharma on 2014-09-11 07:07:57 EDT ---

1. Reproducing bug.
=====================
[root@dhcp201-109 export]# rpm -qa | grep 389
389-ds-console-doc-1.2.7-4.fc21.noarch
389-ds-1.2.2-6.fc21.noarch
389-ds-console-1.2.7-4.fc21.noarch
389-ds-base-libs-1.3.3.0-1.fc21.x86_64
389-admin-console-1.1.8-7.fc21.noarch
389-ds-base-devel-1.3.3.0-1.fc21.x86_64
389-adminutil-1.1.19-4.fc21.x86_64
389-console-1.1.7-7.fc21.noarch
389-ds-base-1.3.3.0-1.fc21.x86_64
389-admin-console-doc-1.1.8-7.fc21.noarch
389-adminutil-devel-1.1.19-4.fc21.x86_64
389-admin-1.1.35-2.fc21.2.x86_64
389-dsgw-1.1.11-4.fc21.x86_64

CIPHERS="-rsa_null_md5,+rsa_fips_3des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_rc4_128_md5,+rsa_des_sha,+rsa_rc2_40_md5,+rsa_rc4_40_md5,+fortezza"


[root@dhcp201-109 ~]# tail -f /var/log/dirsrv/slapd-dhcp201-109/errors
[11/Sep/2014:06:15:16 -0400] - The change of nsslapd-securePort will not take effect until the server is restarted
[11/Sep/2014:06:15:17 -0400] - slapd shutting down - signaling operation threads - op stack size 2 max work q size 1 max work q stack size 1
[11/Sep/2014:06:15:17 -0400] - slapd shutting down - waiting for 1 thread to terminate
[11/Sep/2014:06:15:17 -0400] - slapd shutting down - closing down internal subsystems and plugins
[11/Sep/2014:06:15:17 -0400] - Waiting for 4 database threads to stop
[11/Sep/2014:06:15:18 -0400] - All database threads now stopped
[11/Sep/2014:06:15:18 -0400] - slapd shutting down - freed 1 work q stack objects - freed 2 op stack objects
[11/Sep/2014:06:15:18 -0400] - slapd stopped.
[11/Sep/2014:06:15:19 -0400] - SSL alert: Cipher suite fortezza is not available in NSS 3.17
[11/Sep/2014:06:15:19 -0400] - SSL alert: Security Initialization: Failed to set SSL cipher preference information: unknown cipher fortezza (Netscape Portable Runtime error 0 - no error)
[11/Sep/2014:06:15:19 -0400] - ERROR: SSL Initialization Failed.

And it killed the DS instance

=======================================================
2. Testing new fix

[root@dhcp201-109 export]# rpm -qa | grep 389
389-ds-base-1.3.3.2-a1.fc21.x86_64
389-ds-base-libs-1.3.3.2-a1.fc21.x86_64
389-adminutil-1.1.19-4.fc21.x86_64
389-console-1.1.7-7.fc21.noarch
389-adminutil-devel-1.1.19-4.fc21.x86_64

[root@dhcp201-109 export]# tail -f /var/log/dirsrv/slapd-dhcp201-109/errors
[11/Sep/2014:06:22:01 -0400] - SSL alert: Cipher rsa_rc4_128_md5 is weak. It is enabled since allowWeakCipher is "on" (default setting for the backward compatibility). We strongly recommend to set it to "off"
[11/Sep/2014:06:22:01 -0400] - SSL alert: Security Initialization: Failed to set SSL cipher preference information: unknown cipher rsa_des_sha (Netscape Portable Runtime error 0 - no error)
==============================================================================================
Why it is giving this error? when rsa_des_sha is valid cipher, Is it becasue it is not in NSS3?
===================================================================================================
[11/Sep/2014:06:22:01 -0400] - ERROR: SSL Initialization Failed.  Disabling SSL.
[11/Sep/2014:06:22:01 -0400] - 389-Directory/1.3.3.2.a1 B2014.254.627 starting up
[11/Sep/2014:06:22:01 -0400] - I'm resizing my cache now...cache was 6400000 and is now 5120000
[11/Sep/2014:06:22:01 -0400] attrcrypt - No symmetric key found for cipher AES in backend userRoot, attempting to create one...
[11/Sep/2014:06:22:01 -0400] attrcrypt - Key for cipher AES successfully generated and stored
[11/Sep/2014:06:22:01 -0400] attrcrypt - No symmetric key found for cipher 3DES in backend userRoot, attempting to create one...
[11/Sep/2014:06:22:01 -0400] attrcrypt - Key for cipher 3DES successfully generated and stored
[11/Sep/2014:06:22:01 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests

[root@dhcp201-109 export]# ps -aef | grep slapd
nobody    5049     1  0 06:22 ?        00:00:00 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-dhcp201-109 -i /var/run/dirsrv/slapd-dhcp201-109.pid -w /var/run/dirsrv/slapd-dhcp201-109.startpid
root      5099  4286  0 06:23 pts/1    00:00:00 grep --color=auto slapd

2.1. After removing rsa_des_sha and fortezza, SSL turned ON

nsSSL3Ciphers: -rsa_null_md5,+rsa_fips_3des_sha,+rsa_fips_des_sha,+rsa_3des_sh
 a,+rsa_rc4_128_md5,+rsa_rc2_40_md5,+rsa_rc4_40_md5
 
Error Logs
=========
[11/Sep/2014:06:53:35 -0400] - SSL alert: Cipher rsa_rc2_40_md5 is weak. It is enabled since allowWeakCipher is "on" (default setting for the backward compatibility). We strongly recommend to set it to "off"
[11/Sep/2014:06:53:35 -0400] - SSL alert: Cipher rsa_rc4_40_md5 is weak. It is enabled since allowWeakCipher is "on" (default setting for the backward compatibility). We strongly recommend to set it to "off"
[11/Sep/2014:06:53:35 -0400] - SSL alert: Configured NSS Ciphers
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_WITH_3DES_EDE_CBC_SHA: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_WITH_RC4_128_MD5: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	SSL_RSA_FIPS_WITH_DES_CBC_SHA: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_EXPORT_WITH_RC4_40_MD5: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] SSL Initialization - SSL version range: min: SSL3, max: TLS1.2
[11/Sep/2014:06:53:35 -0400] - 389-Directory/1.3.3.2.a1 B2014.254.627 starting up
[11/Sep/2014:06:53:35 -0400] - I'm resizing my cache now...cache was 2097152 and is now 1677721
[11/Sep/2014:06:53:35 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[11/Sep/2014:06:53:35 -0400] - Listening on All Interfaces port 636 for LDAPS requests


----->Setting allowWeakCipher is "OFF"<----------------------

[root@dhcp201-109 export]# ldapmodify -x -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 << EOF
dn: cn=encryption,cn=config
changetype: modify
replace: allowWeakCipher
allowWeakCipher: off
EOF

modifying entry "cn=encryption,cn=config"
=============================================================================
It is taking an invalid value - Please fix this - not super critical though!
=============================================================================
[root@dhcp201-109 export]# systemctl dirsrv restart
Unknown operation 'dirsrv'.
[root@dhcp201-109 export]# systemctl restart dirsrv.service
Failed to restart dirsrv.service: Unit dirsrv.service failed to load: No such file or directory.
[root@dhcp201-109 export]# systemctl restart dirsrv@dhcp201-109
[root@dhcp201-109 export]# ldapmodify -x -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 << EOF
dn: cn=encryption,cn=config
changetype: modify
replace: allowWeakCipher
allowWeakCipher: poor
> EOF
modifying entry "cn=encryption,cn=config"


error messages on restart while allowWeakCipher:OFF with user settings for cipher
==================================================================================
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - signaling operation threads - op stack size 2 max work q size 1 max work q stack size 1
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - waiting for 1 thread to terminate
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - closing down internal subsystems and plugins
[11/Sep/2014:06:56:51 -0400] - Waiting for 4 database threads to stop
[11/Sep/2014:06:56:51 -0400] - All database threads now stopped
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - freed 1 work q stack objects - freed 2 op stack objects
[11/Sep/2014:06:56:51 -0400] - slapd stopped.
[11/Sep/2014:06:56:52 -0400] - SSL alert: Configured NSS Ciphers
[11/Sep/2014:06:56:52 -0400] SSL Initialization - SSL version range: min: SSL3, max: TLS1.2
[11/Sep/2014:06:56:52 -0400] - 389-Directory/1.3.3.2.a1 B2014.254.627 starting up
[11/Sep/2014:06:56:52 -0400] - I'm resizing my cache now...cache was 1677721 and is now 1342176
[11/Sep/2014:06:56:52 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[11/Sep/2014:06:56:52 -0400] - Listening on All Interfaces port 636 for LDAPS requests

Question 1:: What is the effect of allowWeakCipher:OFF , description of patch says it rejects weak ciphers, but how and what is the impact? It just stopped logging messages for weak ciphers.
My instance started normally with SSL enabled and in secure mode with weak ciphers while I have allowWeakCipher:OFF.

Question 2 :: I would like to know what were the default values set by FreeIPA for nsslapd CIPHERS(Adam, can you please share), if these have some unsupported values like fortezza then that would be an issue, that should be a bug for IPA to not to set such unsupported values by default in 389.

--- Additional comment from Adam Williamson (Red Hat) on 2014-09-11 12:48:45 EDT ---

"I would like to know what were the default values set by FreeIPA for nsslapd CIPHERS(Adam, can you please share)"

the setting I had from an install of FreeIPA 4.0.2 with 389-ds 1.3.2.22 was:

nsssl3ciphers: -rsa_null_md5,+rsa_rc4_128_md5,+rsa_rc4_40_md5,+rsa_rc2_40_md5,
 +rsa_des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_fips_3des_sha,+fortezza,+fo
 rtezza_rc4_128_sha,+fortezza_null,+tls_rsa_export1024_with_rc4_56_sha,+tls_rs
 a_export1024_with_des_cbc_sha

"if these have some unsupported values like fortezza then that would be an issue, that should be a bug for IPA to not to set such unsupported values by default in 389."

FreeIPA is working on it for 4.0.3:

https://www.redhat.com/archives/freeipa-devel/2014-September/msg00235.html

--- Additional comment from Noriko Hosoi on 2014-09-23 17:34:00 EDT ---

Upstream ticket:
https://fedorahosted.org/389/ticket/47908

--- Additional comment from Noriko Hosoi on 2014-09-23 18:16:41 EDT ---

Hi Adam, Hi Ami,

Could you please take a look at this upstream ticket?
   https://fedorahosted.org/389/ticket/47908

I tested this cipher list from Comment 3 and am attaching the directory server log next.
nsssl3ciphers: -rsa_null_md5,+rsa_rc4_128_md5,+rsa_rc4_40_md5,+rsa_rc2_40_md5,
 +rsa_des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_fips_3des_sha,+fortezza,+fo
 rtezza_rc4_128_sha,+fortezza_null,+tls_rsa_export1024_with_rc4_56_sha,+tls_rs
 a_export1024_with_des_cbc_sha

As you see in the error log, the specified ciphers are all weak or deprecated.

By default, allowWeakCipher is "on" for the user specified cipher list.  So, you could start the server with the cipher list listening on the secure port.  But if explicitly set "allowWeakCipher: off", then all the above ciphers are skipped and the server does not listen on the secure port.

--- Additional comment from Noriko Hosoi on 2014-09-23 18:17:35 EDT ---

Comment 1 Sankar Ramalingam 2014-09-24 18:50:02 UTC
Its already automated in SSL startup tests. If SSL startup tests passes, we can mark the bug as Verified.

Comment 3 Sankar Ramalingam 2014-10-16 10:14:31 UTC
SSL startup tests passed as per the acceptance test reports for 389-ds-base-1.3.3.1-6.

nsslapd-listen-backlog-size: 128
nsslapd-dynamic-plugins: off
nsslapd-cn-uses-dn-syntax-in-dns: off
nsslapd-malloc-mxfast: -10
nsslapd-malloc-trim-threshold: -10
nsslapd-malloc-mmap-threshold: -10
nsslapd-ignore-time-skew: off
passwordStorageScheme: SSHA
nsslapd-rootpwstoragescheme: SSHA
nsslapd-errorlog-list:
nsslapd-auditlog-list:
nsslapd-ssl-check-hostname: on
nsslapd-hash-filters: off
TestCase [startup] result-> [PASS]

Hence, marking the bug as Verified.

Comment 5 errata-xmlrpc 2015-03-05 09:36:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0416.html