It seems the bug here is that 389-ds should update the cipher suite config on upgrade, because even after working around the fortezza bug, I hit another fatal error later which also seems to be fallout from 47838: pki-server fails to start because it and 389-ds now have no ciphers in common. /var/log/dirsrv/slapd-DOMAIN1-LOCAL/access shows this when pki-server tries to connect to it:

conn-8 op=-1 fd=69 closed - Cannot communicate securely with peer: no common encryption algorithm(s).

if I edit the LDAP config by hand and throw out the nsSSL3Ciphers setting entirely (which, in 1.3.3.0, causes 389-ds to use a default list of ciphers provided by NSS), everything works and my server finally starts up.

it seems the new checks and logic for ciphers in 1.3.3.0 are just too strict for the cipher suite that was configured by 1.3.2.22, so the upgrade process really needs to adjust the suite somehow, or else upgraded servers are very likely to break.

1. Reproducing bug.
=====================
[root@dhcp201-109 export]# rpm -qa | grep 389
389-ds-console-doc-1.2.7-4.fc21.noarch
389-ds-1.2.2-6.fc21.noarch
389-ds-console-1.2.7-4.fc21.noarch
389-ds-base-libs-1.3.3.0-1.fc21.x86_64
389-admin-console-1.1.8-7.fc21.noarch
389-ds-base-devel-1.3.3.0-1.fc21.x86_64
389-adminutil-1.1.19-4.fc21.x86_64
389-console-1.1.7-7.fc21.noarch
389-ds-base-1.3.3.0-1.fc21.x86_64
389-admin-console-doc-1.1.8-7.fc21.noarch
389-adminutil-devel-1.1.19-4.fc21.x86_64
389-admin-1.1.35-2.fc21.2.x86_64
389-dsgw-1.1.11-4.fc21.x86_64

CIPHERS="-rsa_null_md5,+rsa_fips_3des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_rc4_128_md5,+rsa_des_sha,+rsa_rc2_40_md5,+rsa_rc4_40_md5,+fortezza"


[root@dhcp201-109 ~]# tail -f /var/log/dirsrv/slapd-dhcp201-109/errors
[11/Sep/2014:06:15:16 -0400] - The change of nsslapd-securePort will not take effect until the server is restarted
[11/Sep/2014:06:15:17 -0400] - slapd shutting down - signaling operation threads - op stack size 2 max work q size 1 max work q stack size 1
[11/Sep/2014:06:15:17 -0400] - slapd shutting down - waiting for 1 thread to terminate
[11/Sep/2014:06:15:17 -0400] - slapd shutting down - closing down internal subsystems and plugins
[11/Sep/2014:06:15:17 -0400] - Waiting for 4 database threads to stop
[11/Sep/2014:06:15:18 -0400] - All database threads now stopped
[11/Sep/2014:06:15:18 -0400] - slapd shutting down - freed 1 work q stack objects - freed 2 op stack objects
[11/Sep/2014:06:15:18 -0400] - slapd stopped.
[11/Sep/2014:06:15:19 -0400] - SSL alert: Cipher suite fortezza is not available in NSS 3.17
[11/Sep/2014:06:15:19 -0400] - SSL alert: Security Initialization: Failed to set SSL cipher preference information: unknown cipher fortezza (Netscape Portable Runtime error 0 - no error)
[11/Sep/2014:06:15:19 -0400] - ERROR: SSL Initialization Failed.

And it killed the DS instance

=======================================================
2. Testing new fix

[root@dhcp201-109 export]# rpm -qa | grep 389
389-ds-base-1.3.3.2-a1.fc21.x86_64
389-ds-base-libs-1.3.3.2-a1.fc21.x86_64
389-adminutil-1.1.19-4.fc21.x86_64
389-console-1.1.7-7.fc21.noarch
389-adminutil-devel-1.1.19-4.fc21.x86_64

[root@dhcp201-109 export]# tail -f /var/log/dirsrv/slapd-dhcp201-109/errors
[11/Sep/2014:06:22:01 -0400] - SSL alert: Cipher rsa_rc4_128_md5 is weak. It is enabled since allowWeakCipher is "on" (default setting for the backward compatibility). We strongly recommend to set it to "off"
[11/Sep/2014:06:22:01 -0400] - SSL alert: Security Initialization: Failed to set SSL cipher preference information: unknown cipher rsa_des_sha (Netscape Portable Runtime error 0 - no error)
==============================================================================================
Why it is giving this error? when rsa_des_sha is valid cipher, Is it becasue it is not in NSS3?
===================================================================================================
[11/Sep/2014:06:22:01 -0400] - ERROR: SSL Initialization Failed.  Disabling SSL.
[11/Sep/2014:06:22:01 -0400] - 389-Directory/1.3.3.2.a1 B2014.254.627 starting up
[11/Sep/2014:06:22:01 -0400] - I'm resizing my cache now...cache was 6400000 and is now 5120000
[11/Sep/2014:06:22:01 -0400] attrcrypt - No symmetric key found for cipher AES in backend userRoot, attempting to create one...
[11/Sep/2014:06:22:01 -0400] attrcrypt - Key for cipher AES successfully generated and stored
[11/Sep/2014:06:22:01 -0400] attrcrypt - No symmetric key found for cipher 3DES in backend userRoot, attempting to create one...
[11/Sep/2014:06:22:01 -0400] attrcrypt - Key for cipher 3DES successfully generated and stored
[11/Sep/2014:06:22:01 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests

[root@dhcp201-109 export]# ps -aef | grep slapd
nobody    5049     1  0 06:22 ?        00:00:00 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-dhcp201-109 -i /var/run/dirsrv/slapd-dhcp201-109.pid -w /var/run/dirsrv/slapd-dhcp201-109.startpid
root      5099  4286  0 06:23 pts/1    00:00:00 grep --color=auto slapd

2.1. After removing rsa_des_sha and fortezza, SSL turned ON

nsSSL3Ciphers: -rsa_null_md5,+rsa_fips_3des_sha,+rsa_fips_des_sha,+rsa_3des_sh
 a,+rsa_rc4_128_md5,+rsa_rc2_40_md5,+rsa_rc4_40_md5
 
Error Logs
=========
[11/Sep/2014:06:53:35 -0400] - SSL alert: Cipher rsa_rc2_40_md5 is weak. It is enabled since allowWeakCipher is "on" (default setting for the backward compatibility). We strongly recommend to set it to "off"
[11/Sep/2014:06:53:35 -0400] - SSL alert: Cipher rsa_rc4_40_md5 is weak. It is enabled since allowWeakCipher is "on" (default setting for the backward compatibility). We strongly recommend to set it to "off"
[11/Sep/2014:06:53:35 -0400] - SSL alert: Configured NSS Ciphers
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_WITH_3DES_EDE_CBC_SHA: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_WITH_RC4_128_MD5: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	SSL_RSA_FIPS_WITH_DES_CBC_SHA: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_EXPORT_WITH_RC4_40_MD5: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] - SSL alert: 	TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5: enabled, (WEAK CIPHER)
[11/Sep/2014:06:53:35 -0400] SSL Initialization - SSL version range: min: SSL3, max: TLS1.2
[11/Sep/2014:06:53:35 -0400] - 389-Directory/1.3.3.2.a1 B2014.254.627 starting up
[11/Sep/2014:06:53:35 -0400] - I'm resizing my cache now...cache was 2097152 and is now 1677721
[11/Sep/2014:06:53:35 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[11/Sep/2014:06:53:35 -0400] - Listening on All Interfaces port 636 for LDAPS requests


----->Setting allowWeakCipher is "OFF"<----------------------

[root@dhcp201-109 export]# ldapmodify -x -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 << EOF
dn: cn=encryption,cn=config
changetype: modify
replace: allowWeakCipher
allowWeakCipher: off
EOF

modifying entry "cn=encryption,cn=config"
=============================================================================
It is taking an invalid value - Please fix this - not super critical though!
=============================================================================
[root@dhcp201-109 export]# systemctl dirsrv restart
Unknown operation 'dirsrv'.
[root@dhcp201-109 export]# systemctl restart dirsrv.service
Failed to restart dirsrv.service: Unit dirsrv.service failed to load: No such file or directory.
[root@dhcp201-109 export]# systemctl restart dirsrv@dhcp201-109
[root@dhcp201-109 export]# ldapmodify -x -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 << EOF
dn: cn=encryption,cn=config
changetype: modify
replace: allowWeakCipher
allowWeakCipher: poor
> EOF
modifying entry "cn=encryption,cn=config"


error messages on restart while allowWeakCipher:OFF with user settings for cipher
==================================================================================
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - signaling operation threads - op stack size 2 max work q size 1 max work q stack size 1
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - waiting for 1 thread to terminate
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - closing down internal subsystems and plugins
[11/Sep/2014:06:56:51 -0400] - Waiting for 4 database threads to stop
[11/Sep/2014:06:56:51 -0400] - All database threads now stopped
[11/Sep/2014:06:56:51 -0400] - slapd shutting down - freed 1 work q stack objects - freed 2 op stack objects
[11/Sep/2014:06:56:51 -0400] - slapd stopped.
[11/Sep/2014:06:56:52 -0400] - SSL alert: Configured NSS Ciphers
[11/Sep/2014:06:56:52 -0400] SSL Initialization - SSL version range: min: SSL3, max: TLS1.2
[11/Sep/2014:06:56:52 -0400] - 389-Directory/1.3.3.2.a1 B2014.254.627 starting up
[11/Sep/2014:06:56:52 -0400] - I'm resizing my cache now...cache was 1677721 and is now 1342176
[11/Sep/2014:06:56:52 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[11/Sep/2014:06:56:52 -0400] - Listening on All Interfaces port 636 for LDAPS requests

Question 1:: What is the effect of allowWeakCipher:OFF , description of patch says it rejects weak ciphers, but how and what is the impact? It just stopped logging messages for weak ciphers.
My instance started normally with SSL enabled and in secure mode with weak ciphers while I have allowWeakCipher:OFF.

Question 2 :: I would like to know what were the default values set by FreeIPA for nsslapd CIPHERS(Adam, can you please share), if these have some unsupported values like fortezza then that would be an issue, that should be a bug for IPA to not to set such unsupported values by default in 389.

"I would like to know what were the default values set by FreeIPA for nsslapd CIPHERS(Adam, can you please share)"

the setting I had from an install of FreeIPA 4.0.2 with 389-ds 1.3.2.22 was:

nsssl3ciphers: -rsa_null_md5,+rsa_rc4_128_md5,+rsa_rc4_40_md5,+rsa_rc2_40_md5,
 +rsa_des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_fips_3des_sha,+fortezza,+fo
 rtezza_rc4_128_sha,+fortezza_null,+tls_rsa_export1024_with_rc4_56_sha,+tls_rs
 a_export1024_with_des_cbc_sha

"if these have some unsupported values like fortezza then that would be an issue, that should be a bug for IPA to not to set such unsupported values by default in 389."

FreeIPA is working on it for 4.0.3:

https://www.redhat.com/archives/freeipa-devel/2014-September/msg00235.html

Upstream ticket:
https://fedorahosted.org/389/ticket/47908

Hi Adam, Hi Ami,

Could you please take a look at this upstream ticket?
   https://fedorahosted.org/389/ticket/47908

I tested this cipher list from Comment 3 and am attaching the directory server log next.
nsssl3ciphers: -rsa_null_md5,+rsa_rc4_128_md5,+rsa_rc4_40_md5,+rsa_rc2_40_md5,
 +rsa_des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_fips_3des_sha,+fortezza,+fo
 rtezza_rc4_128_sha,+fortezza_null,+tls_rsa_export1024_with_rc4_56_sha,+tls_rs
 a_export1024_with_des_cbc_sha

As you see in the error log, the specified ciphers are all weak or deprecated.

By default, allowWeakCipher is "on" for the user specified cipher list.  So, you could start the server with the cipher list listening on the secure port.  But if explicitly set "allowWeakCipher: off", then all the above ciphers are skipped and the server does not listen on the secure port.

Created attachment 940614 [details]
error log

thanks, I'll try and get to taking a look at the current state of this after the various changes soon - lots of projects on ATM :(