The same problem with RHEL-6.2:

openldap-2.4.23-20.el6
nss-3.12.10-14.el6

+++ This bug was initially created as a clone of Bug #748394 +++

Description of problem:

The problem can be reproduced with OpenLDAP. It was reported in OpenLDAP upstream issue tracker (http://www.openldap.org/lists/openldap-bugs/201110/msg00021.html). In fact this is a problem of Mozilla NSS. Fedora is affected.

Version-Release number of selected component (if applicable):

openldap-2.4.26-4.fc15.x86_64
nss-3.12.10-6.fc15.x86_64

Steps to Reproduce:
1. Create CA certificate with DSA signature
2. Create server certificate with RSA signature
3. Sign the certificate by generated CA certificate
4. setup slapd to use these certificates (TLSCertificateFile, TLSCertificateKeyFile, do not set TLSCACertificateFile) and start the server
5. LDAPTLS_CACERT=/your/cacert.pem ldapsearch -x -ZZ -d1 -H ldap://your-server

Actual results:

TLS: loaded CA certificate file /tmp/CA/CA/cacert.pem.
TLS: certificate [CN=alioth.usersys.redhat.com,O=jvcelak Red Hat Test,L=Brno,C=CZ] is not valid - error -8182:Unknown code ___f 10.
TLS: error: connect - force handshake failure: errno 21 - moznss error -8182
TLS: can't connect: TLS error -8182:Unknown code ___f 10.
ldap_err2string
ldap_start_tls: Connect error (-11)
        additional info: TLS error -8182:Unknown code ___f 10

The server side result:

connection_get(12): got connid=1002
connection_read(12): checking for input on id=1002
connection_get(12): got connid=1002
connection_read(12): checking for input on id=1002
TLS: error: accept - force handshake failure: errno 11 - moznss error -12271
TLS: can't accept: TLS error -12271:Unknown code ___P 17.
connection_read(12): TLS accept failure error=-1 id=1002, closing

Expected results:

The connection will succeed.

Additional info:

The failure comes from DSAU_ConvertSignedToFixedUnsigned ->  line 120.

116     if (zCount <= 0) {
117         /* Source is longer than destination.  Check for leading zeros. */
118         while (zCount++ < 0) {
119             if (*pSrc++ != 0)
120                 goto loser;
121         }
122     }

I will attach full backtrace. And configuration file with certificates.

(Do not set TLSCACertificateFile in step 4., otherwise the server certificate validation will fail on the server side with -8182 and all following TLS requests will be refused. This configuration is easier for debugging, you do not have to restart the server with every request.)

--- Additional comment from jvcelak on 2011-10-24 12:50:16 CEST ---

Created attachment 529788 [details]
full backtrace

--- Additional comment from jvcelak on 2011-10-24 13:06:16 CEST ---

Created attachment 529827 [details]
data for reproduction

Extract the archive into /tmp.

start the server: /tmp/slapd-dsa-rsa/run-server.sh
query the server: /tmp/slapd-dsa-rsa/query-server.sh