Bug 748401 - SEC_ERROR_BAD_SIGNATURE with a certificate trusted by OpenSSL
Summary: SEC_ERROR_BAD_SIGNATURE with a certificate trusted by OpenSSL
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nss
Version: 7.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Elio Maldonado Batiz
QA Contact: BaseOS QE Security Team
Depends On: 748394
TreeView+ depends on / blocked
Reported: 2011-10-24 11:08 UTC by Jan Vcelak
Modified: 2016-01-22 16:38 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 748394
Last Closed: 2016-01-22 16:38:34 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Jan Vcelak 2011-10-24 11:08:35 UTC
The same problem with RHEL-6.2:


+++ This bug was initially created as a clone of Bug #748394 +++

Description of problem:

The problem can be reproduced with OpenLDAP. It was reported in OpenLDAP upstream issue tracker (http://www.openldap.org/lists/openldap-bugs/201110/msg00021.html). In fact this is a problem of Mozilla NSS. Fedora is affected.

Version-Release number of selected component (if applicable):


Steps to Reproduce:
1. Create CA certificate with DSA signature
2. Create server certificate with RSA signature
3. Sign the certificate by generated CA certificate
4. setup slapd to use these certificates (TLSCertificateFile, TLSCertificateKeyFile, do not set TLSCACertificateFile) and start the server
5. LDAPTLS_CACERT=/your/cacert.pem ldapsearch -x -ZZ -d1 -H ldap://your-server

Actual results:

TLS: loaded CA certificate file /tmp/CA/CA/cacert.pem.
TLS: certificate [CN=alioth.usersys.redhat.com,O=jvcelak Red Hat Test,L=Brno,C=CZ] is not valid - error -8182:Unknown code ___f 10.
TLS: error: connect - force handshake failure: errno 21 - moznss error -8182
TLS: can't connect: TLS error -8182:Unknown code ___f 10.
ldap_start_tls: Connect error (-11)
        additional info: TLS error -8182:Unknown code ___f 10

The server side result:

connection_get(12): got connid=1002
connection_read(12): checking for input on id=1002
connection_get(12): got connid=1002
connection_read(12): checking for input on id=1002
TLS: error: accept - force handshake failure: errno 11 - moznss error -12271
TLS: can't accept: TLS error -12271:Unknown code ___P 17.
connection_read(12): TLS accept failure error=-1 id=1002, closing

Expected results:

The connection will succeed.

Additional info:

The failure comes from DSAU_ConvertSignedToFixedUnsigned ->  line 120.

116     if (zCount <= 0) {
117         /* Source is longer than destination.  Check for leading zeros. */
118         while (zCount++ < 0) {
119             if (*pSrc++ != 0)
120                 goto loser;
121         }
122     }

I will attach full backtrace. And configuration file with certificates.

(Do not set TLSCACertificateFile in step 4., otherwise the server certificate validation will fail on the server side with -8182 and all following TLS requests will be refused. This configuration is easier for debugging, you do not have to restart the server with every request.)

--- Additional comment from jvcelak@redhat.com on 2011-10-24 12:50:16 CEST ---

Created attachment 529788 [details]
full backtrace

--- Additional comment from jvcelak@redhat.com on 2011-10-24 13:06:16 CEST ---

Created attachment 529827 [details]
data for reproduction

Extract the archive into /tmp.

start the server: /tmp/slapd-dsa-rsa/run-server.sh
query the server: /tmp/slapd-dsa-rsa/query-server.sh

Comment 1 Elio Maldonado Batiz 2011-10-24 20:12:11 UTC
In http://www.openldap.org/lists/openldap-bugs/201110/msg00021.html
the original reporter states:
>>The problem appears to be related to our particular CA, that uses a DSA key.
>>I've never been able to reproduce the problem creating new CA with an rsa key;
The NSS PEM module only supports RSA. I suspect this may be reason.

Comment 2 RHEL Program Management 2012-05-03 04:58:09 UTC
Since RHEL 6.3 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 4 Elio Maldonado Batiz 2013-04-23 15:47:55 UTC
This would be a request for a new feature, support for DSA in the PEM module. For RHEL-6.5 we must concentrate on several fixing defects on existing features only and cannot afford introduce new features as this request actually is. I recommend deferring this to RHEL-6.6.

Comment 5 Nathan Kinder 2015-10-07 20:28:42 UTC
We do not plan to address this in RHEL 6.x.  We will need to look at addressing this as a part of a larger PEM module cleanup effort for RHEL 7.x.  Pushing this out to RHEL 7.x so we can evaluate when it should be fixed there.

Note You need to log in before you can comment on or make changes to this bug.