Bug 748401

Summary:	SEC_ERROR_BAD_SIGNATURE with a certificate trusted by OpenSSL
Product:	Red Hat Enterprise Linux 7	Reporter:	Jan Vcelak <jvcelak>
Component:	nss	Assignee:	Elio Maldonado Batiz <emaldona>
Status:	CLOSED WONTFIX	QA Contact:	BaseOS QE Security Team <qe-baseos-security>
Severity:	high	Docs Contact:
Priority:	medium
Version:	7.0	CC:	amarecek, emaldona, eparis, kdudka, kengert, nkinder, nmavrogi, rmeggins, rrelyea, tsmetana
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	748394	Environment:
Last Closed:	2016-01-22 16:38:34 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	748394
Bug Blocks:

Description Jan Vcelak 2011-10-24 11:08:35 UTC

The same problem with RHEL-6.2:

openldap-2.4.23-20.el6
nss-3.12.10-14.el6

+++ This bug was initially created as a clone of Bug #748394 +++

Description of problem:

The problem can be reproduced with OpenLDAP. It was reported in OpenLDAP upstream issue tracker (http://www.openldap.org/lists/openldap-bugs/201110/msg00021.html). In fact this is a problem of Mozilla NSS. Fedora is affected.

Version-Release number of selected component (if applicable):

openldap-2.4.26-4.fc15.x86_64
nss-3.12.10-6.fc15.x86_64

Steps to Reproduce:
1. Create CA certificate with DSA signature
2. Create server certificate with RSA signature
3. Sign the certificate by generated CA certificate
4. setup slapd to use these certificates (TLSCertificateFile, TLSCertificateKeyFile, do not set TLSCACertificateFile) and start the server
5. LDAPTLS_CACERT=/your/cacert.pem ldapsearch -x -ZZ -d1 -H ldap://your-server

Actual results:

TLS: loaded CA certificate file /tmp/CA/CA/cacert.pem.
TLS: certificate [CN=alioth.usersys.redhat.com,O=jvcelak Red Hat Test,L=Brno,C=CZ] is not valid - error -8182:Unknown code ___f 10.
TLS: error: connect - force handshake failure: errno 21 - moznss error -8182
TLS: can't connect: TLS error -8182:Unknown code ___f 10.
ldap_err2string
ldap_start_tls: Connect error (-11)
        additional info: TLS error -8182:Unknown code ___f 10

The server side result:

connection_get(12): got connid=1002
connection_read(12): checking for input on id=1002
connection_get(12): got connid=1002
connection_read(12): checking for input on id=1002
TLS: error: accept - force handshake failure: errno 11 - moznss error -12271
TLS: can't accept: TLS error -12271:Unknown code ___P 17.
connection_read(12): TLS accept failure error=-1 id=1002, closing

Expected results:

The connection will succeed.

Additional info:

The failure comes from DSAU_ConvertSignedToFixedUnsigned ->  line 120.

116     if (zCount <= 0) {
117         /* Source is longer than destination.  Check for leading zeros. */
118         while (zCount++ < 0) {
119             if (*pSrc++ != 0)
120                 goto loser;
121         }
122     }

I will attach full backtrace. And configuration file with certificates.

(Do not set TLSCACertificateFile in step 4., otherwise the server certificate validation will fail on the server side with -8182 and all following TLS requests will be refused. This configuration is easier for debugging, you do not have to restart the server with every request.)

--- Additional comment from jvcelak on 2011-10-24 12:50:16 CEST ---

Created attachment 529788 [details]
full backtrace

--- Additional comment from jvcelak on 2011-10-24 13:06:16 CEST ---

Created attachment 529827 [details]
data for reproduction

Extract the archive into /tmp.

start the server: /tmp/slapd-dsa-rsa/run-server.sh
query the server: /tmp/slapd-dsa-rsa/query-server.sh

Comment 1 Elio Maldonado Batiz 2011-10-24 20:12:11 UTC

In http://www.openldap.org/lists/openldap-bugs/201110/msg00021.html
the original reporter states:
>>The problem appears to be related to our particular CA, that uses a DSA key.
>>I've never been able to reproduce the problem creating new CA with an rsa key;
The NSS PEM module only supports RSA. I suspect this may be reason.

Comment 2 RHEL Program Management 2012-05-03 04:58:09 UTC

Since RHEL 6.3 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 4 Elio Maldonado Batiz 2013-04-23 15:47:55 UTC

This would be a request for a new feature, support for DSA in the PEM module. For RHEL-6.5 we must concentrate on several fixing defects on existing features only and cannot afford introduce new features as this request actually is. I recommend deferring this to RHEL-6.6.

Comment 5 Nathan Kinder 2015-10-07 20:28:42 UTC

We do not plan to address this in RHEL 6.x.  We will need to look at addressing this as a part of a larger PEM module cleanup effort for RHEL 7.x.  Pushing this out to RHEL 7.x so we can evaluate when it should be fixed there.