Bug 1142137

Summary: GnuTLS doesn't find alternative chain of trust if SSL/TLS server includes additional intermediate CA certificate pointing to removed legacy root CA certificate (in this case, to a Verisign 1024-bit certificate from 1996 that got recently distrusted)
Product: [Fedora] Fedora Reporter: Dimitris <dimitris.on.linux>
Component: ca-certificatesAssignee: Kai Engert (:kaie) (inactive account) <kengert>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: dwmw2, hkario, jorton, kengert, nmavrogi, pwouters, tmraz
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-25 11:41:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
2
none
3
none
dump of 2
none
dump of 3 none

Description Dimitris 2014-09-16 08:48:51 UTC
Description of problem:
Trying to connect to https://na15.salesforce.com:

- With 2013.1.97-1.fc20, connection succeeds.
- Wtih 2014.2.1-1.0.fc20, certificate validation fails.


Version-Release number of selected component (if applicable):
2014.2.1-1.0.fc20

How reproducible:
Always

Steps to Reproduce:
gnutls-cli na15.salesforce.com

Actual results:
Processed 153 CA certificate(s).
Resolving 'na15.salesforce.com'...
Connecting to '96.43.146.88:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `C=US,ST=California,L=San Francisco,O=Salesforce.com\, Inc.,OU=Applications,CN=*.salesforce.com', issuer `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=Terms of use at https://www.verisign.com/rpa (c)10,CN=VeriSign Class 3 Secure Server CA - G3', RSA key 2048 bits, signed using RSA-SHA1, activated `2014-01-10 00:00:00 UTC', expires `2017-01-21 23:59:59 UTC', SHA-1 fingerprint `2003b7e5316c3b99437d895e013fe6ae54009c6e'
	Public Key Id:
		1ee4319a09b2303600145644a5f9378cb078ef58
	Public key's random art:
		+--[ RSA 2048]----+
		|=+=+..           |
		|o   o            |
		|oo.+.   +        |
		|.ooo+.o* o       |
		| ..o o++S        |
		|  . . ....       |
		|     E  .        |
		|    +            |
		|   . .           |
		+-----------------+

- Certificate[1] info:
 - subject `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=Terms of use at https://www.verisign.com/rpa (c)10,CN=VeriSign Class 3 Secure Server CA - G3', issuer `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=(c) 2006 VeriSign\, Inc. - For authorized use only,CN=VeriSign Class 3 Public Primary Certification Authority - G5', RSA key 2048 bits, signed using RSA-SHA1, activated `2010-02-08 00:00:00 UTC', expires `2020-02-07 23:59:59 UTC', SHA-1 fingerprint `5deb8f339e264c19f6686f5f8f32b54a4c46b476'
- Certificate[2] info:
 - subject `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=(c) 2006 VeriSign\, Inc. - For authorized use only,CN=VeriSign Class 3 Public Primary Certification Authority - G5', issuer `C=US,O=VeriSign\, Inc.,OU=Class 3 Public Primary Certification Authority', RSA key 2048 bits, signed using RSA-SHA1, activated `2006-11-08 00:00:00 UTC', expires `2021-11-07 23:59:59 UTC', SHA-1 fingerprint `32f30882622b87cf8856c63db873df0853b4dd27'
- Status: The certificate is NOT trusted. The certificate issuer is unknown. 
*** Verifying server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.

Expected results:
Processed 153 CA certificate(s).
Resolving 'na15.salesforce.com'...
Connecting to '96.43.146.88:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `C=US,ST=California,L=San Francisco,O=Salesforce.com\, Inc.,OU=Applications,CN=*.salesforce.com', issuer `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=Terms of use at https://www.verisign.com/rpa (c)10,CN=VeriSign Class 3 Secure Server CA - G3', RSA key 2048 bits, signed using RSA-SHA1, activated `2014-01-10 00:00:00 UTC', expires `2017-01-21 23:59:59 UTC', SHA-1 fingerprint `2003b7e5316c3b99437d895e013fe6ae54009c6e'
	Public Key Id:
		1ee4319a09b2303600145644a5f9378cb078ef58
	Public key's random art:
		+--[ RSA 2048]----+
		|=+=+..           |
		|o   o            |
		|oo.+.   +        |
		|.ooo+.o* o       |
		| ..o o++S        |
		|  . . ....       |
		|     E  .        |
		|    +            |
		|   . .           |
		+-----------------+

- Certificate[1] info:
 - subject `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=Terms of use at https://www.verisign.com/rpa (c)10,CN=VeriSign Class 3 Secure Server CA - G3', issuer `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=(c) 2006 VeriSign\, Inc. - For authorized use only,CN=VeriSign Class 3 Public Primary Certification Authority - G5', RSA key 2048 bits, signed using RSA-SHA1, activated `2010-02-08 00:00:00 UTC', expires `2020-02-07 23:59:59 UTC', SHA-1 fingerprint `5deb8f339e264c19f6686f5f8f32b54a4c46b476'
- Certificate[2] info:
 - subject `C=US,O=VeriSign\, Inc.,OU=VeriSign Trust Network,OU=(c) 2006 VeriSign\, Inc. - For authorized use only,CN=VeriSign Class 3 Public Primary Certification Authority - G5', issuer `C=US,O=VeriSign\, Inc.,OU=Class 3 Public Primary Certification Authority', RSA key 2048 bits, signed using RSA-SHA1, activated `2006-11-08 00:00:00 UTC', expires `2021-11-07 23:59:59 UTC', SHA-1 fingerprint `32f30882622b87cf8856c63db873df0853b4dd27'
- Status: The certificate is trusted. 

Additional info:
As far as I can tell these certs are still valid, and were not in the list of 1024-bit CA certificates that were removed by Mozilla.

Comment 1 Kai Engert (:kaie) (inactive account) 2014-09-16 20:32:37 UTC
The certificate[2] sent by the server has issuer:
`C=US,O=VeriSign\, Inc.,OU=Class 3 Public Primary Certification Authority'

This certificate had its trust for SSL/TLS removed in version 2.1 of the CA list.

Because GnuTLS insists on finding a trust chain for the topmost intermediate CA certificate sent by the server, only, it fails to find a valid chain.

There are two solutions to this problem:

- enhance GnuTLS to be more flexible, and attempt to find an alternative
  trust chain for other included intermediates.
  This would work, because there is another trusted root CA cert in
  the global Mozilla CA list, that matches the issuer of cert[1].
  (This is what the NSS library does, and software using NSS
   still works fine with servers that are configured like this one.)

- reconfigure all servers on the Internet to no longer send any unnecessary
  intermediates.
  This is unrealistic, because there are too many servers and certs
  that are still valid,
  and because server operators might be worried about breaking some old
  legacy clients (software or devices) if they make that change.

Comment 2 David Woodhouse 2014-09-16 21:52:04 UTC
Note: This fails with openssl too:

$ openssl s_client -verify 3  -connect na15.salesforce.com:443
...
    Verify return code: 27 (certificate not trusted)

It appears to fail with NSS tstclint too, even with the *old* ca-certificates package. But perhaps I'm doing something wrong:

$ /usr/lib64/nss/unsupported-tools/tstclnt  -h na15.salesforce.com -p 443
tstclnt: unable to open cert database: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
$ /usr/lib64/nss/unsupported-tools/tstclnt  -h na15.salesforce.com -p 443 -d sql:/etc/pki/nssdb
tstclnt: authentication of server cert failed: SEC_ERROR_UNKNOWN_ISSUER: Peer's Certificate issuer is not recognized.

Comment 3 Kai Engert (:kaie) (inactive account) 2014-09-16 22:01:13 UTC
(In reply to David Woodhouse from comment #2)
> It appears to fail with NSS tstclint too, even with the *old*
> ca-certificates package. But perhaps I'm doing something wrong:
> 
> $ /usr/lib64/nss/unsupported-tools/tstclnt  -h na15.salesforce.com -p 443
> tstclnt: unable to open cert database: SEC_ERROR_LEGACY_DATABASE: The
> certificate/key database is in an old, unsupported format.
> $ /usr/lib64/nss/unsupported-tools/tstclnt  -h na15.salesforce.com -p 443 -d
> sql:/etc/pki/nssdb
> tstclnt: authentication of server cert failed: SEC_ERROR_UNKNOWN_ISSUER:
> Peer's Certificate issuer is not recognized.

By default, NSS uses an empty trust list. Either the application must load a trust list module, or the database used must be configured to load it.

mkdir /tmp/nssdb
certutil -d sql:/tmp/nssdb -N --empty-password
modutil -dbdir sql:/tmp/nssdb -add roots \
        -libfile /etc/alternatives/libnssckbi.so.x86_64
tstclnt -d sql:/tmp/nssdb -p 443 -h na15.salesforce.com

works

Comment 4 David Woodhouse 2014-09-16 22:06:51 UTC
I'm a little confused. A certificate bears a signature from a specific issuer, and it specifies the issuer. Usually using the X509v3 Authority Key Identifier. In the example at hand, it is as follows:

  X509v3 Authority Key Identifier: 
       keyid:0D:44:5C:16:53:44:C1:82:7E:1D:20:AB:25:F4:01:63:D8:BE:79:A5

The server also includes the certificate with ID 0d445c....be79a5. Which in turn is signed by a certificate with ID 7fd365....333133, etc.

At what point do we get to *choose* to use a different trust chain? I think I must be missing something.

If the server presents an *invalid* trust chain in its handshake, where one of the certs it presents is *not* the issuer of the previous cert in the chain, GnuTLS does truncate the chain at the brokenness and then attempt to find the trust root from there (I fixed this a year or two ago). But it's not clear to me what is expected here.

Comment 5 Kai Engert (:kaie) (inactive account) 2014-09-16 22:49:49 UTC
0: server cert
   signed by: ca-cert A

1: ca-cert A
   signed by: ca-cert B

2: ca-cert B
   signed by: ca-cert C

3: ca-cert B
   self-signed

If 2 and 3 are based on the same key pair, then the signature contained in 1 can be verified by either 2 or 3.

ca-cert C got removed from the pre-configured trust list.

The server sends 0+1+2

openssl and gnutls search for ca-cert C, fail, and give up.

NSS considers alternative 3, because the subject name of 3 matches the issuer name listed in 1.

If 3 contains a key that can be used to verify the signature in 1, verification succeeds.

If the authority key identifier is indeed based on the key, not on the whole issuer certificate, then the authority key ID should be identical for 2 and 3.

I'll attach the relevant certs 2 and 3 for this scenario.
Both certs contain identical subject key IDs, which means, they indeed both match the authority key ID of the subordinate cert 1.

Comment 6 Kai Engert (:kaie) (inactive account) 2014-09-16 22:51:56 UTC
Created attachment 938252 [details]
2

Comment 7 Kai Engert (:kaie) (inactive account) 2014-09-16 22:52:15 UTC
Created attachment 938253 [details]
3

Comment 8 Kai Engert (:kaie) (inactive account) 2014-09-16 22:53:40 UTC
Created attachment 938254 [details]
dump of 2

Comment 9 Kai Engert (:kaie) (inactive account) 2014-09-16 22:54:00 UTC
Created attachment 938255 [details]
dump of 3

Comment 10 Nikos Mavrogiannopoulos 2014-09-17 06:51:53 UTC
Kai,
 There is no reason to play ping-pong with this bug. It is already reported in ca-certificates (#1134602). Removing such important CA certificates from the system trust list is a system wide change and you must follow the process in:
http://fedoraproject.org/wiki/Changes/Policy

Since you have not followed that changes policy I reassign it to the ca-certificates package. The reason to use the changes policy is so that we can coordinate and offer a workable system to users.


As to the point of your request, it is a wishlist item. There is no protocol requirement from an implementation to discover alternative paths. We can disagree on that, but don't put the users of Fedora in the middle by forcing that change on a stable release.

Comment 11 Nikos Mavrogiannopoulos 2014-09-17 06:56:31 UTC
(In reply to Kai Engert (:kaie) from comment #1)

> There are two solutions to this problem:
> 
> - enhance GnuTLS to be more flexible, and attempt to find an alternative
>   trust chain for other included intermediates.
>   This would work, because there is another trusted root CA cert in
>   the global Mozilla CA list, that matches the issuer of cert[1].
>   (This is what the NSS library does, and software using NSS
>    still works fine with servers that are configured like this one.)
> 
> - reconfigure all servers on the Internet to no longer send any unnecessary
>   intermediates.
>   This is unrealistic, because there are too many servers and certs
>   that are still valid,
>   and because server operators might be worried about breaking some old
>   legacy clients (software or devices) if they make that change.

There is a third solution you forgot. Add the intermediate certificates issued by the CAs that are being removed to the trusted list.

Comment 12 Kai Engert (:kaie) (inactive account) 2014-09-29 19:33:00 UTC
(In reply to Nikos Mavrogiannopoulos from comment #11)
> 
> There is a third solution you forgot. Add the intermediate certificates
> issued by the CAs that are being removed to the trusted list.

It's futile to come up with that full set. Additionally, you'll lose the OCSP ability, if you no longer trust OCSP signatures of the CA cert that issued those intermediates. It's unclear which certificates are still good or which ones need to be revoked.

These difficulties can be avoided by finding the trust chain to the alternative issuer and ignoring the topmost intermediate in the chain.

Comment 13 Nikos Mavrogiannopoulos 2014-09-30 08:19:38 UTC
(In reply to Kai Engert (:kaie) from comment #12)
> (In reply to Nikos Mavrogiannopoulos from comment #11)
> > 
> > There is a third solution you forgot. Add the intermediate certificates
> > issued by the CAs that are being removed to the trusted list.
> 
> It's futile to come up with that full set.

Not really, because for every intermediate that was removed a new anchor certificate  was introduced. So if we have X new anchors added to allow verification after the change we need the X intermediate certificates with the same key and DN as them.

> Additionally, you'll lose the
> OCSP ability, if you no longer trust OCSP signatures of the CA cert that
> issued those intermediates. It's unclear which certificates are still good
> or which ones need to be revoked.

OCSP will have to work anyway because as you said sites will still be using the old CA anchors and the CAs will have to provide OCSP responses for them.

Comment 14 Nikos Mavrogiannopoulos 2015-02-25 11:41:51 UTC
I believe that issue is addressed in F21.