(In reply to comment #0)
> It yes, and if openssl currently uses the whitelist-only file ca-bundle.crt,
> then IMHO openssl should eventually switch to the "smarter" file
> ca-bundle.trust.crt which technically might be able to handle blacklisted
> entries, too.

Is that confirmed it works as you expect (i.e. as in NSS)?  There should be little motivation for the switch if it does not.

Also can such switch result in application starting to trust code signing or mail signing CAs when they want to validate servers, as a consequence of the switch (if they are not modified to explicitly specify intended use)?

After quick tests with "openssl verify", that file doesn't appear to be used by openssl by default at all.

As a first step, we should research which libraries/applications make use of the files
/etc/pki/tls/cert.pem
/etc/pki/tls/certs/ca-bundle.crt

> Is that confirmed it works as you expect (i.e. as in NSS)?

No. Feedback from people who know would be good. I don't know when openssl makes use of the default filename cert.pem that is embedded in the code.


>  There should be
> little motivation for the switch if it does not.

True, if openssl did nothing with that file, it doesn't help anyone to switch to a different file.

But we can try to find which tools/libs are using out default CA list from ca-certificates, track those, and track there potential switchover to the new file in the format containing the trust flags?


> Also can such switch result in application starting to trust code signing or
> mail signing CAs when they want to validate servers, as a consequence of the
> switch (if they are not modified to explicitly specify intended use)?

The switchover must be done correctly.
It seems that openssl verification commands already allow to specify which kind of "usage" the verification is supposed to check.
At least the "openssl verify" command offers to do so:

$ openssl verify -?
usage: verify [-verbose] [-CApath path] [-CAfile file] [-purpose purpose] [-crl_check] [-engine e] cert1 cert2 ...
recognized usages:
	sslclient 	SSL client
	sslserver 	SSL server
	nssslserver	Netscape SSL server
	smimesign 	S/MIME signing
	smimeencrypt	S/MIME encryption
	crlsign   	CRL signing
	any       	Any Purpose
	ocsphelper	OCSP helper
	timestampsign	Time Stamp signing

When we are switching the default CA bundle file we should probably also discuss whether we shouldn't switch also to the CA directory format as it allows much easier local configuration of trusted CA certs by the sysadmin.

(In reply to comment #4)
> When we are switching the default CA bundle file we should probably also
> discuss whether we shouldn't switch also to the CA directory format as it
> allows much easier local configuration of trusted CA certs by the sysadmin.

If the RPM package install hundreds of individual files, and the user adds or modifies some of them, I'm worried about the maintenance nightmare. Our package is supposed to change a cert to distrusted if necessary - which we can't do if the user has modified it. I'm worried that having a directory with a mix of automatically and manually installed certs causes lots of confusion. The automatically installed set of certs should be strictly separate from user/system manipulations/overrides.

(In reply to comment #2)
> After quick tests with "openssl verify", that file doesn't appear to be used
> by openssl by default at all.

Looks like I was wrong.

Using strace on "openssl verify" I saw that it DOES open /etc/pki/tls/cert.pem

(and using -CAfile otherfile prevents that file from being opened).

So, we could revert the summary of this bug to again say "default should switchover". But as I just tested in bug 873369, the trust flags in that file don't seem be even evaluated by openssl yet at all.

(In reply to comment #4)
> When we are switching the default CA bundle file we should probably also
> discuss whether we shouldn't switch also to the CA directory format as it
> allows much easier local configuration of trusted CA certs by the sysadmin.

However, CApath is less widely supported than CAfile.  We may not be able to get rid of the CAfile completely, needing some tool (bug 466626?) to build one file from the CApath directory.  Maybe a directory with ca-certificates certs, one per file, another directory for user added CAs, and a tool to combine that (with an easy option to exclude certain CAs from ca-certificates) to a CAfile and CApath?  But discussion this is most likely off-topic here.

So, um, we built the awesome tool (p11-kit and update-ca-trust), and ca-bundle.trust.crt - aka /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt - now handles blacklisted certs properly, I believe.

Shouldn't we switch /etc/pki/tls/cert.pem to point to /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt instead of /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem now?

The one possible danger I can see is that GnuTLS still does not understand the 'trusted certificate' format (so anything using GnuTLS which has been patched to use /etc/pki/tls/cert.pem would break). However, packages patched to use explicit locations are usually patched to point to /etc/pki/tls/certs/ca-bundle.crt , not /etc/pki/tls/cert.pem . There aren't too many packages left that are linked against gnutls - looks like a few dozen - so it should be feasible to eyeball them all and make sure they won't have problems.

For the record, openssl uses /etc/pki/tls/cert.pem when an app asks for the 'default' trust store (via SSL_CTX_set_default_verify_paths() ). /etc/pki/tls is the 'OPENSSLDIR' on Fedora: we build with --openssldir=%{_sysconfdir}/pki/tls . crypto/cryptlib.h does:

#ifndef OPENSSL_SYS_VMS
#define X509_CERT_DIR           OPENSSLDIR "/certs"
#define X509_CERT_FILE          OPENSSLDIR "/cert.pem"

crypto/x509/x509_def.c does:

const char *X509_get_default_cert_dir(void)
        { return(X509_CERT_DIR); }

const char *X509_get_default_cert_file(void)
        { return(X509_CERT_FILE); }

There's a whole messy stack from ssl/ssl_lib.c SSL_CTX_set_default_verify_paths() down, but basically it calls crypto/x509/x509_d2.c X509_STORE_set_default_paths() which tries to load the 'default' cert file, then tries to load the 'default' cert dir; if you trace that all the way down to crypto/x509/by_file.c and crypto/x509/by_dir.c, you see that the lookups use X509_get_default_cert_dir() and X509_get_default_cert_file().

So on Fedora it'll load /etc/pki/tls/cert.pem, which is what X509_get_default_cert_file() will return; it'll also try to load /etc/pki/tls/certs as a directory, but that won't work as it's not in the expected format, so it'll just get the certs from /etc/pki/tls/cert.pem.

Huh, well I tested it, and it's kinda the opposite of what you'd expect.

/tmp/ca.pem is a CA cert:

[adamw@adam openssl (master %)]$ openssl x509 -in /tmp/ca.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Certification Authority
        Validity
            Not Before: Sep 17 19:46:36 2006 GMT
            Not After : Sep 17 19:46:36 2036 GMT
        Subject: C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Certification Authority

/tmp/sub.class2.server.ca.pem is an intermediate cert signed by it:

[adamw@adam openssl (master %)]$ openssl x509 -in /tmp/sub.class2.server.ca.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 26 (0x1a)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Certification Authority
        Validity
            Not Before: Oct 24 20:57:09 2007 GMT
            Not After : Oct 24 20:57:09 2017 GMT
        Subject: C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Class 2 Primary Intermediate Server CA

Now, check this:

[adamw@adam openssl (master %)]$ openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.crt /tmp/sub.class2.server.ca.pem 
/tmp/sub.class2.server.ca.pem: OK
[adamw@adam openssl (master %)]$ openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.trust.crt /tmp/sub.class2.server.ca.pem 
/tmp/sub.class2.server.ca.pem: OK
[adamw@adam openssl (master %)]$ sudo cp /tmp/ca.pem /etc/pki/ca-trust/source/blacklist/
[adamw@adam openssl (master %)]$ sudo update-ca-trust
[adamw@adam openssl (master %)]$ openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.crt /tmp/sub.class2.server.ca.pem 
/tmp/sub.class2.server.ca.pem: C = IL, O = StartCom Ltd., OU = Secure Digital Certificate Signing, CN = StartCom Class 2 Primary Intermediate Server CA
error 20 at 0 depth lookup:unable to get local issuer certificate
[adamw@adam openssl (master %)]$ openssl verify -CAfile /etc/pki/tls/certs/ca-bundle.trust.crt /tmp/sub.class2.server.ca.pem 
/tmp/sub.class2.server.ca.pem: OK
[adamw@adam openssl (master %)]$ 

So if I blacklist the CA cert, then validating the intermediate cert against ca-bundle.crt (correctly) fails, but validating it against ca-bundle.trust.crt succeeds. That wasn't what I'd expected. Maybe I'm missing something, please advise if so.

Examining the produced files, two encoded certs are entirely dropped from ca-bundle.crt when the ca.pem file is in the blacklist directory. In ca-bundle.trust.crt, it seems like just one certificate is changed in some way by the blacklisting:

@@ -40,7 +40,8 @@
 yvqCUqDvr0tVk+vBtfAii6w0TiYiBKGHLHVKt+V9E9e4DGTANtLJL4YSjCMJwRuC
 O3NJo2pXh5Tl1njFmUNj403gdy3hZZlyaQQaRwnmDwFWJPsfvw55qVguucQJAX6V
 um0ABj6y6koQOdjQK/W/7HW/lwLFCRsI3FU34oH7N4RDYiDK51ZLZer+bMEkkySh
-NOsF/5oirpt9P/FlUQqmMGqz9IgcgA38corog14wQjAeBggrBgEFBQcDBAYIKwYB
-BQUHAwEGCCsGAQUFBwMDDCBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
-eQ==
------END TRUSTED CERTIFICATE-----
+NOsF/5oirpt9P/FlUQqmMGqz9IgcgA38corog14wdKBQBggrBgEFBQcDAQYIKwYB
+BQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwUGCCsGAQUFBwMGBggr
+BgEFBQcDBwYIKwYBBQUHAwgMIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9y
+aXR5
+------END TRUSTED CERTIFICATE-----

if I pull out the changed certificate from each version of the file, dump them with x509 -text and diff, the diff looks like what you'd expect:

[adamw@adam openssl (master %)]$ diff -u /tmp/added.txt /tmp/removed.txt 
--- /tmp/added.txt	2015-01-13 02:33:24.444260115 -0800
+++ /tmp/removed.txt	2015-01-13 02:33:10.065091491 -0800
@@ -106,9 +106,9 @@
          62:20:ca:e7:56:4b:65:ea:fe:6c:c1:24:93:24:a1:34:eb:05:
          ff:9a:22:ae:9b:7d:3f:f1:65:51:0a:a6:30:6a:b3:f4:88:1c:
          80:0d:fc:72:8a:e8:83:5e
-Trusted Uses:
-  E-mail Protection, TLS Web Server Authentication, Code Signing
-No Rejected Uses.
+No Trusted Uses.
+Rejected Uses:
+  TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection, IPSec End System, IPSec Tunnel, IPSec User, Time Stamping
 Alias: StartCom Certification Authority
 -----BEGIN CERTIFICATE-----
 MIIHyTCCBbGgAwIBAgIBATANBgkqhkiG9w0BAQUFADB9MQswCQYDVQQGEwJJTDEW

but it seems like OpenSSL still accepts it...

I believe you have to use -purpose for OpenSSL to consider the trust settings at all. I do not think we can switch to the trusted cert bundle by default, only applications calling proper API to set the purpose can use it.

Aha - but a more real-worldish test works more as expected. If I use PHPMailer (picked as I recently wrote the patch for it to do certificate validation...) then things are as they should be. If I set the trust store to be a copy of ca-bundle.trust.crt with the CA cert blacklisted and try connecting to mail.happyassassin.net , the connection fails (but connecting to smtp.gmail.com still works fine, as a sanity check). If I set it to be a copy without the CA cert blacklisted, connecting to mail.happyassassin.net works fine.

To try and replicate my tests, the files are:

http://www.startssl.com/certs/ca.pem
http://www.startssl.com/certs/sub.class2.server.ca.pem

and you're welcome to make connections to www.happyassassin.net and mail.happyassassin.net for testing.

tomas: I did try all the apparently possibly relevant documented -purpose settings (sslclient, sslserver, nssslserver) and they were all rejected no matter what bundle I used. Of course, I've found various other cases where the openssl docs weren't up to scratch, so maybe I missed one.

If the app has to specify a purpose for the 'trusted certificate' format to be useful, yeah, I agree changing the default would obviously be incorrect.

I did not look at the sources directly. So probably for the TLS server certificate verification the purpose is already being set in the TLS implementation. So the most important use-case works fine for the trusted certs format. But I am still quite hesitant to do the switch by-default as there are also the CMS and other uses.

it may also be that PHP sets a default purpose, or something. definitely needs more testing / research, some time when it's not 3am...night.

OK, so after a mere six fucking hours of light trawling through OpenSSL source (ow ow ow where is the tylenol), behold, in openssl/ssl/ssl_cert.c:

int ssl_verify_cert_chain(SSL *s,STACK_OF(X509) *sk)
...
	X509_STORE_CTX_set_default(&ctx,
				s->server ? "ssl_client" : "ssl_server");
...
#ifndef OPENSSL_NO_X509_VERIFY
		i=X509_verify_cert(&ctx);

that second bit is where the actual certificate verification takes place: ctx is an X509_STORE_CTX containing an X509_STORE of trusted certificates, the server (leaf) cert, and the rest of the cert chain. So far as the 'set_default' bit goes, over in crypto/x509/x509_vfy.c we find:

int X509_STORE_CTX_set_default(X509_STORE_CTX *ctx, const char *name)
	{
	const X509_VERIFY_PARAM *param;
	param = X509_VERIFY_PARAM_lookup(name);
	if (!param)
		return 0;
	return X509_VERIFY_PARAM_inherit(ctx->param, param);
	}

quick, over to crypto/x509/x509_vpm.c and we find:

const X509_VERIFY_PARAM *X509_VERIFY_PARAM_lookup(const char *name)
	{
	int idx;
	X509_VERIFY_PARAM pm;

	pm.name = (char *)name;
	if (param_table)
		{
		idx = sk_X509_VERIFY_PARAM_find(param_table, &pm);
		if (idx != -1)
			return sk_X509_VERIFY_PARAM_value(param_table, idx);
		}
	return OBJ_bsearch_table(&pm, default_table,
			   sizeof(default_table)/sizeof(X509_VERIFY_PARAM));
	}

and finally at the top of the same file we find a big:

static const X509_VERIFY_PARAM default_table[]

which sets default check time, flags, purpose, trust, depth, and policies for various cases. 'default' is commented as 'X509 default parameters' and sets purpose to 0 and trust to 0. 'ssl_client' is commented as 'SSL/TLS client parameters' and sets purpose to X509_PURPOSE_SSL_CLIENT and trust to X509_TRUST_SSL_CLIENT; 'ssl_server' is commented as 'SSL/TLS server parameters' and sets purpose to X509_PURPOSE_SSL_SERVER and trust to X509_TRUST_SSL_SERVER.

On the other end, ssl/s3_clnt.c ssl3_connect() calls ssl3_get_server_certificate() which calls ssl_verify_cert_chain(). The stuff above the ssl_verify_cert_chain() call is pretty hard to read but I *think* the upshot is it just gets called on the server certificate. There's a whole bunch of crap about all the different protocols but basically anything SSLv3+ runs through ssl3_connect(), I'm pretty sure.

Basically, yes, it took a hell of a lot of digging, but I'm pretty sure anything that's really doing SSL/TLS will get a default trust and purpose if it doesn't set one itself.

There's still this:

[adamw@adam tmp]$ openssl verify -purpose sslserver -CAfile black/StartCom_Certification_Authority.pem -untrusted sub.class2.server.ca.pem www.happyassassin.net.crt
www.happyassassin.net.crt: OK

www.happyassassin.net.crt is my server cert. Passing the intermediate CA with '-untrusted' like that is the right thing to do to let OpenSSL construct the whole chain. black/StartCom_Certification_Authority.pem is the 'START TRUSTED CERTIFICATE' format file output by `trust extract --format openssl-directory --filter blacklist` when I put the CA cert in /etc/pki/ca-trust/sources/blacklist .

so 'verify' is happy with what should be the same chain, even if I explicitly set a purpose.

But I'm not entirely sure that's wrong. I've been trying to compare the code paths used by s_client and verify for hours now but it's just...hard. Really really hard. :P I'm guessing there's some subtle difference in how they construct the chain and what checks they run on each certificate in the chain that I just haven't figured out yet, and I'm not really sure if it's exactly *wrong* for 'verify' to behave like this, exactly. I'm gonna ask upstream.

Whew, jeez, I return from another six hours at the openssl coalface exhausted but rather enlightened.

So, here's a key thing to understand: 'purpose' and 'trust' checks are quite different, to OpenSSL. They're checking different things in different certificates.

A given OpenSSL cert chain consists of 'trusted certificates' and 'untrusted certificates'. Note that 'trusted certificate' in this context is *not* a "BEGIN TRUSTED" format certificate file. In a typical case, there will be just one 'trusted certificate' - the root CA cert - and all others in the chain (the leaf cert and any intermediates) will be 'untrusted'.

'purpose' checks are run on the *untrusted* certificates, and mostly check that their extensions are consistent with the specified purpose. They're documented in the CERTIFICATE EXTENSIONS section of 'man x509' (and I've checked that the code really lines up with the docs). The code is in crypto/x509v3/v3_purp.c , all the 'check_purpose_foo()' functions. The key point here is *what's* being checked. The 'BEGIN TRUSTED CERTIFICATE' trust stuff is not checked at all by the 'purpose' checks.

It's the 'trust' checks that really matter to us. 'trust' checks are run on the *trusted* certificates (in master) or the last certificate in the chain (in 1.0.1k, presumed to be the root CA cert), and they do check the 'BEGIN TRUSTED CERTIFICATE' trust information - that's exactly what they're for.

You can follow the code from crypto/x509/x509_vfy.c . All the action starts at:

	/* The chain extensions are OK: check trust */

	if (param->trust > 0) ok = check_trust(ctx);

check_trust(ctx) calls X509_check_trust() (which is in x509_trs.c) on the certificate(s), with 'id' set to the store context's param->trust. X509_check_trust() then calls one of the actual trust checking functions; which one it calls depends on the trust parameter (there are mappings between trust types and checking functions in the trstandard[] table at the top of the file). So if we have the trust parameter set to X509_TRUST_SSL_SERVER, it calls trust_1oidany(). If the certificate doesn't have any trust settings this considers it trusted (that's as far as a standard 'BEGIN TRUSTED' certificate would get). If it does have trust settings, it checks whether the X509_TRUST object we found from the param->trust identifier is rejected, trusted or not trusted, and returns X509_TRUST_REJECTED, X509_TRUST_TRUSTED, or X509_TRUST_UNTRUSTED.

Basically - so long as the certificate chain store context has its 'trust' parameter set to something, trust checks will be run on the root CA cert for the appropriate trust type.

I figured out in #c15 that anything which uses the SSL stuff - anything which sets up an SSL context - will wind up with either X509_TRUST_SSL_CLIENT or X509_TRUST_SSL_SERVER (depending on whether it defines itself as a server; if it does it gets X509_TRUST_SSL_CLIENT because it'll be checking client certs if anything, otherwise it gets X509_TRUST_SSL_SERVER because it'll be checking server certs). As p11-kit explicitly distrusts 'blacklisted' certificates for both SSL_CLIENT and SSL_SERVER purposes, I'm about 99% confident in saying that real world stuff should be safe with a p11-kit openssl trust store.

We also distrust blacklisted certs for X509_TRUST_EMAIL, which is the only other possible non-0 default trust setting from default_table[]. So the only default case where our blacklisted certs would *not* be rejected would be the 'X509 default parameters' case, where trust is set to 0 and trust checking is skipped. I don't *think* real world code ought to be hitting that case, but it'd be good if someone else can check my assumption there.

Code could conceivably set trust to 0 explicitly, but then, I mean, they get what they asked for - they're explicitly disabling trust checks.

The reason 'openssl verify' does not fail is because it does no trust checking - even if you pass -purpose. There are some bits that *look* like they're designed so that explicitly setting a purpose should implicitly set a trust, but they don't ever actually seem to have been wired up. More details at https://mta.openssl.org/pipermail/openssl-users/2015-January/000271.html .

Whew. I think I need a break. :)

Tomas,
is this bug still relevant ?

Yes, I think we should consider this for Fedora and RHEL-9.

This package has changed maintainer in the Fedora.
Reassigning to the new maintainer of this component.

Addressed in OpenSSL 3.0.0