I just tracked down a bug in git send-email (#1043194), which uses /etc/ssl/certs as the directory for OpenSSL to find its CA certificates in.
A fix for that bug is forthcoming, but I found it quite surprising that /etc/ssl/certs is not in the proper format for OpenSSL to use.
If CApath is not NULL, it points to a directory containing CA certificates in PEM format. The files each contain one CA certificate. The files are looked up by the CA subject name hash value, which must hence be available. If more than one CA certificate with the same name hash value exist, the extension must be different (e.g. 9d66eef0.0, 9d66eef0.1 etc). The search is performed in the ordering of the extension number, regardless of other properties of the certificates. Use the c_rehash utility to create the necessary links.
I checked Debian, which does this right, would it be possible for Fedora to also create the right links to individual CA files?
(In reply to Ruben Kerkhof from comment #0)
> A fix for that bug is forthcoming, but I found it quite surprising that
> /etc/ssl/certs is not in the proper format for OpenSSL to use.
Why? I guess there wasn't a standard, and each distribution invented their own files and directories in the past. I think recent Fedora versions have been using a bundle file, not a directory of individual files/hashes.
Therefore it's unfortunately necessary to configure applications to load the system trust anchors from the distribution specific location.
> From http://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html:
> If CApath is not NULL, it points to a directory containing CA certificates
> in PEM format. The files each contain one CA certificate. The files are
> looked up by the CA subject name hash value, which must hence be available.
> If more than one CA certificate with the same name hash value exist, the
> extension must be different (e.g. 9d66eef0.0, 9d66eef0.1 etc). The search is
> performed in the ordering of the extension number, regardless of other
> properties of the certificates. Use the c_rehash utility to create the
> necessary links.
> I checked Debian, which does this right, would it be possible for Fedora to
> also create the right links to individual CA files?
I cannot say which approach is "right". I think your application code simply assumed that what debian had done is what everyone does (which was an incorrect assumption).
If necessary, we could begin to create such a directory, but it would have to be at a different path (because we cannot move the bundle files, and we shouldn't mix bundles and individual files/hashes in a single directory).
So, in bug 1043194, it makes sense that you intend to use the default location that openssl has embedded.
I believe your application code should rely on the openssl distribution package to know from where it can load the system's default trust anchors. On Fedora it does. I don't know about other distributions.
If for whatever reason you still required a directory with individual files and hashes, we could create a new directory at
But I think this won't solve your problem, as you were looking for something to work out of the box, and you'd still require a patch to your application to load from this directory.
Note that since Fedora 19 we have introduced a new system. Read about it at
I understand other distributions are planning to use the same approach, so there's hope that we might have consistency across distributions.
I just came across a similar issue in Bip (package bip), an IRC proxy. It also expects you to provide a CApath-type directory.
I submitted a patch to upstream which checks whether the provided location is a file or a directory and treats it appropriately:
but since there've been now at least two apps which ran into this, perhaps it might be a good idea for ca-certificates to provide such a representation, as there may be more.
for the record, in case it helps anyone working with this, the openssl function SSL_CTX_load_verify_locations() - https://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html - handles both 'CApath' locations (of the type discussed in this bug) and 'CAfile' locations (which are expected to be pem certificate bundles, like the ones ca-certificates provides).
Some historical context here in case it helps: the /etc/ssl/certs symlink was added in 2010, in response to https://bugzilla.redhat.com/show_bug.cgi?id=572725 , where it was claimed that "3rd party tools (eg. flash) look in /etc/ssl/certs for Certificate Authorities SUSE, Debian, and upstream all use this path", and that a simple symlink worked.
Poking through the Debian ca-certificates changelog:
and various bugs:
there appears to have been some malarkey with these capath directories around OpenSSL 1.0, but we can at least infer that on Debian, that path has been c_rehashed since 2011 and probably before. The changelog also strongly implies that Debian has it set up as a kind of hybrid, e.g. this entry:
" * sbin/update-ca-certificates: move the ca-certificates.crt bundle out of
the way before calling c_rehash, so that symlinks don't accidentally get
pointed here, breaking openssl certificate verification LP: #854927"
It sounds like they do exactly what Kai was worried about in c#1, i.e. "mix bundles and individual files/hashes in a single directory". Indeed I just fired up an Ubuntu 13.04 image I have lying around, and /etc/ssl/certs contains a bunch of pem files and hash symlinks, but also a bundle file, ca-certificates.crt .
I agree with Kai that this seems like a bad approach, but since we're explicitly providing /etc/ssl/certs as a compatibility symlink (per #572725 ), we should probably try and do as good a job of it as we can.
See also: http://manpages.ubuntu.com/manpages/utopic/man8/update-ca-certificates.8.html
The current Firefox flash plugin appears to be using NSS, and /usr/bin/strings implies it is not using /etc/ssl/certs directly.
I'd be +1 on removing the /etc/ssl/certs symlink entirely, any other votes?
Upstream projects should be encouraged to use SSL_CTX_set_default_verify_paths() by default which it looks like you are doing here Adam, that's great.
IMO, the long term goal should be for consumers of OpenSSL to be agnostic to the location of the trusted CAs on the filesystem, and for that to be an implementation detail of OpenSSL/ca-certificates.
The original bug is talking about the Flash plugin, not Firefox. At that time the default WP uploader used Flash.
The Flash plugin for Firefox, yes that is what I'm looking at too.
$ strings /usr/lib64/flash-plugin/libflashplayer.so | grep ssl
$ ldd /usr/lib64/flash-plugin/libflashplayer.so | grep ssl
libssl3.so => /lib64/libssl3.so (0x00007f91f877d000)
$ rpm -qf /usr/lib64/flash-plugin/libflashplayer.so
ah, sorry, I must've blipped right over 'current Firefox flash plugin' in your post. However, as the other examples cited in earlier comments on this bug show, it certainly wasn't the only thing. I'd say that when someone files a bug which can be characterized as 'our attempt to be compatible with Debian's cert layout is imperfect', it's a bit strange to suggest the appropriate response is 'give up entirely'. I'm fairly sure that's not what the reporter was expecting. ;)
As the directory is explicitly for compat purposes, I'd say the appropriate thing to do is make a best effort to hash it at some point in the process, and dump a text file in it stating explicitly that it's not a canonical source for Fedora/Red Hat, we provide it only to try and be compatible with code that assumes Debian's layout is available everywhere, and we just do our best to make it work like Debian's.
Whenever I don't see clear, I try to summarize, so let's do that, and please correct me where I'm wrong.
- Fedora never shipped a directory with individual CA certificates
and hashed symlinks
- I assume that applications that are built as Fedora packages have
been adjusted to use the resources that we have on Fedora.
- In the past, some applications were known to look for
/etc/ssl/certs/ca-bundle.crt, which was the motiviation
to create the symlink in bug 572725 from
/etc/ssl/certs/ to /etc/pki/tls/certs/
and it had helped.
- You recently discovered there are additional apps, probably something
you compiled from source (?), that expect
/etc/ssl/certs to be a directory containing individual CA cert files
with hashed filenames.
You claim, in order to fully support all scenarios, we'd have to change /etc/ssl/certs to be a directory, filled with the hashed files.
If we did that, because we shouldn't break what we already provide, we'd still have to provide the /etc/ssl/certs/ca-bundle.crt file, in addition to all the other files in that directory.
But here is another confusion that we need to clarify.
As said before, in bug 572725, the expected bundle file was
But as said in comment 3, and something that I can also confirm, recent debian doesn't provide that file any more, instead, the bundle file is available as:
That's confusing. Did we get any confirmation in the past that bug 572725 had indeed helped? After Joe had provided the updated build, there hasn't been any follow up comment in the bug.
In other words, did the change from bug 572725 never work, or did debian change the filename?
Whatever happened, I believe it's correct to say that as of today:
- Fedora doesn't provide the same bundle file that debian currently uses
- Fedora doesn't provide the hashed directory.
It seems that right now, we don't provide anything that's compatible with debian.
(In reply to Adam Williamson (Red Hat) from comment #3)
> " * sbin/update-ca-certificates: move the ca-certificates.crt bundle out of
> the way before calling c_rehash, so that symlinks don't accidentally get
> pointed here, breaking openssl certificate verification LP: #854927"
Based on this quote from the debian changelog, it seems like they are in a messy situation, and are required to use ugly hacks to keep their configuration working.
If it's correct that we've never been compatible, I would very much prefer if we avoided such a mess (a mixed directory).
If there are good arguments to start being compatible with debian, then I think we'd have to make a decision, whether we provide the hashed directory, or the symlink. We shouldn't do both (and avoid the mixed directory).
If my understanding is correct that so far, all Fedora packages have been configured to work correctly with the CA certificates list available on Fedora, and the only motivation here is to make software built from source work out of the box, without any configuration adjustments, then I'd like to argue against it. Given that there hasn't been a standard yet, software should try to detect what's available.
Furthermore, we see the certificate files as legacy data. The files are too stupid, they cannot transport blacklisted certs, and they cannot contain trust flags (e.g. CAs that are trusted only for email communication). The recommendation is to instead load a pkcs#11 device (p11-kit-trust) that provides the set with details.
Based on the above, I'd like to keep everything as it is.
I don't see a strong need to remove the /etc/ssl/certs symlink, just in case there are legacy applications that still look for ca-bundle.crt and find it that way. If we wanted to remove it, we should research if any Fedora packages use that filename to access it.
I wouldn't try to provide the new /etc/ssl/certs directory, because no applications have ever been able to expect it on Fedora. I wouldn't want to go half-way and provide the hashed files, only. If we did, someone could argue that we are trying to be compatible, but in order to be fully compatible, we'd have to provide the mixed directory, too. I don't want to go there.
If there is indeed a strong need for some applications to insist on using a directory with hashed filenames, we could provide it using a new path, but I wouldn't want to provide it as /etc/ssl/certs.
"- Fedora never shipped a directory with individual CA certificates and hashed symlinks"
Agreed (unless it was *really* old), AFAIK we never had such a 'hybrid'.
"- I assume that applications that are built as Fedora packages have been adjusted to use the resources that we have on Fedora."
I'm not sure this would be 100% the case. There are non-obvious cases here - bip is a good example, because its SSL functionality isn't very prominent. The current Fedora bip package can't be said to work well with 'the resources that we have on Fedora', really. Once it's updated with the fix I got merged upstream it will be, but I wouldn't be surprised if there are other cases too.
In an *ideal world* all Fedora packages that have any kind of certificate checking would be set up to work well with Fedora's SSL libs and certificate locations OOTB, but I'm not sure it's currently the case. Thus the presence of a /etc/ssl/certs directory *might* happen to help some existing Fedora package, as well as third-party stuff.
"- In the past, some applications were known to look for /etc/ssl/certs/ca-bundle.crt, which was the motiviation to create the symlink in bug 572725 from /etc/ssl/certs/ to /etc/pki/tls/certs/ and it had helped."
That's more or less my reading of #572725 too. Though I suppose I'd note that although the bug summary and text keeps talking in the plural, it only ever appears that one specific case was actually tested - the Flash plugin at that time.
Googling around for /etc/ssl/certs/ca-bundle.crt doesn't give much. https://bbs.archlinux.org/viewtopic.php?id=187409 looks interesting at first but when I traced it out it looks more or less like Arch simply switched from the Debian system to our system; I don't think they actually use /etc/ssl/certs/ca-bundle.crt as a canonical location in either setup.
I'd say the evidence suggests that few if any projects use the specific /etc/ssl/certs/ca-bundle.crt location (any more?), though there are various forum threads etc that refer to it.
":- You recently discovered there are additional apps, probably something you compiled from source (?), that expect /etc/ssl/certs to be a directory containing individual CA cert files with hashed filenames."
No, not compiled from source. We have reported two separate cases where Fedora-packaged software expected a hashed directory, and I expected there are others.
Between 2013-07-18 (commit https://github.com/git/git/commit/35035bbf074d1a4c59cd5f99282c12197105da08 ) and 2014-01-15 (commit https://github.com/git/git/commit/01645b749376b7026276112d69d5b9c22b000ee4 ), upstream git send-email by default expected /etc/ssl/certs to exist and be a hashed directory, and performed no verification if it was not, unless you explicitly configured a correct bundle file location. (Admittedly, prior to 2013-07-18 it never did any verification in any case, so things didn't get any *worse* on Fedora, but we were missing the benefit of an upstream change). This was not patched downstream in Fedora's package, or anything. It just plain didn't work.
From ~2008-05-20 (when it was first introduced as a Fedora package: https://bugzilla.redhat.com/show_bug.cgi?id=437694 ) to date, it is not possible to do certification validation with Fedora's packaged bip unless you manually create a hashed cert directory, because bip will only use a hashed directory as a trust store, it has no ability to use a bundle file. Upstream merged my fix for this last month - https://projects.duckcorp.org/issues/350 - but there has not yet been a new upstream release and the Fedora package has not yet been updated.
The fact that issues like this went unnoticed in a very high-profile piece of software (git!) for ~6 months, and in a lower-profile but not completely obscure one like bip for 6 years, suggests to me that it's reasonably likely other such issues may exist in other Fedora packages. Upstreams don't necessarily always understand this stuff. It seems that sometimes they just look at the layout on their dev platform and assume it's a universal standard.
"You claim, in order to fully support all scenarios, we'd have to change /etc/ssl/certs to be a directory, filled with the hashed files."
I claim that in order to achieve compatibility with the Debian layout - which is the stated goal of the directory - it should both contain individual certificate files and have c_rehash run on it at some point, and contain a bundle file named ca-certificates.crt . You make a valid point with the confusion over the file name, which my above research into the /etc/ssl/certs/ca-bundle.crt path seems to corroborate. I cannot find any support other than that sole bug report for the idea that anything actually uses /etc/ssl/certs/ca-bundle.crt (possibly NixOS? http://comments.gmane.org/gmane.linux.distributions.nixos/11170 - I'll look into that).
"In other words, did the change from bug 572725 never work, or did debian change the filename?"
There's a third possibility, which is that Debian never had that file, but at some point the Flash plugin would somehow be happy if that file existed. I imagine it'd be hard to check, at this point in time.
"It seems that right now, we don't provide anything that's compatible with debian."
Yes, I agree.
OK, so Kai and I talked this over a bit more on IRC. We also spotted that OpenSUSE has its own trust store mechanism along the lines of (but different to both) ours and Debian's:
We more or less agreed that it'd make more sense to have /etc/ssl/certs as a hashed directory than have the /etc/ssl/certs/ca-bundle.crt file, but we also agreed that it might be possible after all to do the 'hybrid' approach, and this would be the best choice if we could do it cleanly.
I believe in theory this should be possible. Here's my proposed approach. Add another few commands to update-ca-trust:
/usr/bin/p11-kit extract --format=pem-hash-directory --filter=ca-anchors --overwrite --purpose server-auth $DEST/pem/hash
ln -s $DEST/pem/hash/ca-certificates.crt ../tls-ca-bundle.pem
ln -s $DEST/pem/hash/ca-bundle.crt ../tls-ca-bundle.pem
and have the symlink:
/etc/ssl/certs -> /etc/pki/ca-trust/extracted/pem/hash
A small snag is that "/usr/bin/p11-kit extract --format=pem-hash-directory" doesn't actually exist, but it's kinda supposed to. SUSE actually wrote a patch to p11-kit which allows it to output a hashed directory in the more basic PEM format (not the OpenSSL-specific format with 'trusted certificates') and submitted it upstream:
Stef is OK with it in principle, but didn't like the hack to use an environment variable to change the behaviour of p11_extract_pem_directory() and asked for it to be done as a new --format parameter instead:
Ludwig said he'd get back to it 'later', but neither he nor Stef ever did, so now there's a small fork between upstream and SUSE, unfortunately - SUSE still carries the patch and uses the environment variable in their tool.
So my idea is that we go ahead and do what was proposed then and add that new --format parameter, then we can use it in the plan outlined above. I think that ought to allow us to have a fully Debian-compatible /etc/ssl/certs . CCing stefw to see what he thinks now.
Optionally we could also do:
/usr/bin/p11-kit extract --format=openssl-directory --filter=certificates --overwrite $DEST/openssl/hash
and possibly have:
/etc/ssl/certs.trust -> /etc/pki/ca-trust/extracted/openssl/hash
but that'd be kinda making stuff up, I don't know if it's really needed. AFAICS Debian doesn't have any kind of trust store in OpenSSL "trusted certificate" format at present.
I'm willing to try the revised p11-kit patch if no-one else is, but it'd be good to have Stef's opinion on what the best design would be - where should p11_openssl_symlink() go (maybe renamed p11_hash_symlink()?), should the new parameter get its own function or should it be part of p11_extract_pem_directory()?
Filed https://bugs.freedesktop.org/show_bug.cgi?id=88347 for the required enhancement to p11-kit.
While investigating this whole area today I did come across at least one case where Fedora was using the /etc/ssl/certs/ca-bundle.crt location: python-httplib2 was patched to use that location (instead of an upstream bundled file). I've now corrected it to use /etc/pki/tls/certs/ca-bundle.crt instead for Rawhide (and the fix can be backported if necessary), but there may be others.
As well as everything else, we probably ought to have a README or something in /etc/ssl which declares that it's present for compat purposes and Fedora things are intended to use /etc/pki/tls...
http://xkahn.zoned.net/blog/2010/03/10/managing-certificate-authorities-in-adobe-flash-on-fedorarhel/ seems to be a contemporary source for the Flash thing, though it's rather sketchy and I really don't see exactly how it was supposed to work. Found that from the Mandriva bug where they copied the same compatibility symlink from us.
I kinda couldn't resist my historical urge so I did some digging on what the heck is the deal with our certs/ directory in the first place. We really seem to have just pretty much made it up as we were going along. ca-bundle.crt was invented in 2000:
* Wed Oct 25 2000 Nalin Dahyabhai <firstname.lastname@example.org>
- add a ca-bundle file for packages like Samba to reference for CA certificates
almost immediately it was symlinked to cert.pem:
* Tue Oct 31 2000 Nalin Dahyabhai <email@example.com>
- add a symlink from cert.pem to ca-bundle.crt
at that time, it was /usr/share/ssl/certs/ca-bundle.crt linked to /usr/share/ssl/cert.pem .
Why we invented this 'ca-bundle.crt' file and placed it in certs/ instead of simply making it cert.pem - the file OpenSSL would've expected - I've really no idea. I can't find any kind of reference to 'ca-bundle.crt' prior to RH's introduction of it, it seems to have just been made up out of thin air.
So our OPENSSLDIR/certs directory really is kind of an odd duck; AFAICT it was initially created along with the initial openssl package to follow OpenSSL's expectation of a 'default certificate directory', but the only thing we ever put into it is a bundle file, and OpenSSL never, so far as I can tell, has done anything with bundle files place in that directory. The only thing OpenSSL has ever expected to find in there is hashed individual certificate files (well, I don't know *precisely* when the c_rehash stuff was added, but it's been there at least since 0.9.5a). We're a bit stuck with it now, though, it's just a historical note.
For the record, #c16 isn't quite the whole story about certs/ and ca-bundle.crt, but to avoid spamming the bug any more I added the further details to https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/ . Briefly, ca-bundle.crt was actually introduced by mod_ssl in 1998, and copied into the openssl package by RH to make it available without needing Apache and mod_ssl to be installed, though I still can't tell for sure why it was placed in OPENSSLDIR/certs.
Yesterday I revised Ludwig's patch to allow p11-kit to output a hashed directory of basic "BEGIN CERTIFICATE" files (rather than the OpenSSL-extended "BEGIN TRUSTED CERTIFICATE" ones) with Stef's suggestions from January 2014, and Stef kindly merged it:
With that p11-kit, I've now written and tested changes to ca-certificates which (for me) provide a /etc/ssl/certs that's both Debian-compatible and backwards-compatible (provides a /etc/ssl/certs/ca-bundle.crt). It also includes a /etc/ssl/README that explains the purpose of the directory, and that anything which relies on it is bad code and ought to be fixed. I'll attach a spec diff.
Created attachment 980200 [details]
ca-certificates diff to make /etc/ssl/certs properly Debian-compatible (requires upstream master p11-kit at present)
Created attachment 980201 [details]
ca-certificates diff to make /etc/ssl/certs properly Debian-compatible (requires upstream master p11-kit at present)
Notes: I didn't include /etc/ssl/certs/ca-bundle.trust.crt , not sure it's worth it, though it'd be easy to add. The ca-certificates package doesn't own any of the files in /etc/ssl/certs , which is slightly unfortunate, but I don't think can possibly be avoided - we don't know what the files will be so we can't ghost them all even if we wanted to. I think it's just an occupational hazard of the design where it's populated by a script. It'd be just the same if we wanted to provide a Fedora canonical CApath-style directory, I think.
any thoughts on the proposed change, kai, stef?
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.
More information and reason for this action is here:
Ping? Kai? Stef? The necessary bits made it into p11-kit a while ago, IIRC, so I think it ought to be possible to make this change for F23 now if desired.
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
Thank you for reporting this bug and we are sorry it could not be fixed.
Nothing about this has changed, which makes me wonder why I wasted all that fucking time researching it. Thanks everyone.
In general, I think this makes sense. Kai is the package maintainer for the certificates, so I don't think I'd be able to push your patch.
(In reply to Adam Williamson from comment #25)
> Nothing about this has changed, which makes me wonder why I wasted all that
> fucking time researching it. Thanks everyone.
I have more things on my plate that I can handle in the amount of time that this universe makes available to me. I don't know when I could get to it.
I'll try to work on this in december. We got a new bug report related to this, bug 1396811.
(In reply to Kai Engert (:kaie) from comment #28)
> I'll try to work on this in december. We got a new bug report related to
> this, bug 1396811.
Which December? ;) I know, we are all overloaded.
I just fell down this rabbit hole trying to get the SSL/TLS tests for a deployed-on-Debian system running on a Fedora client.
Looking at Adam's patch at https://bugzilla.redhat.com/attachment.cgi?id=980201&action=diff, the following worked for me:
$ output_dir=$(mktemp -d)
$ /usr/bin/p11-kit extract --format=pem-directory-hash --filter=ca-anchors --overwrite --purpose server-auth $output_dir
$ sudo cp -r $output_dir/* /etc/ssl/certs
Making the edit directly like this is a gross hack, and will no doubt cause problems on this particular machine when the cert bundle is updated in the future, but it confirms that Adam's proposal fixes the compatibility problem.
Given the process changes since this was last looked at, I'm guessing the next required step here would be for Adam's old diff to be converted to a PR against https://src.fedoraproject.org/rpms/ca-certificates/tree/master created using the instructions at https://docs.fedoraproject.org/en-US/ci/pull-requests/ (along with any changes needs to make the patch apply correctly)?
I mean, in theory sure, but I'm only going to spend the time on that if someone says they'll merge it. I already poured enough time into this then saw it sitting around untouched for years.