Bug 1042993

Ping, 

Any thought's about this ?

Dear Patrik, thanks for your detailed report, I'm investigating this issue now and will provide updates today.

A very general statement first:

We haven't updated all applications to load the trust anchors from the new locations, so it's possible that some applications don't know about your root.

Only applications that load the trust anchors from the applications we have updated already make use of the new system and will get your manually configured root CA cert, too.

I'll go through the list you have reported, and investigate which ones are unexpected (need bugfix), and which ones should get future attention to load trustanchors from the new locations (need future enhancement).

Hi Kai, 

I've done some further investigation with pkcs11_inspect and nssdb. 

This is how the default nssdb (/etc/pki/nssdb) looks like, 

$ /usr/bin/modutil -list -dbdir /etc/pki/nssdb/

> Listing of PKCS #11 Modules
> -----------------------------------------------------------
>  1. NSS Internal PKCS #11 Module
>	 slots: 2 slots attached
>	status: loaded
>
>	 slot: NSS Internal Cryptographic Services
>	token: NSS Generic Crypto Services
>
>	 slot: NSS User Private Key and Certificate Services
>	token: NSS Certificate DB


To get pkcs11_inspect to accept our certificates we need to either, 

1 ) Import them manually into nssdb with, 
$ /usr/bin/certutil -A -n "SMHIRootCA" -t "CT,C,C" -a -d /etc/pki/nssdb -i xxROOTCA.cer
$ /usr/bin/certutil -A -n "SMHIIssuingEnterpriseCA1" -t "CT,C,C" -a -d /etc/pki/nssdb -i xxintermediate.cer

2 ) Import the tls-ca-bundle.pem (which is updated by update-ca-trust command and includes the certificates above)
$ /usr/bin/certutil -A -n "tls-ca-bundle" -t "CT,C,C" -a -d /etc/pki/nssdb -i /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

3) Import the /etc/alternatives/libnssckbi.so.x86_64 (that is symlinked to /usr/lib64/pkcs11/p11-kit-trust.so) directly into nssdb with, 
$ sudo /usr/bin/modutil -force -dbdir /etc/pki/nssdb -add p11-kit-trust -libfile /etc/alternatives/libnssckbi.so.x86_64

If we go with the third alternative, which is the one that seems to make the most sense, certutil nssdb now also have following module, 

>  2. p11-kit-trust
>	library name: /etc/alternatives/libnssckbi.so.x86_64
>	 slots: 2 slots attached
>	status: loaded
>
>	 slot: /etc/pki/ca-trust/source
>	token: System Trust
>
>	 slot: /usr/share/pki/ca-trust-source
>	token: Default Trust

I was under the impression that the default setup for nssdb was to always load the libnssckbi.so (what I mean is, I didn't think I needed to manually import it into the nssdb), but maybe I'm wrong here.

Best regards, 
Patrik Martinsson,
Sweden

(In reply to Patrik from comment #0)
> 
> Both openssl/curl uses openssl, right ? 

curl uses NSS, too, in my understanding.


> > firefox https://xxxx.xxxx.xx
> > google-chrome https://xxxx.xxxx.xx
> 
> Both of the browsers use nssdb, right ?

Firefox: yes.
Chrome: I think yes.


> And by using nssdb, does that mean that they implicitly make use of the
> "libnssckbi.so" (that is linked to p11-kit-trust.so after running
> update-ca-trust enable).

Correct.


> To be honest I'm a bit confused when applications
> make us of the library and if so that is handeled by the application itself
> or if't "implicit" when using the nss-tools. I find this area somewhat
> confusing. However, the browsers are working as expected so I'm satisfied
> withe the result. 

Applications that use NSS usually use the libnssckbi module. However, an application must explicitly request to load it to get the default roots loaded.

The browsers do so. The basic NSS utilities don't load it automatically, but can be instructed to load it (by creating a new database with certutil, and by using modutil to register the module, and use that database for further work with the command line tools.)


> Here's whats *NOT* working after following the above mention method,
> 
> > gnutls-cli xxxxx.xxxx.xx
> > gnutls-cli bugzilla.redhat.com 
> - The hostname in the certificate matches 'bugzilla.redhat.com'.
> - Peer's certificate issuer is unknown
> - Peer's certificate is NOT trusted
> Interestingly enough not even bugzilla.redhat.com (or any host for that
> matter) is recognized, which makes me believe that there is something really
> fishy with tls. 
> 
> > gnutls-cli xxxxx.xxxx.xx --x509cafile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 
> > gnutls-cli bugzilla.redhat.com --x509cafile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 
> Works like a charm though. 

I confirm what you say, both scenarios.

RHEL 6.5 still uses gnutls version 2.x

The changelog for gnutls shows that default loading of trust anchors was implemented in more recent version 3.0.13

So as of today on RHEL 6.5, you still need to use the parameters you already identified.

...

> Same goes with the ldaptools*/sssd, 
> > ldapsearch -ZZ cn=xx cn -h xxxxx.xxxx.xx -d1
> - TLS: can't connect: TLS error -8179:Peer's Certificate issuer is not
> recognized..

This I cannot confirm.
It works for me using openldap-clients-2.4.23-32.el6_4.1.x86_64

Using an internal ldap server here, which also requires a corporate CA to be trusted, I get an error with standard config, but I get a successful connection after having the CA certificate added. 


> So, using TLS-based tools, it seems that they wont work unless i
> specifically specify a cafile/cadir-option. 
> Both ldap/sssd and works with the options, 
> /etc/openldap/ldap.conf,  tls_cacert =
> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
> /etc/sssd/sssd.conf, ldap_tls_cacert =
> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (this actually default to
> whatever is set in /etc/openldap/ldap.conf, but just for the clarity I
> specify it)

Ok, it's good you have this solution.


> I was just under the impression that I wouldn't
> need to specify the cacert when using this "new methodology", am I missing
> something ? 

The "new methodology" only helps for tools that had already loaded the default CA trust anchors, and which now makes it easier to integrate CAs from your own deployment.

Regarding pkcs11 and your comment 6, you're doing a great job of finding your way around!

The /etc/pki/nssdb/ had been introduced in the past as an earlier attempt to provide some systemwide NSS database consolidation.

I suspect libnssckbi had not been registered in /etc/pki/nssdb/ for compatibility purposes, to no change the behaviour for applications that don't want to load libnssckbi for whatever reason.

Don't use idea (2) from comment 6, in my opinion.

You could use idea (3) from comment 6, if you don't mind that some applications might potentially load the module twice, and that any applications accessing that directory will load it as a consequence. Although I cannot think of problems of loading that module twice, I'm not making guarantees.

Your idea (1) might be safest, with the least potential side effects.

(Note, if SMHIIssuingEnterpriseCA1 was issued by SMHIRootCA, and if your TLS server applications are configured to supply the full chain to clients (including the intermediate SMHIIssuingEnterpriseCA1), then it's likely sufficient to import SMHIRootCA, only.)

Another tought as a follow up for comment 6 and comment 9.

Since /etc/pki/nssdb is kind of universal for many applications, which might or might not want to load libnssckbi / p11-kit-trust...

I wonder if it made sense to register libnssckbi.so in /etc/pam_pkcs11/pam_pkcs11.conf which, I assume, should have also solved the issue you had demonstrated with pkcs11_inspect.

I'll ask around and will update this bug with a reference later.

> > Same goes with the ldaptools*/sssd, 
> > > ldapsearch -ZZ cn=xx cn -h xxxxx.xxxx.xx -d1
> > - TLS: can't connect: TLS error -8179:Peer's Certificate issuer is not
> > recognized..
> 
> This I cannot confirm.
> It works for me using openldap-clients-2.4.23-32.el6_4.1.x86_64
> 
> Using an internal ldap server here, which also requires a corporate CA to be
> trusted, I get an error with standard config, but I get a successful
> connection after having the CA certificate added. 

Ah, after specifying 
- TLS_CACERTDIR /etc/openldap/certs
in /etc/openldap/ldap.conf I get the behavior you describe (I guess this is because the directory contains nssdb-files and loads the libnssckbi.so)

> Regarding pkcs11 and your comment 6, you're doing a great job of finding
> your way around!
Slowly making some progress ;) 
 
> The /etc/pki/nssdb/ had been introduced in the past as an earlier attempt to
> provide some systemwide NSS database consolidation.
I see, 

> Don't use idea (2) from comment 6, in my opinion.
Ok. 
 
> You could use idea (3) from comment 6, if you don't mind that some
> applications might potentially load the module twice, and that any
> applications accessing that directory will load it as a consequence.
> Although I cannot think of problems of loading that module twice, I'm not
> making guarantees.
Well, this is the method that makes most sense to me. Since it means that every application that uses nssdb will have access to "central-system-trust" right ? 
I guess I'll find it out if i causes any trouble. 
 
> Your idea (1) might be safest, with the least potential side effects.
This is the method we have used before, but I was eager to manage our root-ca in *one* central place that basically every application could use. 
 
> (Note, if SMHIIssuingEnterpriseCA1 was issued by SMHIRootCA, and if your TLS
> server applications are configured to supply the full chain to clients
> (including the intermediate SMHIIssuingEnterpriseCA1), then it's likely
> sufficient to import SMHIRootCA, only.)
Yes, well the only reason I need to import the intermediate certificate is because pkcs11_inspect doesn't handle intermediate certificates, or at least I can't seem to get it to work.

> Another tought as a follow up for comment 6 and comment 9.
> 
> Since /etc/pki/nssdb is kind of universal for many applications, which might
> or might not want to load libnssckbi / p11-kit-trust...
> 
> I wonder if it made sense to register libnssckbi.so in
> /etc/pam_pkcs11/pam_pkcs11.conf which, I assume, should have also solved the
> issue you had demonstrated with pkcs11_inspect.
> 
> I'll ask around and will update this bug with a reference later.

Hmm, I'm not really sure how to "register libnssckbi.so" in pam_pkcs11.conf, but please let me know if you find something.

> Another tought as a follow up for comment 6 and comment 9.
> I suspect libnssckbi had not been registered in /etc/pki/nssdb/ for 
> compatibility purposes, to no change the behaviour for applications that don't
> want to load libnssckbi for whatever reason.
> 
> Since /etc/pki/nssdb is kind of universal for many applications, which might
> or might not want to load libnssckbi / p11-kit-trust...

Hmm, can you think of a reason why an application would want to use the central-nssdb but not load libnssckbi (and also break if libnssckbi was loaded) ?

> > java testxxxx443 (oracle) 
> Modifying the java-class (so it connects to our host instead of bugzilla)
> found at
> https://fedoraproject.org/wiki/Features/SharedSystemCertificates:Testing
> - The oracle-java makes use of the
> "/usr/lib/jvm/jre-x.x.x-oracle.x86_64/lib/security/cacerts"-database, and
> not the /etc/pki/ca-trust/exported/java/cacert as I was hoping for. 

Isn't java-oracle a closed source component? If so, we cannot change it to load from a different location by default.

However, I think I found a workaround for you.

The following command line works for me with a site that requires a custom root CA installed in the new root store.

java -Djavax.net.ssl.trustStore=/etc/pki/ca-trust/extracted/java/cacerts testxxxx443

> > java testxxxx443 (oracle) 
> > Modifying the java-class (so it connects to our host instead of bugzilla)
> > found at
> > https://fedoraproject.org/wiki/Features/SharedSystemCertificates:Testing
> > - The oracle-java makes use of the
> > "/usr/lib/jvm/jre-x.x.x-oracle.x86_64/lib/security/cacerts"-database, and
> > not the /etc/pki/ca-trust/exported/java/cacert as I was hoping for. 
> 
> Isn't java-oracle a closed source component? If so, we cannot change it to
> load from a different location by default.

Yes I guess, however RedHat provides the rpms, so i figure you build them as well ? 
 
> However, I think I found a workaround for you.
> The following command line works for me with a site that requires a custom
> root CA installed in the new root store.
> java -Djavax.net.ssl.trustStore=/etc/pki/ca-trust/extracted/java/cacerts
> testxxxx443
Well this is okey for me when doing tests, however I cannot change hundreds of applications to this. 


Here's what I don't understand, 

- By default it uses, $JRE-path/lib/security/cacerts as default keystore. 

So today we symlink that path to /etc/pki/ca-trust/extracted/java/cacerts which works as expected, however the cacerts gets overwritten every time there's an java-update (which is *very often*). 

I would like to have a way for us to use oracle's java with our root-ca, without having to, 
- specify any options on the command-line 
- add rootca at every update/"re-symlink" the cacert-file

*** Bug 1042992 has been marked as a duplicate of this bug. ***

(In reply to Patrik from comment #16)
> Yes I guess, however RedHat provides the rpms, so i figure you build them as
> well ? 

If you need a definitive answer, I can find out, but my guess is that we are simply packaging the binaries into an RPM.

# yum info java-1.7.0-oracle |grep License
License     : Oracle Binary Code License Agreement for the Java SE Platform


> Here's what I don't understand, 
> 
> - By default it uses, $JRE-path/lib/security/cacerts as default keystore. 
> 
> So today we symlink that path to /etc/pki/ca-trust/extracted/java/cacerts
> which works as expected, however the cacerts gets overwritten every time
> there's an java-update (which is *very often*). 
> 
> I would like to have a way for us to use oracle's java with our root-ca,
> without having to, 
> - specify any options on the command-line 
> - add rootca at every update/"re-symlink" the cacert-file

I think this should be filed as an enhancement request against the oracle-java package.

I could think of a solution implemented as part of the oracle-java RPM package scripts, which are being executed at installation time.

I'll file a separate bug and cc you.

> I'll file a separate bug and cc you.

Filed as bug 1056224.

(In reply to Patrik from comment #14)
> Hmm, can you think of a reason why an application would want to use the
> central-nssdb but not load libnssckbi (and also break if libnssckbi was
> loaded) ?

Not right now. I might be overly cautious.

>I think this should be filed as an enhancement request against the oracle-java >package.
>
>I could think of a solution implemented as part of the oracle-java RPM package >scripts, which are being executed at installation time.
>
I'll file a separate bug and cc you.
> > I'll file a separate bug and cc you.
> 
> Filed as bug 1056224.

This is great work, I would really like to see this. 
I don't know how "other companies" that uses oracle's java does this without any "manual steps" at every update.

I'm resolving this bug as worksforme.

I believe we have identified workarounds for the individual components, and file the separate enhancement proposal.

If there are other TODO items, they should be filed against the individual packages that aren't using the shared store yet.

If you disagree, please add another comment. Thanks.