Bug 1284655 - TLS certificate validation fails with alternative chains [NEEDINFO]
TLS certificate validation fails with alternative chains
Status: CLOSED DUPLICATE of bug 1286034
Product: Fedora
Classification: Fedora
Component: glib-networking (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Matthias Clasen
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2015-11-23 14:34 EST by Kai Engert (:kaie)
Modified: 2015-11-27 05:02 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-11-27 05:02:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
kengert: needinfo? (mclasen)

Attachments (Terms of Use)
test.c (2.26 KB, text/plain)
2015-11-24 03:55 EST, Milan Crha
no flags Details
patch for glib-networking package in fedora f22 dist/git (13.83 KB, patch)
2015-11-24 07:45 EST, Kai Engert (:kaie)
no flags Details | Diff

External Trackers
Tracker ID Priority Status Summary Last Updated
GNOME Bugzilla 750457 None None None Never

  None (edit)
Description Kai Engert (:kaie) 2015-11-23 14:34:18 EST
The evolution mail clients fails to verify a server certificate, if it's necessary to use a different chain of CA certificates to find a trusted root CA certificates.

(For a general explanation of the background, you could read my explanation at
https://rt.openssl.org/Ticket/Display.html?id=3621#txn-49999 )

evolution displays the error message:

"A mail account '...@gmail.com' cannot connect, because an SSL certificate 
 for 'imap.googlemail.com' is not trusted.
 Reason: The signing authority is not known."

This can be reproduced using the following environment:
- Fedora 22
- ca-certificates version 2015.2.6
  (from https://bodhi.fedoraproject.org/updates/FEDORA-2015-6fb2c59536 )
- CA legacy configuration set to disabled
  (execute as root: ca-legacy disable
   then the following should report DISABLED: ca-legacy check

The connection works fine, if the ca-legacy configuration is set to default.

The cause must be the trust status of the following root CA:
# Issuer: OU=Equifax Secure Certificate Authority,O=Equifax,C=US
# Serial Number: 903804111 (0x35def4cf)
# Subject: OU=Equifax Secure Certificate Authority,O=Equifax,C=US
# Not Valid Before: Sat Aug 22 16:41:51 1998
# Not Valid After : Wed Aug 22 16:41:51 2018
# Fingerprint (MD5): 67:CB:9D:C0:13:24:8A:82:9B:B2:17:1E:D1:1B:EC:D4
# Fingerprint (SHA1): D2:32:09:AD:23:D3:14:23:21:74:E4:0D:7F:9D:62:13:97:86:63:3A

Upstream Mozilla has removed the "trusted for TLS servers" from this CA in version 2.6

The IMAP server is configured to send multiple CA certificates:

        Serial Number:26:b1:28:a9:52:c6:c6:49
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Google Internet Authority G2,O=Google Inc,C=US"
            Not Before: Thu Nov 12 18:49:26 2015
            Not After : Wed Feb 10 00:00:00 2016
        Subject: "CN=imap.googlemail.com,O=Google Inc,L=Mountain View,ST=California,C=US"
    Fingerprint (SHA-256): F3:B9:8F:AB:42:D4:95:F1:DA:21:5B:47:6F:4B:34:8E:9F:40:2D:89:FA:3A:27:30:C5:7E:4E:D1:CA:B7:9C:02
    Fingerprint (SHA1): 38:17:7E:0A:EF:EE:0E:14:A8:AF:9D:11:5D:92:8D:3A:40:BC:30:A5
        Serial Number: 146051 (0x23a83)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US"
            Not Before: Fri Apr 05 15:15:56 2013
            Not After : Sat Dec 31 23:59:59 2016
        Subject: "CN=Google Internet Authority G2,O=Google Inc,C=US"
    Fingerprint (SHA-256): A4:12:4F:DA:F9:CA:C7:BA:EE:1C:AB:32:E3:22:5D:74:65:00:C0:9F:3C:F3:EB:B2:53:EF:3F:BB:08:8A:FD:34
    Fingerprint (SHA1): 17:8F:7E:93:A7:4E:D7:3D:88:C2:90:42:22:0B:9A:E6:E4:B3:71:CD
        Serial Number: 1227750 (0x12bbe6)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "OU=Equifax Secure Certificate Authority,O=Equifax,C=US"
            Not Before: Tue May 21 04:00:00 2002
            Not After : Tue Aug 21 04:00:00 2018
        Subject: "CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US"
    Fingerprint (SHA-256): 3C:35:CC:96:3E:B0:04:45:13:23:D3:27:5D:05:B3:53:23:50:53:49:0D:9C:D8:37:29:A2:FA:F5:E7:CA:1C:C0
    Fingerprint (SHA1): 73:59:75:5C:6D:F9:A0:AB:C3:06:0B:CE:36:95:64:C8:EC:45:42:A3

This is a *VALID* server configuration, and a configuration that is used by many Internet hosts.
We must be able to correctly support this configuration.

As can be seen using the command line diagnostic utilities of openssl, gnutls and nss, the libraries are able to correctly find a trusted root CA, and allow the connection to the server:

- openssl s_client -tls1_2 -connect imap.googlemail.com:993
  -> Verify return code: 0 (ok)

- tstclnt -V tls1.2: -C -Db  -h imap.googlemail.com -p 993
  -> locally found issuer certificate:
     Issuer: "CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US"

- gnutls-cli -p 993 imap.googlemail.com
  -> Status: The certificate is trusted. 

I'm surprised that evolution doesn't trust the certificate.

Does evolution attempt to use its own implementation, instead of using library code to verify a certificate?

Does evolution need to set a new library flag to enable the discovery of alternative chains of trust?
Comment 1 Kai Engert (:kaie) 2015-11-23 14:36:28 EST
When testing, please ensure you NEVER click the "accept permanently" buttons when evolution reports the error, to ensure your verification environment remains at the clean default.
Comment 2 Milan Crha 2015-11-24 03:55 EST
Created attachment 1098069 [details]

Thanks for a bug report. Evolution's IMAPx (and Camel in general) uses glib's gio streams and relies on its results. If I recall correctly, the TLS gio stream uses gnutls in the background.

I extracted the relevant code from the evolution-data-server to a single file (attached). The first line contains a comment with a command line to compile and run it.

I can reproduce the issue with the test in the environment setup as you suggested above. Either the evolution-data-server is using the GIO callbacks incorrectly (unlikely, from my point of view, because before the update to the ca-certificates the same code works like a charm), or there should be done a fix on the glib/gio side, or somewhere deeper, just as you suggested above.
Comment 3 Nikos Mavrogiannopoulos 2015-11-24 04:47:05 EST
Could that be related with
https://bugzilla.redhat.com/show_bug.cgi?id=1246492 ? It seems gio reimplements certificate verification rather than using gnutls' functions.
Comment 4 Tomas Mraz 2015-11-24 05:18:14 EST
Yes, that looks like a duplicate.
Comment 5 Kai Engert (:kaie) 2015-11-24 06:47:27 EST
Milan, thank for the test attachment.

I tried a few experiments, and the results are confusing.

First, I compiled the test as you said. I can reproduce the failure using it.
I tried to debug/trace with gdb, but I failed, because the optimized binaries make it difficult to follow the execution path.

Because of that, I decided to build glib2 myself.

I cloned branch glib-2-44 of glib and installed it to /opt/evolution.
With that, the test reported TLS not available.

In addition I built glib-2-44 of glib-networking.

With that combination, to my surprise, your test no longer reports a failure.
It prints Connection 0x... error: none

Does that mean that something has been fixed in latest git already?

In addition, I've also built evolution-data-server and evolution locally (prefix /opt/evolution), following the instructions in the evolution README.

When running my local evolution build, the connection to imap.googlemail.com works fine.

However, there are other errors remaining.

I didn't mention it earlier, but the mentioned error wasn't the only one.
In addition, I have google calendars configured.
With the standard f22 evolution, I also get error messages for each of the google calendar connections, which seem to be about the same cause, because they mention a certificate for www.google.com which has been issued by the same Google intermediate CA.

Can you suggest which additional component might require a rebuild to fix the calendar TLS verification?

Should we try to suggest to the maintainer of glib* to produce rebuilt RPM packages for f22 testing?
Comment 6 Kai Engert (:kaie) 2015-11-24 06:48:58 EST
FYI, I built branch gnome-3-16 of evolution and data-server.
I think for all locally builds I used the same branches that are used by f22.
Comment 7 Kai Engert (:kaie) 2015-11-24 06:55:26 EST
I see that my environment still executes data-server processes from /usr/libexec ... I need to figure out how to make it use the builds from /opt/evolution/libexec
Comment 8 Kai Engert (:kaie) 2015-11-24 07:04:00 EST
Milan has explained to me how I run the local libexec binaries.
I need to start them first manually, source-registry first, then calendar-factory -w and addressbook-factory-w.

With that, no more certificates errors, both imap and calendar connections work.
Comment 9 Kai Engert (:kaie) 2015-11-24 07:09:50 EST
Changing component to glib-networking.

Milan pointed me to 

This commit from september seems the likely cause for the fix with the local build:

The commit points to
Comment 10 Kai Engert (:kaie) 2015-11-24 07:10:50 EST
Could you please rebuild glib-networking for Fedora 22 and later, to ensure this fix gets picked up?
Comment 11 Kai Engert (:kaie) 2015-11-24 07:43:38 EST
This is a scratch build that applies the patch.

With this build installed, the bug is fixed for me.
Comment 12 Kai Engert (:kaie) 2015-11-24 07:45 EST
Created attachment 1098181 [details]
patch for glib-networking package in fedora f22 dist/git
Comment 13 Kai Engert (:kaie) 2015-11-24 09:24:10 EST
(In reply to Nikos Mavrogiannopoulos from comment #3)
> Could that be related with
> https://bugzilla.redhat.com/show_bug.cgi?id=1246492 ? It seems gio
> reimplements certificate verification rather than using gnutls' functions.

(In reply to Tomas Mraz from comment #4)
> Yes, that looks like a duplicate.

In bug 1246492, David said: not a duplicate
Comment 14 David Woodhouse 2015-11-24 09:29:24 EST
This should be testable with the 'socket-client' test from gio, rather than having to use the Evolution stack?
Comment 15 David Woodhouse 2015-11-24 09:38:44 EST
What I said in bug 1246492 was slightly more nuanced than 'not a duplicate'.

If you retitle this bug to "glib-networking reimplements stuff it shouldn't", and fix it *properly* to use the GnuTLS functionality, then it's a duplicate.

If you're going to paper over the cracks without fixing the real underlying problem, then it isn't — bug 1246492 covers a *different* symptom of the same problem.

So actually, I think we really *should* be treating them as the same bug.
Comment 16 Kai Engert (:kaie) 2015-11-25 07:13:25 EST
(In reply to Kai Engert (:kaie) from comment #10)
> Could you please rebuild glib-networking for Fedora 22 and later, to ensure
> this fix gets picked up?

FYI, I just upgraded to Fedora 23, and it already uses a version of glib-networking that has picked up the "papering" fix, so, this specific bug doesn't occur in F23.
Comment 17 Nikos Mavrogiannopoulos 2015-11-27 04:56:07 EST
(In reply to David Woodhouse from comment #15)
> What I said in bug 1246492 was slightly more nuanced than 'not a duplicate'.
> If you retitle this bug to "glib-networking reimplements stuff it
> shouldn't", and fix it *properly* to use the GnuTLS functionality, then it's
> a duplicate.

Let's do that then. I believe we are going to have many more issues like that if we don't. I'll open a new bug and mark these as duplicates.
Comment 18 Nikos Mavrogiannopoulos 2015-11-27 05:02:45 EST

*** This bug has been marked as a duplicate of bug 1286034 ***

Note You need to log in before you can comment on or make changes to this bug.