Bug 1850512
| Summary: | ca-certificates update causes certificate verification failure | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Brian J. Murrell <brian> | ||||||||
| Component: | gnutls | Assignee: | Red Hat Crypto Team <crypto-team> | ||||||||
| Status: | NEW --- | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | low | ||||||||||
| Version: | 37 | CC: | ansasaki, avi.kivity, caillon+fedoraproject, crypto-team, dueno, fkrenzel, jorton, lucilanga, mcatanza, mcrha, nmavrogi, paul.wouters, pemensik, rhughes, rrelyea, rstrode, sandmann, tmraz, tm, zfridric | ||||||||
| Target Milestone: | --- | Keywords: | Reopened, Triaged | ||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2023-02-07 09:57:02 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Thanks for a bug report. If this happened after update of ca-certificates, then I'd say the problem is there, not in Evolution, though Evolution uses glib-networking, which uses GnuTLS instead, which had a similar problem recently: https://mail.gnome.org/archives/desktop-devel-list/2020-June/msg00000.html As it seems you can reach the site with OpenSSL with no problem, I move this to GnuTLS. I'm sorry for taking long to get to this bug. I believe this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1842178 which was fixed by gnutls-3.6.13-6.fc32. Could you please confirm the issue is fixed? Brian, just in case, make sure you restart Evolution and all of its background processes after you install the updated gnutls, thus the processes use the updated bits. One way to do that is to run from a terminal: $ evolution --force-shutdown Seems to be resolved. Thank you for the confirmation! *** This bug has been marked as a duplicate of bug 1842178 *** This is still happening on F35 with gnutls-3.7.2-2.fc35.x86_64. So perhaps not a duplicate of bug 1842178 so reopening. To be clear, all I did was to update ca-certificates and Evolution started complaining about perfectly valid certificates. What is the exact error message, please? SSL certificate for address book “[redacted]@yahoo.com : Contacts” is not trusted. Reason: The signing certificate authority is not known. As just one example. Created attachment 1846624 [details]
Screenshot of error
See attached for a screen shot of some unselectable text. I wish the certification display widget was not broken so as to not use the full size of the dialog so that you could see more of the cert, but it is what it is. Maybe you can open a ticket to fix that as an aside.
FWIW, I produced this simply by installing the ca-certificates package from Fedora 35's testing repo. A short while after installing that, the errors started showing up in Evolution. Would you be able to test with the gnutls-cli utility with debug output enabled? Something like: $ dnf install gnutls-utils $ gnutls-cli -d 10 -p imaps imap.mail.yahoo.com Created attachment 1846633 [details]
gnutls-cli output
Is there any more information that I can provide here while my evolution is in this state? I need to be able to connect to the servers that it's complaining that it cannot verify the certificates of and I keep punting it so that I can collect information for this ticket. I cannot keep punting this for many more hours or days or weeks, but I want to be able to provide you whatever information you need to resolve this as this is not the first time this happened. I have installed ca-certificates-2021.2.52-1.0.fc35.noarch and there is no better option in update-stesting and with it the gnutls-cli command also succeeds with "- Status: The certificate is trusted." I added a Yahoo! account on that machine, with OAuth2 authentication, and it showed my mail folders with no problem. I restarted the machine and then I receive "Rate limit hit" from the server, but that's nothing about the certificate. Do you have any Yahoo! certificate shown in Edit->Preferences->Certificates->Mail? I do not have any there. Indeed. I am not asserting that there is anything wrong with the ca-certifcates package so we can dismiss that option. What I am saying is that if you update ca-certificates, while evolution is still running, evolution starts returning errors about certificates for accounts/servers that were configured and working before the upgrade. The certificates are indeed valid, but evolution is complaining about them all the same. Currently evolution is complaining about: www.googleapis.com imap.gmail.com imap.mail.yahoo.com The error is "The signing certificate authority is not known." I don't believe that is actually true and gnutls-cli will probably verify that, but that is what evolution is complaining about all the same. I suspect the key to reproducing is to have a yahoo/gmail imap account configured and working and evolution running *before* you upgrade/downgrade ca-certificates. Evolution doesn't touch the certificates on its own, it uses NSS and glib-networking uses GnuTLS in the background. If the glib-networking or GnuTLS caches anything in memory and gets confused after a file change on the disk, then Evolution has just bad luck, I think. Being it so, other applications using this could be affected as well. One I can think of is Epiphany (which uses WebKitGTK). You mentioned above that the `evolution --force-shutdown` fixes the problem. It can be the clue supporting the above theory about in-memory caching. But what would it be caching that is likely to change through a ca-certificates update? The CA certs that were used in both copies (pre and post upgrade) of ca-certificates for such well known domains is not likely to change, is it? I have not yet tried `evolution --force-shutdown` on this instance of the problem as I want to make sure you have all of the information you might need while it's broken before I try that and it resolves the problem. So where do we go from here? I don't know anywhere near enough of what is going on "under the hood" to formulate a reasonable bug report against, heh, I don't even know what. Are you able to reproduce the problem at least by doing up/downgrades of ca-certificates? Does anyone need anything more from by broken instance of evolution or can I restart it so that I can connect to my IMAP mailboxes? (In reply to Brian J. Murrell from comment #17) > Are you able to reproduce the problem at least by doing up/downgrades of > ca-certificates? Unfortunately not. I did an update (`dnf update`) and restarted the machine. After that I started Evolution, which connected to a Google IMAP. The I left Evolution running and opened a terminal and downgraded the ca-certificates package. It did not change anything, I've been able to get message content from the server. Then I switched Evolution offline and online again (using menu File->Work Offline/Online). Neither that reproduced the problem. I restarted the machine (with the downgraded ca-certificates) and repeated the same, only updating the package. Neither that reproduced the problem. It appears this bug was missed in the EOL closure for F32 on 2021-05-25. If this bug still exists on supported versions, please reopen and update the version. If you cannot update the version, please needinfo the assignee. This is happening again in F37. ca-certificates was updated 45 minutes ago:
Upgrade ca-certificates-2023.2.60-1.0.fc37.noarch @updates-home
Upgraded ca-certificates-2022.2.54-5.fc37.noarch @@System
and now I am getting certificate errors from Evolution again. More info in https://gitlab.gnome.org/GNOME/evolution/-/issues/2225.
I propose to close this one and reopen it only if the resolution for the upstream bug (or its glib-networking counterpart) will require it. Simply to avoid effort duplication. By the way, this bug is filled against gnutls at the moment. Created attachment 1940418 [details]
socket.c
For what it's worth, I cannot reproduce this, no matter whether I update or downgrade ca-certificates-2023.2.60-1.0.fc37.noarch, the evolution-data-server can connect to the Yahoo! server without any restart.
I also tried with an external application (attached), which uses glib's connections exclusively, and neither that reproduces the problem. It might not be exactly the same to what libsoup does, but it mimics the 'connect' phase properly, I believe.
Indeed, as noted previously by me (comment #19), downgrade/upgrade of ca-certificates does not reproduce the behaviour here either. My current Evolution process is still popping up these warnings. I choose to temporarily accept the certificate and that lasts a while then it pops them up again. Just in case there is anything that can be extracted from a running instance of this problem. > Just in case there is anything that can be extracted from a running instance of this problem.
No idea, this is too low level for me.
Seeing it too. Downgrading ca-certificates helped. @avi.kivity It would probably be helpful if you would add your experience to the downstream bug report at https://gitlab.gnome.org/GNOME/glib-networking/-/issues/205 so that it's understood there that this is not just a one-off issue. Will do. Reading the whole discussion here and on the issue mentioned on comment#28, this is not a bug on gnutls which correctly verifies the certificate after ca-certificates update. Except that this does happen and happens repeatedly and to more than just me. So if it's not a bug in gnutls where is the bug? If this needs to be reassigned, then so be it, but CLOSED is simply not acceptable give that this continues to happen. @milan crha: Since you were the one to re-assign to gnutls, do you have any further ideas how this will get resolved? I'm moving this back to ca-certificates. It never should have moved to gnutls because evolution uses NSS,not gnutls, so any gnutls fix would not fix the issue. Let's encrypt had an intermediate cert that changed to a different root which is now expired about the time of this bug. I believe that root has been removed now. There were chainbuilding issues where the presence of that intermediate could cause an issue with some of our libraries (but NSS wasn't one of them). Let's Encrypt is now validated by ISRG Root X1, which is properly trusted in ca-certificates. What I need from the reporter is a url to the failing imap or website. (any SSL site is possible that fails should be fine). We can find out what is missing (maybe the site isn't including the correct Let's Encrypt intermediate in the SSL connection. If that's the case, we can't really add all the intermediates in ca-certificates, but you can solve our issue by importing the intermediate into your own certificate store). bob > It never should have moved to gnutls because evolution uses NSS,not gnutls, so any gnutls fix would not fix the issue.
Afaik evolution does use gnutls for TLS connections but it's well hidden behind gio and glib-networking (which dlopen's gnutls).
(In reply to Daiki Ueno from comment #33) > > It never should have moved to gnutls because evolution uses NSS,not gnutls, so any gnutls fix would not fix the issue. > > Afaik evolution does use gnutls for TLS connections but it's well hidden > behind gio and glib-networking (which dlopen's gnutls). Correct, there is used gio and glib-networking to connect to the servers, where the glib-networking is compiled with GnuTLS by default (it can do OpenSSL as well). Evolution uses NSS for some certificate work as well (email sing/encryption, aka S/MIME), though not for the connections. I mean, as long as the connection does not return "invalid certificate", the NSS part of the evolution(-data-server) should not play any role here. (In reply to Bob Relyea from comment #32) > What I need from the reporter is a url to the failing imap or website. From the upstream bug, it's caldav.calendar.yahoo.com, but the problem here is not one that can be fixed by changes in ca-certificates. The request here is for libraries that read ca-certificates to detect that the ca-certificates package has been updated and then reload certificates rather than break. For GnuTLS as used in Fedora, where the default trust store is p11-kit rather than a certificate bundle, I believe that has to happen in p11-kit, because GnuTLS probably has no way to do that itself... right? That doesn't seem to be the URL specified by the reporter. calendar.yahoo.com is validated by Digicert, but the reporter's cert is a Let's Encrypt Cert. I think we are talking different issues here, so it would be good to get the URL from the actual reporter. This is from the reporter, from the screenshots in https://gitlab.gnome.org/GNOME/glib-networking/-/issues/205. Probably Brian switched calendars during the past three years, or maybe has multiple calendars, or perhaps Yahoo uses a different chain of trust than before. (In reply to Bob Relyea from comment #32) > Let's encrypt had an intermediate cert that changed to a > different root which is now expired about the time of this bug. This issue has nothing to do with LE. It happens to very well known sites such as GMail and Yahoo!Mail when it happens. And it has been ongoing over the last few years, not just a single time when LE changed intermediate certs. > What I need from the reporter is a url to the failing imap or website. As mentioned above, very well known mail service URLs such as GMiail and Yahoo!Mail IMAP/CalDav interfaces. caldav.calendar.yahoo.com for example. > We can find out what is > missing (maybe the site isn't including the correct Let's Encrypt > intermediate in the SSL connection. Again, this has nothing to do with LE. It happens to very big sites that are paying top-dollar for high-profile SSL certs. (In reply to Bob Relyea from comment #36) > calendar.yahoo.com is validated by Digicert, but the reporter's cert is a > Let's Encrypt Cert. *One* of the reporter (me)'s certs is an LE cert. The problem happen{ed|s} to multiple certs. The other cert was for caldav.calendar.yahoo.com as described in https://gitlab.gnome.org/GNOME/glib-networking/-/issues/205. > I think we are talking different issues here, so it > would be good to get the URL from the actual reporter. I don't think we are talking different issues. We just have to realize that when this hits it can hit multiple different certs and/or providers. In the above incident it was both an LE cert and a caldav.calendar.yahoo.com. So we can't blame this all on LE. I am noting just now that both of the certs in the above ticket are short-lived (i.e. 3-month) certs. Thanks Brian, I'll reset the componet to gnutls based on yours and Daiki's comments. If the issue is multiple cert paths, then that's not a ca-certificate issue, but does belong to the base library validating the certificate. |
Description of problem: ca-certificates was updated to 2020.2.41-1.1.fc32 a couple of hours ago and since then the running copy of evolution has been rejecting some very well known CAs. Version-Release number of selected component (if applicable): evolution-3.36.3-1.fc32.x86_64 How reproducible: Unknown. It has happened before though with a previous ca-certificates update. Steps to Reproduce: 1. Start evolution 2. Update ca-certificates Actual results: Evolution starts complaining about certificates: SSL certificate for address book “[redacted]” is not trusted. Reason: The signing certificate authority is not known. Expected results: There should be no certificate errors. Additional info: Example certificate being flagged: [redacted] Identity: [redacted] Verified by: Let's Encrypt Authority X3 Expires: 2020-08-19 Subject Name CN (Common Name): [redacted] Issuer Name C (Country): US O (Organization): Let's Encrypt CN (Common Name): Let's Encrypt Authority X3 Issued Certificate Version: 3 Serial Number: [redacted] Not Valid Before: 2020-05-21 Not Valid After: 2020-08-19 openssl s_client for the same server reports: Verify return code: 0 (ok)