Bug 1850512 - ca-certificates update causes certificate verification failure
Summary: ca-certificates update causes certificate verification failure
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: gnutls
Version: 40
Hardware: x86_64
OS: Linux
low
high
Target Milestone: ---
Assignee: Red Hat Crypto Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2242796 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-24 12:33 UTC by Brian J. Murrell
Modified: 2024-02-15 22:53 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-02-07 09:57:02 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Screenshot of error (46.76 KB, image/png)
2021-12-16 16:27 UTC, Brian J. Murrell
no flags Details
gnutls-cli output (39.97 KB, text/plain)
2021-12-16 18:39 UTC, Brian J. Murrell
no flags Details
socket.c (3.40 KB, text/plain)
2023-01-25 14:03 UTC, Milan Crha
no flags Details
soup.c (3.82 KB, text/plain)
2023-10-13 02:43 UTC, Milan Crha
no flags Details


Links
System ID Private Priority Status Summary Last Updated
GNOME Gitlab GNOME glib-networking issues 205 0 None closed Evolution starts hitting certificate errors when ca-certificates is updated 2023-01-26 12:47:13 UTC
Red Hat Issue Tracker FC-355 0 None None None 2021-12-16 14:36:14 UTC

Description Brian J. Murrell 2020-06-24 12:33:31 UTC
Description of problem:
ca-certificates was updated to 2020.2.41-1.1.fc32 a couple of hours ago and since then the running copy of evolution has been rejecting some very well known CAs.

Version-Release number of selected component (if applicable):
evolution-3.36.3-1.fc32.x86_64

How reproducible:
Unknown.  It has happened before though with a previous ca-certificates update.

Steps to Reproduce:
1. Start evolution
2. Update ca-certificates

Actual results:
Evolution starts complaining about certificates:
SSL certificate for address book “[redacted]” is not trusted.

Reason: The signing certificate authority is not known.

Expected results:
There should be no certificate errors.

Additional info:
Example certificate being flagged:

[redacted]
Identity: [redacted]
Verified by: Let's Encrypt Authority X3
Expires: 2020-08-19

Subject Name
CN (Common Name):	[redacted]

Issuer Name
C (Country):	US
O (Organization):	Let's Encrypt
CN (Common Name):	Let's Encrypt Authority X3

Issued Certificate
Version:	3
Serial Number:	[redacted]
Not Valid Before:	2020-05-21
Not Valid After:	2020-08-19

openssl s_client for the same server reports:

    Verify return code: 0 (ok)

Comment 1 Milan Crha 2020-06-24 14:03:56 UTC
Thanks for a bug report. If this happened after update of ca-certificates, then I'd say the problem is there, not in Evolution, though Evolution uses glib-networking, which uses GnuTLS instead, which had a similar problem recently:
https://mail.gnome.org/archives/desktop-devel-list/2020-June/msg00000.html

As it seems you can reach the site with OpenSSL with no problem, I move this to GnuTLS.

Comment 2 Anderson Sasaki 2020-07-03 08:41:34 UTC
I'm sorry for taking long to get to this bug. I believe this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1842178 which was fixed by gnutls-3.6.13-6.fc32.

Could you please confirm the issue is fixed?

Comment 3 Milan Crha 2020-07-03 09:10:18 UTC
Brian, just in case, make sure you restart Evolution and all of its background processes after you install the updated gnutls, thus the processes use the updated bits. One way to do that is to run from a terminal:

   $ evolution --force-shutdown

Comment 4 Brian J. Murrell 2020-07-06 15:03:32 UTC
Seems to be resolved.

Comment 5 Anderson Sasaki 2020-07-07 07:41:20 UTC
Thank you for the confirmation!

*** This bug has been marked as a duplicate of bug 1842178 ***

Comment 6 Brian J. Murrell 2021-12-16 14:31:20 UTC
This is still happening on F35 with gnutls-3.7.2-2.fc35.x86_64.

So perhaps not a duplicate of bug 1842178 so reopening.

To be clear, all I did was to update ca-certificates and Evolution started complaining about perfectly valid certificates.

Comment 7 Milan Crha 2021-12-16 15:53:50 UTC
What is the exact error message, please?

Comment 8 Brian J. Murrell 2021-12-16 16:09:02 UTC
SSL certificate for address book “[redacted]@yahoo.com : Contacts” is not trusted.

Reason: The signing certificate authority is not known.

As just one example.

Comment 9 Brian J. Murrell 2021-12-16 16:27:11 UTC
Created attachment 1846624 [details]
Screenshot of error

See attached for a screen shot of some unselectable text.  I wish the certification display widget was not broken so as to not use the full size of the dialog so that you could see more of the cert, but it is what it is.  Maybe you can open a ticket to fix that as an aside.

Comment 10 Brian J. Murrell 2021-12-16 16:39:41 UTC
FWIW, I produced this simply by installing the ca-certificates package from Fedora 35's testing repo.

A short while after installing that, the errors started showing up in Evolution.

Comment 11 Daiki Ueno 2021-12-16 18:17:47 UTC
Would you be able to test with the gnutls-cli utility with debug output enabled? Something like:

$ dnf install gnutls-utils
$ gnutls-cli -d 10 -p imaps imap.mail.yahoo.com

Comment 12 Brian J. Murrell 2021-12-16 18:39:54 UTC
Created attachment 1846633 [details]
gnutls-cli output

Comment 13 Brian J. Murrell 2021-12-16 20:32:38 UTC
Is there any more information that I can provide here while my evolution is in this state?

I need to be able to connect to the servers that it's complaining that it cannot verify the certificates of and I keep punting it so that I can collect information for this ticket.  I cannot keep punting this for many more hours or days or weeks, but I want to be able to provide you whatever information you need to resolve this as this is not the first time this happened.

Comment 14 Milan Crha 2021-12-17 07:30:34 UTC
I have installed ca-certificates-2021.2.52-1.0.fc35.noarch and there is no better option in update-stesting and with it the gnutls-cli command also succeeds with "- Status: The certificate is trusted."

I added a Yahoo! account on that machine, with OAuth2 authentication, and it showed my mail folders with no problem. I restarted the machine and then I receive "Rate limit hit" from the server, but that's nothing about the certificate.

Do you have any Yahoo! certificate shown in Edit->Preferences->Certificates->Mail? I do not have any there.

Comment 15 Brian J. Murrell 2021-12-17 11:48:16 UTC
Indeed.  I am not asserting that there is anything wrong with the ca-certifcates package so we can dismiss that option.

What I am saying is that if you update ca-certificates, while evolution is still running, evolution starts returning errors about certificates for accounts/servers that were configured and working before the upgrade.  The certificates are indeed valid, but evolution is complaining about them all the same.

Currently evolution is complaining about:

www.googleapis.com
imap.gmail.com
imap.mail.yahoo.com

The error is "The signing certificate authority is not known."

I don't believe that is actually true and gnutls-cli will probably verify that, but that is what evolution is complaining about all the same.

I suspect the key to reproducing is to have a yahoo/gmail imap account configured and working and evolution running *before* you upgrade/downgrade ca-certificates.

Comment 16 Milan Crha 2021-12-17 12:44:46 UTC
Evolution doesn't touch the certificates on its own, it uses NSS and glib-networking uses GnuTLS in the background. If the glib-networking or GnuTLS caches anything in memory and gets confused after a file change on the disk, then Evolution has just bad luck, I think. Being it so, other applications using this could be affected as well. One I can think of is Epiphany (which uses WebKitGTK).

You mentioned above that the `evolution --force-shutdown` fixes the problem. It can be the clue supporting the above theory about in-memory caching.

Comment 17 Brian J. Murrell 2021-12-17 12:55:57 UTC
But what would it be caching that is likely to change through a ca-certificates update?  The CA certs that were used in both copies (pre and post upgrade) of ca-certificates for such well known domains is not likely to change, is it?

I have not yet tried `evolution --force-shutdown` on this instance of the problem as I want to make sure you have all of the information you might need while it's broken before I try that and it resolves the problem.

So where do we go from here?  I don't know anywhere near enough of what is going on "under the hood" to formulate a reasonable bug report against, heh, I don't even know what.

Are you able to reproduce the problem at least by doing up/downgrades of ca-certificates?

Comment 18 Brian J. Murrell 2021-12-17 15:26:09 UTC
Does anyone need anything more from by broken instance of evolution or can I restart it so that I can connect to my IMAP mailboxes?

Comment 19 Milan Crha 2022-01-03 13:48:45 UTC
(In reply to Brian J. Murrell from comment #17)
> Are you able to reproduce the problem at least by doing up/downgrades of
> ca-certificates?

Unfortunately not. I did an update (`dnf update`) and restarted the machine. After that I started Evolution, which connected to a Google IMAP. The I left Evolution running and opened a terminal and downgraded the ca-certificates package. It did not change anything, I've been able to get message content from the server. Then I switched Evolution offline and online again (using menu File->Work Offline/Online). Neither that reproduced the problem. I restarted the machine (with the downgraded ca-certificates) and repeated the same, only updating the package. Neither that reproduced the problem.

Comment 20 Ben Cotton 2022-07-11 19:24:35 UTC
It appears this bug was missed in the EOL closure for F32 on 2021-05-25. If this bug still exists on supported versions, please reopen and update the version. If you cannot update the version, please needinfo the assignee.

Comment 21 Brian J. Murrell 2023-01-25 12:30:19 UTC
This is happening again in F37.  ca-certificates was updated 45 minutes ago:

    Upgrade  ca-certificates-2023.2.60-1.0.fc37.noarch                    @updates-home
    Upgraded ca-certificates-2022.2.54-5.fc37.noarch                      @@System

and now I am getting certificate errors from Evolution again.  More info in https://gitlab.gnome.org/GNOME/evolution/-/issues/2225.

Comment 22 Milan Crha 2023-01-25 12:45:54 UTC
I propose to close this one and reopen it only if the resolution for the upstream bug (or its glib-networking counterpart) will require it. Simply to avoid effort duplication.

Comment 23 Milan Crha 2023-01-25 12:47:06 UTC
By the way, this bug is filled against gnutls at the moment.

Comment 24 Milan Crha 2023-01-25 14:03:19 UTC
Created attachment 1940418 [details]
socket.c

For what it's worth, I cannot reproduce this, no matter whether I update or downgrade ca-certificates-2023.2.60-1.0.fc37.noarch, the evolution-data-server can connect to the Yahoo! server without any restart.

I also tried with an external application (attached), which uses glib's connections exclusively, and neither that reproduces the problem. It might not be exactly the same to what libsoup does, but it mimics the 'connect' phase properly, I believe.

Comment 25 Brian J. Murrell 2023-01-25 14:12:26 UTC
Indeed, as noted previously by me (comment #19), downgrade/upgrade of ca-certificates does not reproduce the behaviour here either.

My current Evolution process is still popping up these warnings.  I choose to temporarily accept the certificate and that lasts a while then it pops them up again.  Just in case there is anything that can be extracted from a running instance of this problem.

Comment 26 Milan Crha 2023-01-25 14:19:23 UTC
> Just in case there is anything that can be extracted from a running instance of this problem.

No idea, this is too low level for me.

Comment 27 Avi Kivity 2023-01-26 12:18:44 UTC
Seeing it too. Downgrading ca-certificates helped.

Comment 28 Brian J. Murrell 2023-01-26 12:47:13 UTC
@avi.kivity It would probably be helpful if you would add your experience to the downstream bug report at https://gitlab.gnome.org/GNOME/glib-networking/-/issues/205 so that it's understood there that this is not just a one-off issue.

Comment 29 Avi Kivity 2023-01-26 13:54:29 UTC
Will do.

Comment 30 Anderson Sasaki 2023-02-07 09:57:02 UTC
Reading the whole discussion here and on the issue mentioned on comment#28, this is not a bug on gnutls which correctly verifies the certificate after ca-certificates update.

Comment 31 Brian J. Murrell 2023-02-07 22:19:41 UTC
Except that this does happen and happens repeatedly and to more than just me.  So if it's not a bug in gnutls where is the bug?  If this needs to be reassigned, then so be it, but CLOSED is simply not acceptable give that this continues to happen.

@milan crha: Since you were the one to re-assign to gnutls, do you have any further ideas how this will get resolved?

Comment 32 Bob Relyea 2023-02-08 00:47:07 UTC
I'm moving this back to ca-certificates. It never should have moved to gnutls because evolution uses NSS,not gnutls, so any gnutls fix would not fix the issue. Let's encrypt had an intermediate cert that changed to a different root which is now expired about the time of this bug. I believe that root has been removed now. There were chainbuilding issues where the presence of that intermediate could cause an issue with some of our libraries (but NSS wasn't one of them). Let's Encrypt is now validated by ISRG Root X1, which is properly trusted in ca-certificates.

What I need from the reporter is a url to the failing imap or website. (any SSL site is possible that fails should be fine). We can find out what is missing (maybe the site isn't including the correct Let's Encrypt intermediate in the SSL connection. If that's the case, we can't really add all the intermediates in ca-certificates, but you can solve our issue by importing the intermediate into your own certificate store).

bob

Comment 33 Daiki Ueno 2023-02-08 07:32:03 UTC
> It never should have moved to gnutls because evolution uses NSS,not gnutls, so any gnutls fix would not fix the issue.

Afaik evolution does use gnutls for TLS connections but it's well hidden behind gio and glib-networking (which dlopen's gnutls).

Comment 34 Milan Crha 2023-02-08 07:58:31 UTC
(In reply to Daiki Ueno from comment #33)
> > It never should have moved to gnutls because evolution uses NSS,not gnutls, so any gnutls fix would not fix the issue.
> 
> Afaik evolution does use gnutls for TLS connections but it's well hidden
> behind gio and glib-networking (which dlopen's gnutls).

Correct, there is used gio and glib-networking to connect to the servers, where the glib-networking is compiled with GnuTLS by default (it can do OpenSSL as well).

Evolution uses NSS for some certificate work as well (email sing/encryption, aka S/MIME), though not for the connections. I mean, as long as the connection does not return "invalid certificate", the NSS part of the evolution(-data-server) should not play any role here.

Comment 35 Michael Catanzaro 2023-02-08 20:28:27 UTC
(In reply to Bob Relyea from comment #32)
> What I need from the reporter is a url to the failing imap or website.

From the upstream bug, it's caldav.calendar.yahoo.com, but the problem here is not one that can be fixed by changes in ca-certificates. The request here is for libraries that read ca-certificates to detect that the ca-certificates package has been updated and then reload certificates rather than break. For GnuTLS as used in Fedora, where the default trust store is p11-kit rather than a certificate bundle, I believe that has to happen in p11-kit, because GnuTLS probably has no way to do that itself... right?

Comment 36 Bob Relyea 2023-02-08 23:00:49 UTC
That doesn't seem to be the URL specified by the reporter. calendar.yahoo.com is validated by Digicert, but the reporter's cert is a Let's Encrypt Cert. I think we are talking different issues here, so it would be good to get the URL from the actual reporter.

Comment 37 Michael Catanzaro 2023-02-08 23:31:59 UTC
This is from the reporter, from the screenshots in https://gitlab.gnome.org/GNOME/glib-networking/-/issues/205. Probably Brian switched calendars during the past three years, or maybe has multiple calendars, or perhaps Yahoo uses a different chain of trust than before.

Comment 38 Brian J. Murrell 2023-02-09 20:13:53 UTC
(In reply to Bob Relyea from comment #32)
> Let's encrypt had an intermediate cert that changed to a
> different root which is now expired about the time of this bug.

This issue has nothing to do with LE.  It happens to very well known sites such as GMail and Yahoo!Mail when it happens.  And it has been ongoing over the last few years, not just a single time when LE changed intermediate certs.

> What I need from the reporter is a url to the failing imap or website.

As mentioned above, very well known mail service URLs such as GMiail and Yahoo!Mail IMAP/CalDav interfaces.  
caldav.calendar.yahoo.com for example.

> We can find out what is
> missing (maybe the site isn't including the correct Let's Encrypt
> intermediate in the SSL connection.

Again, this has nothing to do with LE.  It happens to very big sites that are paying top-dollar for high-profile SSL certs.

(In reply to Bob Relyea from comment #36)
> calendar.yahoo.com is validated by Digicert, but the reporter's cert is a
> Let's Encrypt Cert.

*One* of the reporter (me)'s certs is an LE cert.  The problem happen{ed|s} to multiple certs.  The other cert was for caldav.calendar.yahoo.com as described in https://gitlab.gnome.org/GNOME/glib-networking/-/issues/205.

> I think we are talking different issues here, so it
> would be good to get the URL from the actual reporter.

I don't think we are talking different issues.  We just have to realize that when this hits it can hit multiple different certs and/or providers.  In the above incident it was both an LE cert and a caldav.calendar.yahoo.com.  So we can't blame this all on LE.

I am noting just now that both of the certs in the above ticket are short-lived (i.e. 3-month) certs.

Comment 39 Bob Relyea 2023-02-09 22:36:50 UTC
Thanks Brian, I'll reset the componet to gnutls based on yours and Daiki's comments.

If the issue is multiple cert paths, then that's not a ca-certificate issue, but does belong to the base library validating the certificate.

Comment 40 Brian J. Murrell 2023-10-09 15:27:51 UTC
This has now just happened yet again with the:

* Tue Aug 01 2023 Robert Relyea <rrelyea> - 2023.2.60_v7.0.306-1.0
- Update to CKBI 2.60_v7.0.306 from NSS 3.91

update of ca-certificates that happened Sun 08 Oct 2023 06:02:17 AM EDT on my system.

I now have a bunch of Evolution dialogs telling me it cannot connect to imap.mail.yahoo.com and imap.gmail.com due to "The signing certificate authority is not known.".

This happens to me *every* time ca-certificates is updated.

I wonder if/why others who run Evolution constantly, not stopping and starting at the end/beginning of each day is not running into this.

@mcrha Do you not see this problem?  Do you run Evolution more-or-less constantly on a Fedora system by any chance?

Comment 41 Bob Relyea 2023-10-09 15:47:01 UTC
Brian, which version of fedora are you running?

Thanks,
bob

Comment 42 Milan Crha 2023-10-09 16:49:55 UTC
When I work on Evolution I stop it many times a day. I also update the machine less frequently, not every day/week. I'm not a usual user in this sense, I'm afraid.

Comment 43 Bob Relyea 2023-10-09 20:09:36 UTC
Maybe I should 'recommend desktop restart' on ca-certificates updates.... does restarting evolution cause the issue to clear? If it doesn't does shutting down evolution then running update-ca-trust (as root) cause the issue to clear?

Comment 44 Michael Catanzaro 2023-10-09 20:23:20 UTC
Hi Daiki, can you answer my question from comment #35 please?

Comment 45 Bob Relyea 2023-10-10 00:07:32 UTC
*** Bug 2242796 has been marked as a duplicate of this bug. ***

Comment 46 Brian J. Murrell 2023-10-10 01:29:12 UTC
(In reply to Bob Relyea from comment #41)
> Brian, which version of fedora are you running?
> 
> Thanks,
> bob

Always latest supported.  F38 at this time.

Comment 47 Brian J. Murrell 2023-10-10 01:37:24 UTC
(In reply to Bob Relyea from comment #43)
> Maybe I should 'recommend desktop restart' on ca-certificates updates....

That would most likely be overkill and would sadly start to make Linux look/act more like Windows where you are (unnecessarily) rebooting just about every day due to updates.

> does restarting evolution cause the issue to clear?

From my recollection (and probably a back-reading of this ticket) It usually does, yes.  But even that seems a pretty drastic solution for a problem being caused just because CA certs are being updated.  Why can't this be transparent?  It really seems like it should be.

Looking at this from a different perspective… did the Yahoo or GMail CA certs even change in this most recent ca-certificates update?  If not, why is any of this even a problem?

> If it doesn't does
> shutting down evolution then running update-ca-trust (as root) cause the
> issue to clear?

As above, probably not necessary.

I have left my Evolution in the state of rejecting what are otherwise perfectly valid certs.  I can take the two steps above (restarting Evolution and if that doesn't resolve the problem, update-ca-trust) and see which one is really necessary to work-around the Evolution issue if you want -- and if there is no further information needed while Evolution is in this state.  Just let me know if there is any information I can grab while Evolution is still complaining about invalid CA certs.  If not, I will try the above suggestions and report.

Comment 48 Brian J. Murrell 2023-10-10 01:39:45 UTC
It's also interesting that Evolution seems to be gobbling up memory while in this state of requesting that I either "Cancel/Reject/Accept Temporarily/Accept Permanently" these "invalid" certs.

I might come back and find this machine OOMing in the morning due to Evolution's memory use in this state:

MiB Mem :  15850.7 total,   1462.4 free,   7779.4 used,   6608.9 buff/cache
MiB Swap:   8192.0 total,   1413.3 free,   6778.7 used.   6515.2 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                              
 176070 brian     20   0  104.6g   1.0g  45928 S   2.6   6.7 554:37.50 evolution                                                                            

Hopefully not.

Comment 49 John Dodson 2023-10-10 07:19:12 UTC
Sorry Bob I was incommunicado... (& my bug 2242796 report was closed)
(I take it you don't need the cert details now? - fixed for me now & I have
a little more understanding having read this bug...)

I needed to sighup all the evolution hangers on as well as restarting evolution.
1625562 tty2     Sl+    0:00 /usr/libexec/evolution-data-server/evolution-alarm-notify
1625959 ?        SLsl   0:01 /usr/libexec/evolution-source-registry
1626298 ?        Ssl    0:00 /usr/libexec/evolution-calendar-factory
1626423 ?        Ssl    0:02 /usr/libexec/evolution-addressbook-factory

Just an evolution restart did not fix it.
One of the certs involved was Apple (CardDAV/Contacts).

I'm not sure which sighup resolved it. (A logout/login probably would have too)

I did not, update-ca-trust

# dnf list ca-certificates
Last metadata expiration check: 0:03:06 ago on Tue 10 Oct 2023 17:21:56.
Installed Packages
ca-certificates.noarch           2023.2.60_v7.0.306-1.0.fc38            @updates

/var/log/dnf.rpm.log:2023-10-09T01:00:17+1100 SUBDEBUG Upgraded: ca-certificates-2023.2.60-2.fc38.noarch

Seems to be a dodgy link in the Evolution(ary) chain following ca-cert update ;-)

Comment 50 John Dodson 2023-10-10 07:31:13 UTC
I do run evolution constantly, but will generally reboot if there's a new kernel.

Comment 51 Milan Crha 2023-10-10 07:35:51 UTC
> I'm not sure which sighup resolved it. (A logout/login probably would have too)

You can use `evolution --force-shutdown` to restart all related processes (in a correct order), including those from the evolution-data-server.

> Seems to be a dodgy link in the Evolution(ary) chain following ca-cert update ;-)

I know this is easy to reproduce in the Evolution, but, please, note the problem is not in the Evolution, it does not touch the certificates directly other than though glib-networking and NSS/NSPR (for S/MIME). The glib-netwokring uses gnutls backend in Fedora, thus it's gnutls (possibly?) caching/linking to the cert files and when those files are updated, the links are obsoleted and the cert verification fails - that's only my wild guess on the problem, it can be anywhere else. My point here is that what one calls "Evolution problem" is in fact "gnutls problem". Evolution is only a place where the problem is reproduced most easily.

In fact, you might see the same problem in gnome-contacts too, because it reads the CardDAV contacts through the evolution-data-server (the same as Evolution). Whether gnome-contacts passes the error to the GUI or only prints it on the terminal or hides it completely I do not know.

Comment 52 Daiki Ueno 2023-10-10 07:39:47 UTC
(In reply to Michael Catanzaro from comment #35)
> (In reply to Bob Relyea from comment #32)
> > What I need from the reporter is a url to the failing imap or website.
> 
> From the upstream bug, it's caldav.calendar.yahoo.com, but the problem here
> is not one that can be fixed by changes in ca-certificates. The request here
> is for libraries that read ca-certificates to detect that the
> ca-certificates package has been updated and then reload certificates rather
> than break. For GnuTLS as used in Fedora, where the default trust store is
> p11-kit rather than a certificate bundle, I believe that has to happen in
> p11-kit, because GnuTLS probably has no way to do that itself... right?

Yes, that is correct, though I'm not sure handling it in the library (p11-kit) is worthwhile, as it may imply various corner-cases, such as when the calling library (GnuTLS) is already doing chain verification against the trust store, etc. Therefore I would say suggesting a reboot would be a simpler solution. That should be possible through the Bodhi UI, when submitting a ca-certificates update.

Comment 53 John Dodson 2023-10-10 11:50:29 UTC
> ...note the problem is not in the Evolution

Yes, noted.

Comment 54 Brian J. Murrell 2023-10-10 14:05:59 UTC
(In reply to Daiki Ueno from comment #52)
> Therefore I would say suggesting a reboot would be a
> simpler solution.

But a more disruptive (non-)necessity.  Maybe not for people who only open a browser (which hopefully is saving state like Chrom{e,imum}, Firefox, etc. do) and Evolution and that's the sum total of what they do on their Linux machine.  But for people who open a dozen terminal windows to many different machines, etc. having to reboot is quite disruptive.

Moreover, I think asking people to reboot for such things makes Linux look more finicky than it is.  One of the frustrations I hear from many Windows users is the near-constant need to reboot after updates.  I'd hate for Linux to get that frustrating to use.

Comment 55 Michael Catanzaro 2023-10-10 14:16:53 UTC
Data point: Evolution depends on WebKitGTK, and that is very likely to start crashing if you update while it's running. Maybe I should start marking those updates as "requires logout."

Comment 56 Brian J. Murrell 2023-10-10 14:43:59 UTC
Even after doing an `evolution --force-shutdown` and restarting Evolution (and observing that all of the evolution processes are "new") I am still getting error "banner"s (not dialogs like before I restarted all of Evolution) reporting:

Failed to connect calendar “[redacted]@gmail.com : [redacted]”

Failed to obtain an access token for “[redacted]”: Failed to refresh access token (g-tls-error-quark, 2): Unacceptable TLS certificate

Comment 57 Brian J. Murrell 2023-10-10 14:45:23 UTC
(In reply to Michael Catanzaro from comment #55)
> Data point: Evolution depends on WebKitGTK, and that is very likely to start
> crashing if you update while it's running.

I don't recall *ever* having that happen.  So while it might be a theoretically correct issue, it doesn't seem to manifest in real-life.

> Maybe I should start marking
> those updates as "requires logout."

Adding yet more unnecessary reboot-after-update frustration IMHO.

Comment 58 Michael Catanzaro 2023-10-10 15:24:12 UTC
(In reply to Brian J. Murrell from comment #57)
> I don't recall *ever* having that happen.  So while it might be a
> theoretically correct issue, it doesn't seem to manifest in real-life.

Alas, I receive the bug reports. :P

Comment 59 Bob Relyea 2023-10-10 16:20:05 UTC
OK, I was worried that gnutls was causing the update to fail. (which is why I was asking if restarting evolution was sufficient, or if you need reinstall ca-certificates to clear the issue). It's likely that gnutls has a file handle open and p11-kit (called from ca-certificates) deletes that file handle. If gnu-tls were simply caching, then a ca-certificates update wouldn't cause previously accepted certs to suddenly fail.

Comment 60 Milan Crha 2023-10-10 16:28:34 UTC
(In reply to Brian J. Murrell from comment #56)
> Failed to connect calendar “[redacted]@gmail.com : [redacted]”
> 
> Failed to obtain an access token for “[redacted]”: Failed to refresh access
> token (g-tls-error-quark, 2): Unacceptable TLS certificate

That account is configured in GNOME Online Accounts, right? The token is refreshed on the goa-daemon side, thus you'd need to restart/kill also the goa-daemon process.

Comment 61 Daiki Ueno 2023-10-12 12:25:44 UTC
(In reply to Bob Relyea from comment #59)
> OK, I was worried that gnutls was causing the update to fail. (which is why
> I was asking if restarting evolution was sufficient, or if you need
> reinstall ca-certificates to clear the issue). It's likely that gnutls has a
> file handle open and p11-kit (called from ca-certificates) deletes that file
> handle. If gnu-tls were simply caching, then a ca-certificates update
> wouldn't cause previously accepted certs to suddenly fail.

Sorry Bob, I don't get it. As GnuTLS relies on p11-kit (p11-kit-trust.so) to access ca-certificates, it doesn't open any file handle by itself. The current logic is as follows:
- When C_FindObjectsInit is called for the first time for a new session, it scans the configured directories (/usr/share/ca-trust-source, etc.) and loads the content in memory https://github.com/p11-glue/p11-kit/blob/e5f0be33f5d39ad117e1cb84de5a8ac1ebe75548/trust/module.c#L1210
- The GnuTLS verification code, as far as I reviewed, does not reuse a session handle, but always opens/closes a new one, e.g., https://gitlab.com/gnutls/gnutls/-/blob/9239f7a2dcb92e5565c33a5f392c29e790c39f00/lib/x509/verify.c#L1275 where the first argument is a PKCS#11 URI

I also tried the reproducer at comment 24 but couldn't reproduce the issue. I also wrote my own program using the GnuTLS API, which repeatedly verifies a certificate chain, but even if I reinstall ca-certificates while it's running, verification still works. Perhaps the original issue could only happen around events like https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/.

Comment 62 Michael Catanzaro 2023-10-12 13:14:50 UTC
Note: glib-networking calls gnutls_x509_trust_list_add_system_trust() just once and then reuses that gnutls_x509_trust_list_t for all future certificate verification. That might be relevant.

Comment 63 Daiki Ueno 2023-10-12 13:29:30 UTC
(In reply to Michael Catanzaro from comment #62)
> Note: glib-networking calls gnutls_x509_trust_list_add_system_trust() just
> once and then reuses that gnutls_x509_trust_list_t for all future
> certificate verification. That might be relevant.

That's what my reproducer (and the one at comment 24) does: https://gitlab.com/dueno/repeat-connect/-/blob/2e8b3cdf4f9c8796107336accc23f552d82c4413/ex-client-x509.c#L145

Comment 64 Bob Relyea 2023-10-12 16:47:16 UTC
Hi Daiki, I was just guessing based on what was being described. Clearly something else is happening then. Michael, what imap site are you talking to?
Daiki, Imap does have a different characteristic than https. The latter stays connected and doesn't send eof (I ran into this when trying to ping the sites with vfyserv). So maybe the reproducer needs to  open the TLS connection and then do multiple reads and writes before closing the connection... but then you never get to the point where you are validating the certificate... also if you have connected to the server, should you be doing SSL quick restart and dispensing with the certificate validation anyway? There are a lot of moving parts here (sigh).

Comment 65 Milan Crha 2023-10-13 02:43:58 UTC
Created attachment 1993699 [details]
soup.c

I rewrote the test program to use libsoup3, quite close to what evolution-data-server does, but I still cannot reproduce the problem with it myself, either I downgrade or upgrade ca-certificates on Fedora 38 and 39. My disk partition is Ext4, if it makes any difference.

Brian, could you try to reproduce the problem with this test program, please?

You'd need to install devel packages:

   sudo dnf install gcc libsoup3-devel

then run the command from the first line of the soup.c file, which will also run it and then press "c<Enter>" to download data from the Yahoo! server. You can "connect" multiple times. Keep the test app waiting and on a different terminal update/downgrade the ca-certificates package:

   sudo dnf downgrade ca-certificates

or

   sudo dnf update ca-certificates

and after that switch back to the soup test app and press "c<enter>" again, which will connect to the server and download data from it - it should fail on the certificate error now. The "a<enter>" is for abort of the SoupSession, which, I guess, also disconnects any idle connections, thus a new "c<enter>" will make a new connection to the server. If that will not re-connect, then simple quit of the test app and run it should correct the problem, supposing you can reproduce it.

Comment 66 Aoife Moloney 2023-11-23 00:03:28 UTC
This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 67 Brian J. Murrell 2023-11-24 15:36:33 UTC
Where has the Version: field disappeared to in this Bugzilla instance?  None of the bugs I have open have that field showing and yet I get these e-mails saying that the 'version' is set to '37'.

This is another ticket that needs it's version moved to rawhide as this continues to happen in F39.

Comment 68 Aoife Moloney 2024-02-15 22:53:36 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 40 development cycle.
Changing version to 40.


Note You need to log in before you can comment on or make changes to this bug.