Bug 2542395

SOLVED!

Re-reading the output from perf() I suspected an issue with fontconfig.

I then killed both evolution and WebKitWebProces processes:

  kill $(pgrep evolution) $(pgrep WebKitWebProces)

and then destroyed and recreated the fontconfig cache:

  rm -rf .cache/fontconfig
  fc-cache -rv

and restarted evolution: this solved the issue instantly, all messages could now be displayed.

Looks like some kind of corruption in .cache/fontconfig got WebKitWebProces processes stuck in infinite loops.

Well this is actually and unfortunately not solved. Switching to a couple other folders led to the exact same result: no messages are displayed anymore.

  sudo perf top -p$(pgrep WebKitWebProces) --sort dso,symbol

reports:

  Samples: 359K of event 'cpu/cycles/P', 4000 Hz, Event count (approx.): 57494072256 lost: 0/0 drop: 0/0
  Overhead  Shared Object                       Symbol
    33,50%  libfontconfig.so.1.16.0             [.] FcPatternObjectPosition.lto_priv.0
    21,30%  libc.so.6                           [.] __strcmp_sse42
    14,48%  libfontconfig.so.1.16.0             [.] FcPatternGetString
    11,09%  libfontconfig.so.1.16.0             [.] FcObjectTypeLookup.lto_priv.0
     7,98%  libfontconfig.so.1.16.0             [.] FcPatternObjectGetWithBinding.part.0
     4,57%  libc.so.6                           [.] __strlen_sse2
     3,35%  libwebkit2gtk-4.1.so.0.24.3         [.] SkFontMgr_fontconfig::GetFamilyNames(_FcConfig*)
     1,10%  libfontconfig.so.1.16.0             [.] strcmp@plt
     1,09%  libwebkit2gtk-4.1.so.0.24.3         [.] FcPatternGetString@plt
     1,08%  libfontconfig.so.1.16.0             [.] strlen@plt

Looks like something very wrong happening between evolution, WebKitWebProces and fontconfig

Thanks for a bug report. It can be something from the fontconfig, but I cannot tell for sure. Let's move this to the WebKitGTK for further investigation.

A user pinged me with a similar problem recently and it also turned out to be something with the fontconfig cache. Possibly an update broke something (an update of the fontconfig, or webkit2gtk4.1, or maybe of some font). His fix was quite similar:

   rm -rf .cache/fontconfig
   sudo fc-cache -r -v

Thanks for your message.

I got the issue again today. In this case this happened right after clicking on a scam message (it did get displayed in the bottom panel) and before opening the source code (ctrl-U) → then the source code panel stayed blank and I noticed two runaway WebKitWebProces processes (probably one for the (at least partially displayed) corrupted e-mail message and a second for the (blank) source code window. I checked the (base64-encoded) HTML code and noticed that it is truncated in the middle of the "style" statement of a "table" tag. Could that explain this failure?

If that would be the case, then that would mean that a maliciously crafted HTML e-mail could cause a DoS on WebKit processes by corrupting the fontconfig cache, but I'm surprised by the relation to fontconfig (since the style tag is *not* interrupted in a font-family statement) and I could *not* reproduce the bug by re-opening the same offending (and truncated) message.

Unrelated note: it would be nice (and safe) to have a way to display the source code of an e-mail message in evolution without having to actually open it...

[...]
    <div class="card" style="background:#ffffff;background-color:#ffffff;margin:0px auto;max-width:600px;">
      <table align="center" border="0" cellpadding="0" cellspacing="0" role="presentation" style="background:#ffffff;background-color:#ffffff;width:100%;

Could you check what updates you received recently, please? I doubt the fontconfig cache was broken by the page. Either it's a WebKitGTK regression or some font or fontconfig update accidentally broke something related, on which the WebKitGTK relies.

Could you run also the `sudo fc-cache -r -v` , please? I understood from your text that you executed it for the regular user, not for the system font cache.

Regarding the message source, it's not a problem, you do not need to view the actual message first, the thing is that the message source is also drawn by the WebKitGTK, thus when the WebKitWebProcess is stuck on something it blocks also the message source viewing.

By the way, the other user was still on Fedora Linux 43, while you use the 44 version. The common part for you two seems to be the WebKitGTK (webkit2gtk4.1), but I do not know what exact updates you received recently, neither the other person.

Sure:

  fontconfig-2.17.0-4.fc44.x86_64 was unchanged recently

The following packages have been updated right before the first occurrence of this incident:

  webkit2gtk4.1-0:2.52.5-1.fc44.x86_64  →  webkit2gtk4.1-0:2.54.0-2.fc44.x86_64
  evolution-0:3.60.2-1.fc44.x86_64      →  evolution-0:3.60.2-2.fc44.x86_64

I did rebuild the system font cache as per requested

FWIW,
I found the same issue last Thu of Fri, and because I had a similar issue with some QT application on another computer I also figure that regenerating the cache would help.

My issues was particularly bad because I tend to use Liberation Sans as the system font which I find more pleasing than the Adawaita font. However regenerating the cache and switching to Adawaita is just a temporary fix, from time to time webkitgtk still goes crazy and I have to delete the local cache and regenerate the system one again.

I am still on f43, so it is not specific to f44

>   evolution-0:3.60.2-1.fc44.x86_64      →  evolution-0:3.60.2-2.fc44.x86_64

That change only added a patch for bug #2531477 [1], nothing font-related for sure. I suspect the WebKitGTK update, which has another known regression too [2].

[1] https://src.fedoraproject.org/rpms/evolution/c/82178befdd16527f9c504a3172a88a27ff51612c?branch=f44
[2] https://bugs.webkit.org/show_bug.cgi?id=322568

For what it's worth, the Flathub.org version is also affected:
https://github.com/flathub/org.gnome.Evolution/issues/234

Did any of you notice any pattern how to reproduce this, please? It's possible the WebKitWebProcess broke the fontconfig cache due to concurrency (Michael mentioned to me there can be some WebKitWebProcess-es preloading pages or what that was for, I already forgot the important details); I'm wondering whether it can be anything about that.

The reproducer would be very helpful. It can be just "quickly moving between messages" or some such thing.

It may certainly be a concurrency issue, multiple times when my evolution was stuck I had multiple webkit processes all at 100%, but it wasn't always the case.

Hi, this looks like https://bugs.webkit.org/show_bug.cgi?id=325185

Hi, I downgraded webkit2gtk4.1 to an older version (2.52.1) and still observe an almost identical defect.

Affected packages:

  evolution-3.60.2-2.fc44.x86_64
  fontconfig-2.17.0-4.fc44.x86_64
  webkit2gtk4.1-2.52.1-1.fc44.x86_64

Steps to reproduce:

1) open evolution → messages listed
2) click a message → message content is correctly display
3) [ctrl-u] to open the source of the message → blank window (no text is displayed), runaway WebKitWebProces process

The only difference is that is still possible to click and read the content of another message in the messages list, but it is impossible to open any message (either opening the source code, replying, or just composing a new message).

Each action ends up with the spanning of a new runaway WebKitWebProces process.

$ top

top - 09:57:29 up 14 min,  1 user,  load average: 4,57, 2,46, 1,36
Tasks: 463 total, 8 running, 455 sleep, 0 d-sleep, 0 stopped, 0 zombie
%Cpu(s): 52,7 us,  0,4 sy,  0,0 ni, 46,4 id,  0,0 wa,  0,3 hi,  0,1 si,  0,0 st 
MiB Mem :  24008,4 total,  12772,8 free,   5679,0 used,   5652,0 buff/cache     
MiB Swap:  16384,0 total,  16384,0 free,      0,0 used.  18329,4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                    
  10817 username  20   0   70,5g 154636 126968 R  99,7   0,6   1:45.63 WebKitWebProces                                            
  11442 username  20   0   70,5g 154204 126572 R  99,7   0,6   1:39.89 WebKitWebProces                                            
  12252 username  20   0   70,5g 154108 126532 R  99,7   0,6   0:26.40 WebKitWebProces                                            
   6657 username  20   0   70,5g 154412 126652 R  99,4   0,6  10:44.50 WebKitWebProces                                            
  11434 username  20   0   70,5g 153860 126276 R  99,4   0,6   1:39.90 WebKitWebProces                                            
  12257 username  20   0   70,5g 154364 126776 R  99,4   0,6   0:26.31 WebKitWebProces                                            

$ sudo perf top -p6657 --sort dso,symbol

Samples: 2M of event 'cpu/cycles/P', 4000 Hz, Event count (approx.): 54733853665 lost: 0/0 drop: 0/0
Overhead  Shared Object                Symbol
  24,25%  libfontconfig.so.1.16.0      [.] FcObjectTypeLookup.lto_priv.0
  22,05%  libc.so.6                    [.] __strcmp_sse42
  16,50%  libfontconfig.so.1.16.0      [.] FcPatternGetString
  10,88%  libc.so.6                    [.] __strlen_sse2
   9,88%  libfontconfig.so.1.16.0      [.] FcPatternObjectPosition.lto_priv.0
   8,58%  libfontconfig.so.1.16.0      [.] FcPatternObjectGetWithBinding.part.0
   3,77%  libwebkit2gtk-4.1.so.0.21.5  [.] SkFontMgr_fontconfig::GetFamilyNames(_FcConfig*)
   1,27%  libfontconfig.so.1.16.0      [.] strlen@plt
   1,22%  libwebkit2gtk-4.1.so.0.21.5  [.] FcPatternGetString@plt
   1,18%  libfontconfig.so.1.16.0      [.] strcmp@plt

Again, this is happening now with release 2.52.1 of webkit2gtk4.1, not 2.54.0.

Nervmind: I just noticed the upstream report is against libwebkit2gtk-4.1-0 2.52.6 so not against a potential regression introduced in libwebkit2gtk-4.1-0 2.54

Right, the upstream report is against an older version. It also suggests it's about certain fonts, not having set family names.

OK, just for documentation (and suggested work-arounds until the bug is fixed upstream), as suspected in the upstream bug report, the trigger for this issue is actually Google Chrome 154.

In my case, along with the evolution and the webkit2gtk4.1 packages upgrades, the following package was also updated right before the first occurrence of this bug:

  google-chrome-stable-0:153.0.8010.36-1.x86_64 → google-chrome-stable-0:154.0.8037.57-1.x86_64

As stated in the bug report, Google Chrome introduce with its release 154 the generation of fontconfig cache entries with only "file" and "fontwrapper" tags (no "family" tag).

This could easily be verified by doing:

  1) wipe the existing font cache:

     $ rm -rf .cache/fontconfig

  2) generate a clean font cache:

     $ fc-cache -rv

  3) check for entries without family names:

     $ fc-cat 2>/dev/null | grep '":' | wc -l
     0

  4) launch Google Chrome, e.g. on Microsoft Teams

  5) check for entries without family names:

     $ fc-cat 2>/dev/null | grep '":' | wc -l
     56

At that point, any spawned WebKitWebProcess process will get stuck in an infinite loop.

I could confirm the reproducibility of the issue: starting Chrome *before* evolution will kill the rendering of the messages. Wiping the fontconfig cache after starting Chrome and *then* running evolution is fine (as long as Chrome doesn't create any more font cache entries for WOFF/WOFF2 fonts).

So if we absolutely need to use Google Chrome 154 *and* webkit2gtk-4.1, then we either:
- need to wipe the (user) fontconfig cache after each use of Google Chrome ( rm -rf ~/.cache/fontconfig )
- or to (temporarily, until publication of the webkit2gtk-4.1 patch) remove any WOFF/WOFF2 fonts from the system (don't forget to purge the fontconfig cache and regenerate it after that).

Removing WOFF/WOFF2 fonts (replacing these by OTF fonts) made Google Chrome 154 interoperable again with webkit2gtk-4.1 on my system.

A user posted a workaround for Flatpak at:
https://gitlab.gnome.org/GNOME/evolution/-/work_items/3410#note_2890413