Bug 1666984

+++ This bug was initially created as a clone of Bug #1665661 +++

Description of problem:
Connecting to an Exchange server using Evolution's EWS will not read any messages. Error message:

"Something has gone wrong when displaying the message

"A WebKitWebProcess crashed when displaying the message. You can try again by moving to another message and back. If the issue persists, please file a bug report in the GNOME bugzilla."

This started happening with an "upgrade" to webkitgtk4 in Fedora 26. I assume this has since been replaced with webkit2gtk3 (?), which has the same issues.

Version-Release number of selected component (if applicable):
webkit2gtk3-2.22.5-1.fc28.x86_64
evolution-3.28.5-2.fc28.x86_64
evolution-ews-3.28.5-2.fc28.x86_64

How reproducible:
Unknown

Steps to Reproduce:
1. Load Evolution
2. Set up an account to access an Exchange-based mailbox.
3. Click on a message in the Inbox

Actual results:
Error message (see above)

Expected results:
E-mail messages appear.

Additional info:
This bug prevents me from reading e-mail on my Fedora machines. If it persists much longer, I will be required to seek another distribution that has a working e-mail client.

--- Additional comment from Milan Crha on 2019-01-14 09:52:46 UTC ---

Thanks for a bug report. I tried to reproduce this, but no luck. I used the same versions as you've installed. As it says something about crash, could you provide a crash report, please? The ABRT software is supposed to catch it and let you know and give you an option to report it to the bugzilla.

Also, as it crashes on the WebKitWebProcess side, it's possible the crash is related to certain message format, which means my messages do not trigger it. Could you provide one such message for testing, please? I understand it can contain private information, thus having a test message would be a plus. You can save the message by right-clicking it in the message list and pick Save as mbox.

To debug this further on your side, could you install debuginfo packages for evolution-data-server and evolution, please? You can do that with a command like this one:

   $ sudo dnf install evolution-debuginfo evolution-data-server-debuginfo --enablerepo=updates-debuginfo

Make sure the versions of the debug info packages match the versions of the binary packages, otherwise it won't work.
Then check how many WebKitWebProcess processes are running (ps ax | grep WebKitWebProcess); there would be probably none, depending what you've running in the system. Then run evolution from a terminal with the preview panel disabled:

   $ evolution -c mail --disable-preview

and watch what it'll print there. Check which WebKitWebProcess-es are running now, as there should be some, even when the preview panel is disabled (ps ax | grep WebKitWebProcess). The 'ps' command provides also process ID of the process as the first number on the line of its output. Use the ID of the process to attach gdb to it (replace PID in the below command with it) from another terminal window:

   $ gdb --pid=PID --ex c --ex "bt full" --ex c --ex q

Response to any gdb question with either 'y' or 'return'. Then switch to the evolution, show the preview panel (Ctrl+M) and select one of the messages which can trigger the crash. This should cause new output on at least the gdb terminal, but maybe also on the evolution terminal. The gdb terminal is supposed to print the backtrace of the crash. I'd like to see it, to know where it crashed. You may get similar information from ABRT. Please check the output for any private information, like passwords, email address, server addresses,... I usually search for "pass" at least (quotes for clarity only), though the WebKitWebProcess might not contain more than what is written in the message itself, like recipient addresses or such. Output on the evolution terminal may contain valuable information too.

By the way, does software like 'devhelp' work for you? It also uses webkit2gtk3.

--- Additional comment from Karl on 2019-01-14 18:54:03 UTC ---

The following output is sent to the terminal in which Evolution is running:

"FATAL: Primitive gigacage disabled, but we don't want that in this process."

There were no processes that came up matching WebKitWebProcess; I tried WebKitNetworkProcess instead. No new messages popped up there when I loaded a message, unfortunately.

I did, however, get an ABRT popup when I did this; here's what seems relevant (and I hope it's helpful):

 - exploitable
Likely crash reason: Jump to an invalid address
Exploitable rating (0-9 scale): 6

reason WebKitWebProcess killed by SIGSEGV
crash_function Gigacage::primitiveGigacageDisabled(void*)

 - core_backtrace
{   "signal": 11
,   "executable": "/usr/libexec/webkit2gtk-4.0/WebKitWebProcess"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 140599003599820
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 16008140
                ,   "function_name": "Gigacage::primitiveGigacageDisabled(void*)"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003603086
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 16011406
                ,   "function_name": "Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled()"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599024834017
                ,   "build_id": "e2f2f9df430abf2135473d28d7514cf49d975388"
                ,   "build_id_offset": 9287137
                ,   "function_name": "WebKit::WebProcess::WebProcess()"
                ,   "file_name": "/lib64/libwebkit2gtk-4.0.so.37"
                }
              , {   "address": 140599024834151
                ,   "build_id": "e2f2f9df430abf2135473d28d7514cf49d975388"
                ,   "build_id_offset": 9287271
                ,   "function_name": "WebKit::WebProcess::singleton()"
                ,   "file_name": "/lib64/libwebkit2gtk-4.0.so.37"
                }
              , {   "address": 140599026429717
                ,   "build_id": "e2f2f9df430abf2135473d28d7514cf49d975388"
                ,   "build_id_offset": 10882837
                ,   "function_name": "int WebKit::ChildProcessMain<WebKit::WebProcess, WebKit::WebProcessMain>(int, char**)"
                ,   "file_name": "/lib64/libwebkit2gtk-4.0.so.37"
                } ]
        }
      , {   "frames":
              [ {   "address": 140598829966377
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 980009
                ,   "function_name": "__poll"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 140598873713750
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 314454
                ,   "function_name": "g_main_context_iterate.isra.21"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598873714048
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 314752
                ,   "function_name": "g_main_context_iteration"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598873714129
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 314833
                ,   "function_name": "glib_worker_main"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598873878554
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 479258
                ,   "function_name": "g_thread_proxy"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598950339988
                ,   "build_id": "0edef5aa88708d13d8de6b996b7f57bab1e769dd"
                ,   "build_id_offset": 30100
                ,   "function_name": "start_thread"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140598830010191
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 1023823
                ,   "function_name": "__clone"
                ,   "file_name": "/lib64/libc.so.6"
                } ]
        }
      , {   "frames":
              [ {   "address": 140598950365498
                ,   "build_id": "0edef5aa88708d13d8de6b996b7f57bab1e769dd"
                ,   "build_id_offset": 55610
                ,   "function_name": "pthread_cond_timedwait@@GLIBC_2.3.2"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140599003642620
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 16050940
                ,   "function_name": "bmalloc::Scavenger::threadRunLoop()"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003643055
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 16051375
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140598839645507
                ,   "build_id": "ec68c248ced41419fcea3e5ba9158eb57fe14a5f"
                ,   "build_id_offset": 783683
                ,   "function_name": "execute_native_thread_routine"
                ,   "file_name": "/lib64/libstdc++.so.6"
                }
              , {   "address": 140598950339988
                ,   "build_id": "0edef5aa88708d13d8de6b996b7f57bab1e769dd"
                ,   "build_id_offset": 30100
                ,   "function_name": "start_thread"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140598830010191
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 1023823
                ,   "function_name": "__clone"
                ,   "file_name": "/lib64/libc.so.6"
                } ]
        }
      , {   "frames":
              [ {   "address": 140598829966377
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 980009
                ,   "function_name": "__poll"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 140598873713750
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 314454
                ,   "function_name": "g_main_context_iterate.isra.21"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598873714706
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 315410
                ,   "function_name": "g_main_loop_run"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140599003578160
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15986480
                ,   "function_name": "WTF::RunLoop::run()"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003412287
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15820607
                ,   "function_name": "WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*)"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003570061
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15978381
                ,   "function_name": "WTF::wtfThreadEntryPoint(void*)"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140598950339988
                ,   "build_id": "0edef5aa88708d13d8de6b996b7f57bab1e769dd"
                ,   "build_id_offset": 30100
                ,   "function_name": "start_thread"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140598830010191
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 1023823
                ,   "function_name": "__clone"
                ,   "file_name": "/lib64/libc.so.6"
                } ]
        }
      , {   "frames":
              [ {   "address": 140598829966377
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 980009
                ,   "function_name": "__poll"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 140598873713750
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 314454
                ,   "function_name": "g_main_context_iterate.isra.21"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598873714706
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 315410
                ,   "function_name": "g_main_loop_run"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598882056282
                ,   "build_id": "8bc2dcca66b559c619e5a9c1bd1d230382a8316c"
                ,   "build_id_offset": 866394
                ,   "function_name": "gdbus_shared_thread_func"
                ,   "file_name": "/lib64/libgio-2.0.so.0"
                }
              , {   "address": 140598873878554
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 479258
                ,   "function_name": "g_thread_proxy"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598950339988
                ,   "build_id": "0edef5aa88708d13d8de6b996b7f57bab1e769dd"
                ,   "build_id_offset": 30100
                ,   "function_name": "start_thread"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140598830010191
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 1023823
                ,   "function_name": "__clone"
                ,   "file_name": "/lib64/libc.so.6"
                } ]
        }
      , {   "frames":
              [ {   "address": 140598829966377
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 980009
                ,   "function_name": "__poll"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 140598873713750
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 314454
                ,   "function_name": "g_main_context_iterate.isra.21"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598873714706
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 315410
                ,   "function_name": "g_main_loop_run"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140599003578160
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15986480
                ,   "function_name": "WTF::RunLoop::run()"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003412287
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15820607
                ,   "function_name": "WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*)"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003570061
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15978381
                ,   "function_name": "WTF::wtfThreadEntryPoint(void*)"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140598950339988
                ,   "build_id": "0edef5aa88708d13d8de6b996b7f57bab1e769dd"
                ,   "build_id_offset": 30100
                ,   "function_name": "start_thread"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140598830010191
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 1023823
                ,   "function_name": "__clone"
                ,   "file_name": "/lib64/libc.so.6"
                } ]
        }
      , {   "frames":
              [ {   "address": 140598829966377
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 980009
                ,   "function_name": "__poll"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 140598873713750
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 314454
                ,   "function_name": "g_main_context_iterate.isra.21"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140598873714706
                ,   "build_id": "908676379fc7a7e6700d4f1fd5d0154de0e578e1"
                ,   "build_id_offset": 315410
                ,   "function_name": "g_main_loop_run"
                ,   "file_name": "/lib64/libglib-2.0.so.0"
                }
              , {   "address": 140599003578160
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15986480
                ,   "function_name": "WTF::RunLoop::run()"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003412287
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15820607
                ,   "function_name": "WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*)"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140599003570061
                ,   "build_id": "04929f844e105892f01b00fc9c36e625b4b32033"
                ,   "build_id_offset": 15978381
                ,   "function_name": "WTF::wtfThreadEntryPoint(void*)"
                ,   "file_name": "/lib64/libjavascriptcoregtk-4.0.so.18"
                }
              , {   "address": 140598950339988
                ,   "build_id": "0edef5aa88708d13d8de6b996b7f57bab1e769dd"
                ,   "build_id_offset": 30100
                ,   "function_name": "start_thread"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140598830010191
                ,   "build_id": "f9c839c542defd9068676b3887a5f765416dbc55"
                ,   "build_id_offset": 1023823
                ,   "function_name": "__clone"
                ,   "file_name": "/lib64/libc.so.6"
                } ]
        } ]
}

--- Additional comment from Milan Crha on 2019-01-14 19:47:25 UTC ---

Thanks for the update. I really meant WebKitWebProcess. Could you try to run evolution from a terminal as this, please:

   $ GIGACAGE_ENABLED=0 evolution

It disables the gigacage, thus should, theoretically, avoid the crash. I do not know why the WebKit aborts, but it's clearly its issue, thus I move it there.

--- Additional comment from Karl on 2019-01-14 20:37:58 UTC ---

It works as expected with GIGACAGE_ENABLED=0 set.

--- Additional comment from Michael Catanzaro on 2019-01-14 22:46:54 UTC ---

Probably fixed by https://trac.webkit.org/changeset/239787/webkit

Interesting timing, since it looks like this has been broken for a long time.

(P.S. Gigacage is an important security feature, so you may wish to reconsider whatever limits you have set on virtual memory allocation.)

--- Additional comment from Michael Catanzaro on 2019-01-14 23:25:00 UTC ---

Backported to 2.22: https://trac.webkit.org/log/webkit/releases/WebKitGTK/webkit-2.22

--- Additional comment from Karl on 2019-01-15 22:06:53 UTC ---

Yes, this has been broken for a while, since Fedora 26 or so. I just realized that I probably duplicated a bug with this report: bug 1564970 and an associated upstream bug, https://bugs.webkit.org/show_bug.cgi?id=183329. Sorry if I created duplicated effort! Those were reported against different components and versions, though.

Regarding the virtual memory limits: I did have a soft limit set at 4GB (half the available physical memory) and a hard limit set at 8GB. I tried disabling those, and it still happens, so it appears that is not the problem. Either that, or Gigacage is trying to allocate more than 16 GB (total size of RAM + swap) and thus ran out of memory outright, which would be a problem in itself.

--- Additional comment from Michael Catanzaro on 2019-01-15 22:13:36 UTC ---

https://bugs.webkit.org/show_bug.cgi?id=183329 fixed this originally, but it must have regressed at some point, or maybe the fix never worked. Now it should really be fixed after https://bugs.webkit.org/show_bug.cgi?id=193292. Hopefully.

(In reply to Karl from comment #7)
> Regarding the virtual memory limits: I did have a soft limit set at 4GB
> (half the available physical memory) and a hard limit set at 8GB. I tried
> disabling those, and it still happens, so it appears that is not the
> problem. Either that, or Gigacage is trying to allocate more than 16 GB
> (total size of RAM + swap) and thus ran out of memory outright, which would
> be a problem in itself.

It requests 80-90 GB of virtual memory (address space), but of course does not actually use it: it's just to make it hard to guess valid addresses. The other thing that could break gigacage besides a virtual memory limit would be disabling overcommit at the kernel level. At least, those are the only ways that I'm currently aware of.

--- Additional comment from Tomas Popela on 2019-01-16 05:44:03 UTC ---

Karl, I'm preparing a scratch build with these fixes applied, can you please try it if it works (without setting the GIGACAGE_ENABLED)?

--- Additional comment from Tomas Popela on 2019-01-16 07:03:26 UTC ---

It's ready on https://koji.fedoraproject.org/koji/taskinfo?taskID=32058534.

--- Additional comment from Karl on 2019-01-16 17:52:51 UTC ---

It seems to work with version 2.22.5-2. Thank you!