Bug 1823240 - [abrt] evolution-data-server: vasprintf(): evolution-addressbook-factory killed by SIGSEGV
Summary: [abrt] evolution-data-server: vasprintf(): evolution-addressbook-factory kill...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: evolution-ews
Version: 32
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Milan Crha
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:dac7a25aafc19b6265f89ecb23c...
: 1823992 1828163 1829335 1830127 1830225 1835935 1836457 1836655 1842868 1842972 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-13 00:14 UTC by Ramprasad G
Modified: 2020-06-22 11:12 UTC (History)
12 users (show)

Fixed In Version: evolution-ews-3.36.3
Clone Of:
Environment:
Last Closed: 2020-05-21 14:08:48 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: core_backtrace (51.54 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: cpuinfo (2.47 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: dso_list (19.46 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: environ (1.89 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: exploitable (82 bytes, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: limits (1.29 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: maps (123.12 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: mountinfo (2.46 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: open_fds (1.45 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details
File: proc_pid_status (1.33 KB, text/plain)
2020-04-13 00:14 UTC, Ramprasad G
no flags Details

Description Ramprasad G 2020-04-13 00:14:10 UTC
Description of problem:
I believe it crashed wen i try to hibernate the system

Version-Release number of selected component:
evolution-data-server-3.36.1-1.fc32

Additional info:
reporter:       libreport-2.12.0
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/evolution-addressbook-factory.service
cmdline:        /usr/libexec/evolution-addressbook-factory
crash_function: vasprintf
executable:     /usr/libexec/evolution-addressbook-factory
journald_cursor: s=ede23ebd765f4a808be8eefe1ee953cb;i=201d6;b=7e9660ab15274e51b38b687bc0dd429b;m=1e5660264;t=5a2fb12c22d36;x=35173ff6d80d60ba
kernel:         5.6.2-301.fc32.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000
xsession_errors:

Comment 2 Ramprasad G 2020-04-13 00:14:17 UTC
Created attachment 1678343 [details]
File: core_backtrace

Comment 3 Ramprasad G 2020-04-13 00:14:19 UTC
Created attachment 1678344 [details]
File: cpuinfo

Comment 4 Ramprasad G 2020-04-13 00:14:21 UTC
Created attachment 1678345 [details]
File: dso_list

Comment 5 Ramprasad G 2020-04-13 00:14:22 UTC
Created attachment 1678346 [details]
File: environ

Comment 6 Ramprasad G 2020-04-13 00:14:24 UTC
Created attachment 1678347 [details]
File: exploitable

Comment 7 Ramprasad G 2020-04-13 00:14:26 UTC
Created attachment 1678348 [details]
File: limits

Comment 8 Ramprasad G 2020-04-13 00:14:38 UTC
Created attachment 1678349 [details]
File: maps

Comment 9 Ramprasad G 2020-04-13 00:14:39 UTC
Created attachment 1678350 [details]
File: mountinfo

Comment 10 Ramprasad G 2020-04-13 00:14:40 UTC
Created attachment 1678351 [details]
File: open_fds

Comment 11 Ramprasad G 2020-04-13 00:14:42 UTC
Created attachment 1678352 [details]
File: proc_pid_status

Comment 12 Milan Crha 2020-04-14 07:10:30 UTC
Thanks for a bug report. This is a bug in glib-networking, most likely this one:
https://gitlab.gnome.org/GNOME/glib-networking/-/issues/131

This is supposed to be fixed in glib-networking-2.64.1, which you had installed. Could it be that you updated the system, but did not restart the machine or at least did not relogin, thus the background process still used the old binaries?

Comment 13 Michael Catanzaro 2020-04-14 13:49:44 UTC
This one is not the same. #131 was a simple NULL pointer dereference that was clearly glib-networking's fault and easy to fix. This one is memory corruption, which could be coming from anywhere (libsoup, evolution-data-server, glib-networking, who knows) and which I'm unable to reproduce. I don't see anything actionable here; we really need to catch it in valgrind if possible.

I would usually say: "If it's reproducible, then we can get it in valgrind and fix it." But we already tried that in #131 and failed. That bug is closed; I had left a comment asking to open an evolution-data-server bug, since memory corruption bugs are usually best directed against the main process.

Comment 14 Milan Crha 2020-04-14 16:53:54 UTC
Thanks Michael, I re-read the complete backtrace and it is quite similar, with respect of the circumstances, to the bug #1818995. There are more similar bugs like bug #1672791.

I made some semi-massive changes for 3.36.0 within [1], but they did not address this issue.

I'm moving this to evolution-ews, even though I've not been able to reproduce this myself.

[1] https://gitlab.gnome.org/GNOME/evolution-ews/issues/77

Comment 15 Michael Catanzaro 2020-04-14 18:27:10 UTC
Looks like https://gitlab.gnome.org/GNOME/evolution-data-server/-/issues/203 was created to report this bug, but there were two bugs there and you fixed the obvious one rather than this crash. The reporter maybe didn't realize only one of the two issues was fixed?

Comment 16 Milan Crha 2020-04-15 08:26:39 UTC
*** Bug 1823992 has been marked as a duplicate of this bug. ***

Comment 17 Milan Crha 2020-04-15 08:33:26 UTC
(In reply to Michael Catanzaro from comment #15)
> Looks like https://gitlab.gnome.org/GNOME/evolution-data-server/-/issues/203

My understanding is that the above bug was to address the valgrind report from that bug, which I did.

> was created to report this bug, but there were two bugs there and you fixed
> the obvious one rather than this crash. The reporter maybe didn't realize
> only one of the two issues was fixed?

The reporter had two issues, but both of them were addressed. The one is in comment #13 above, the first paragraph there. The second is the evolution-data-server's 203. The upstream reporter did not face this crash, as I understand it.

Comment 18 Michael Catanzaro 2020-04-15 16:18:22 UTC
Upstream reporter was definitely trying to report this crash.

Comment 19 Milan Crha 2020-04-17 06:09:46 UTC
Ramprasad, it seems you are able to reproduce this quite often. Could you install debuginfo packages for evolution-data-server, evolution-ews, glib2 and glib-networking [1], please (make sure their versions will match the versions of the binary packages), then install also valgrind. With that prepared, start a terminal and in that execute these commands:

   $ export GIGACAGE_ENABLED=0
   $ G_SLICE=always-malloc valgrind --num-callers=30 --leak-check=no --aspace-minaddr=0x100000000 \
         /usr/libexec/evolution-addressbook-factory -w &>log.txt

Then wait several seconds, to give the factory enough time to start and register itself on the D-Bus. You'll see a significant CPU usage due to all the memory checking done by valgrind (getting data from the factory will be also significantly slower). Once the CPU usage goes down, you can start Evolution (or any other application you use to access data provided by evolution-data-server) and hopefully reproduce the crash. Valgrind can avoid certain types of crashes and only logs about them, thus even if the factory will not abort the log can still contain valuable information.

Could you share the resulting log.txt file, please? Check it for any private information, like passwords, email address, server addresses,... I usually search for "pass" at least (quotes for clarity only), though valgrind backtraces usually do not expose those.

Could you also watch for and share what circumstances lead to the crash, please? It can be anything like changing network, resume from suspend, going offline due to unexpected disconnect from the network,... even anything completely unrelated to network connection.



[1] install debuginfo packages for example with:

    $ sudo dnf install --enablerepo=*-debuginfo evolution-data-server-debuginfo evolution-ews-debuginfo glib2-debuginfo glib-networking-debuginfo

In case you've any of those already installed, then use 'update' instead of 'install' command of dnf. To see what versions you've installed you can run:

   $ rpm -qa | egrep "evolution-data-server|evolution-ews|glib2|glib-networking" | sort

Comment 20 Ramprasad G 2020-04-17 18:13:27 UTC
I did all and waiting for the crash.

In my observation the crash occurs when the system goes to hibernate or standby mode.

Comment 21 Milan Crha 2020-04-27 10:26:16 UTC
*** Bug 1828163 has been marked as a duplicate of this bug. ***

Comment 22 Milan Crha 2020-04-30 09:45:11 UTC
By the way, if you ran under valgrind, did the log contain anything usable? Eventually, if you can share it and unsure, then I'll check it.

Comment 23 Milan Crha 2020-05-04 06:14:23 UTC
*** Bug 1829335 has been marked as a duplicate of this bug. ***

Comment 24 Milan Crha 2020-05-04 06:39:13 UTC
*** Bug 1830127 has been marked as a duplicate of this bug. ***

Comment 25 Milan Crha 2020-05-04 06:40:47 UTC
*** Bug 1830225 has been marked as a duplicate of this bug. ***

Comment 26 Milan Crha 2020-05-15 06:49:16 UTC
*** Bug 1835935 has been marked as a duplicate of this bug. ***

Comment 27 Milan Crha 2020-05-18 05:35:31 UTC
*** Bug 1836457 has been marked as a duplicate of this bug. ***

Comment 28 Milan Crha 2020-05-18 05:37:32 UTC
*** Bug 1836655 has been marked as a duplicate of this bug. ***

Comment 29 Milan Crha 2020-05-20 16:48:14 UTC
I tried to reproduce this, but no luck so far. Neither under valgrind, nor without it.

Nonetheless, I noticed some bottlenecks, one being unnecessary re-subscribe after resume from suspend. Another one when there are too many disconnects in a short period of time, in which can I can see an error on the addressbook factory console:

    (evolution-addressbook-factory:97257): evolution-ews-WARNING **: 16:16:09.390:
    ..../evolutionews/src/server/e-ews-notification.c:401: You have exceeded the available
    subscriptions for your account.  Remove unnecessary subscriptions and try your request again.

which is caused by failing to unsubscribe due to being offline. The Exchange server remembers the subscriptions even after being disconnected from the server. After a brief look into the code there doesn't seem to be an easy way to remember the subcription ID, because both the notification object (which knows the ID) and the connection (which knows only the notification object) are freed on the disconnect, thus the information is gone.

I think I'm able to do something with the unnecessary re-subscribe, which will help to relax the server. It should avoid those multiple threads with e_ews_notification_get_events_thread(). They do not do anything special to trigger the crash (at least from what I saw here), but it is possible it can be triggered by certain thread interleaving or something, which I cannot reproduce here. It's only my guess. I can be completely wrong.

Comment 30 Milan Crha 2020-05-21 14:08:48 UTC
I made changes upstream [1] for 3.37.2+ and 3.36.3+. I cannot tell for sure whether this will really help, but it surely avoids some of the symptoms from the backtraces of this and the duplicate bugs.

[1] https://gitlab.gnome.org/GNOME/evolution-ews/commit/9a318b0b

Comment 31 Milan Crha 2020-06-03 06:16:06 UTC
*** Bug 1842972 has been marked as a duplicate of this bug. ***

Comment 32 Andreas Gnau 2020-06-22 11:12:41 UTC
*** Bug 1842868 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.