Bug 1600489 - [abrt] ESourceRegistry's thread main_context can leak
Summary: [abrt] ESourceRegistry's thread main_context can leak
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: evolution-data-server
Version: 28
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Milan Crha
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:34a9941a5ca0854937ad5d5f428...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-12 11:37 UTC by Brian J. Murrell
Modified: 2018-07-18 14:04 UTC (History)
7 users (show)

Fixed In Version: evolution-data-server-3.28.5
Clone Of:
Environment:
Last Closed: 2018-07-18 14:04:37 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: backtrace (72.77 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: cgroup (231 bytes, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: core_backtrace (31.83 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: cpuinfo (1.20 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: dso_list (14.70 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: environ (3.12 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: limits (1.29 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: maps (66.91 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: mountinfo (9.09 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: open_fds (91.77 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details
File: proc_pid_status (1.33 KB, text/plain)
2018-07-12 11:37 UTC, Brian J. Murrell
no flags Details

Description Brian J. Murrell 2018-07-12 11:37:25 UTC
Description of problem:
Not sure what caused this

Version-Release number of selected component:
evolution-data-server-3.28.3-1.fc28

Additional info:
reporter:       libreport-2.9.5
backtrace_rating: 4
cmdline:        /usr/libexec/evolution-source-registry
crash_function: g_wakeup_new
executable:     /usr/libexec/evolution-source-registry
journald_cursor: s=bc64b0deac69405ca8d741b6730dac86;i=5d3072;b=5d70b12ce1b34c1a9b7e05d6df73d3f7;m=f8fb47dc21;t=570c0feb00412;x=cc77da3fdd5d3db6
kernel:         4.17.2-200.fc28.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1001

Truncated backtrace:
Thread no. 1 (8 frames)
 #4 g_wakeup_new at gwakeup.c:161
 #5 g_main_context_new at gmain.c:656
 #6 g_dbus_connection_send_message_with_reply_sync at gdbusconnection.c:2138
 #7 g_dbus_connection_call_sync_internal at gdbusconnection.c:5950
 #8 g_dbus_connection_call_with_unix_fd_list_sync at gdbusconnection.c:6299
 #9 g_dbus_proxy_call_sync_internal at gdbusproxy.c:2870
 #10 g_dbus_proxy_call_sync at gdbusproxy.c:3062
 #11 0x00007f0086cf436b in

Potential duplicate: bug 1394206

Comment 1 Brian J. Murrell 2018-07-12 11:37:31 UTC
Created attachment 1458353 [details]
File: backtrace

Comment 2 Brian J. Murrell 2018-07-12 11:37:32 UTC
Created attachment 1458354 [details]
File: cgroup

Comment 3 Brian J. Murrell 2018-07-12 11:37:33 UTC
Created attachment 1458355 [details]
File: core_backtrace

Comment 4 Brian J. Murrell 2018-07-12 11:37:34 UTC
Created attachment 1458356 [details]
File: cpuinfo

Comment 5 Brian J. Murrell 2018-07-12 11:37:36 UTC
Created attachment 1458357 [details]
File: dso_list

Comment 6 Brian J. Murrell 2018-07-12 11:37:37 UTC
Created attachment 1458358 [details]
File: environ

Comment 7 Brian J. Murrell 2018-07-12 11:37:38 UTC
Created attachment 1458359 [details]
File: limits

Comment 8 Brian J. Murrell 2018-07-12 11:37:39 UTC
Created attachment 1458360 [details]
File: maps

Comment 9 Brian J. Murrell 2018-07-12 11:37:41 UTC
Created attachment 1458361 [details]
File: mountinfo

Comment 10 Brian J. Murrell 2018-07-12 11:37:42 UTC
Created attachment 1458362 [details]
File: open_fds

Comment 11 Brian J. Murrell 2018-07-12 11:37:43 UTC
Created attachment 1458363 [details]
File: proc_pid_status

Comment 12 Milan Crha 2018-07-13 06:02:10 UTC
Thanks for a bug report. According to the backtrace:
> Creating pipes for GWakeup: Too many open files
it looks like there had been opened too many file descriptors. There used to be similar issues in the past, but they had been fixed some time ago. If there are actions/steps which increase (and do not free) opened file descriptors, then those can be found for example by running:
   $ lsof -p `pidof evolution` | wc -l
before the action and then after it. Redirecting the lsof output to a file
   $ lsof -p `pidof evolution` >/tmp/before
and similarly for "after":
   $ lsof -p `pidof evolution` >/tmp/after
and then compare the two:
   $ diff -u /tmp/before /tmp/after
will show what changed.

Comment 13 Milan Crha 2018-07-13 06:04:00 UTC
Oops, except this is not evolution, but evolution-source-registry process. My fault, I'm sorry. The fixes I've been talking about were done in evolution, thus it had been unrelated to your issue.

Comment 14 Brian J. Murrell 2018-07-13 11:04:29 UTC
$ lsof -p `pidof evolution-source-registry` | tail
evolution 2523 brian   87u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   88u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   89u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   90u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   91u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   92u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   93u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   94u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   95u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   96u  a_inode               0,13         0    10694 [eventfd]
$ lsof -p `pidof evolution-source-registry` | grep eventfd | wc -l
86

for a data point.

86 open eventfds is not huge in the context of the 1024 nofiles soft limit, but perhaps it's excessive for what the process does regardless.  I guess you will have to be the judge of that.

Comment 15 Milan Crha 2018-07-17 12:35:32 UTC
Thanks for the update. I tried to call 'Refresh' on the collection backend, which basically re-populates the sources for collections, and every time I do that it increases the eventfd counter by one, thus there is leaking something. It might no the noticeable from the beginning, but the longer you have the evolution-source-registry running and the more the machine goes online and offline for such collection backends, the more file handles are leaked, which reaches the limit after some time.

I'm searching what exactly causes the leak.

Comment 16 Milan Crha 2018-07-18 14:04:37 UTC
This was tricky, because evolution-data-server didn't do anything wrong, as far as I can tell. The problem was that EWebDAVDiscover (used to find out CalDAV calendars/memo lists/task lists and CardDAV books) wanted to receive a proxy ESource, for which it requires either ESourceRegistry or ESourceRegistryServer. As the EWebDAVDiscover API didn't have any such argument it always created its own ESourceRegistry instance, which had been dropped shortly afterwards. This drop also meant that the internal GDBus object scheduled its own events on the ESourceRegistry's thread main_context (to unsubscribe D-Bus signals), but this main_context didn't have enough time to flush the queue, which resulted in a leak of the main_context, because the events in it referenced it (similar to circular reference). The fix was to flush the queue before closing the thread and to reorganize the ESourceRegistry's dispose() function, thus the flush is done on filled main_context.

While I've been in it, some structures unnecessarily long held a GCancellable object, which also means a file descriptor, thus I fixed that as well.

As the last, but not least, I added a new API for the EWebDAVDiscover to be able to provide also a function to reference an ESource, thus there is no need to create the ESourceRegistry instance, thus it is quicker and more effective now.

Created commit 98f133652 in eds master (3.29.90+) [1]
Created commit ed6149387 in eds gnome-3-28 (3.28.5+)

[1] https://gitlab.gnome.org/GNOME/evolution-data-server/commit/98f133652


Note You need to log in before you can comment on or make changes to this bug.