Bug 1600489

Summary: [abrt] ESourceRegistry's thread main_context can leak
Product: [Fedora] Fedora Reporter: Brian J. Murrell <brian>
Component: evolution-data-serverAssignee: Milan Crha <mcrha>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 28CC: alexl, caillon+fedoraproject, john.j5live, mcrha, rhughes, rstrode, sandmann
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/aad623ef281deb070b9644abc8b1af1396219a56
Whiteboard: abrt_hash:34a9941a5ca0854937ad5d5f428bbdb8fd46a090;
Fixed In Version: evolution-data-server-3.28.5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-18 14:04:37 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: cgroup
none
File: core_backtrace
none
File: cpuinfo
none
File: dso_list
none
File: environ
none
File: limits
none
File: maps
none
File: mountinfo
none
File: open_fds
none
File: proc_pid_status none

Description Brian J. Murrell 2018-07-12 11:37:25 UTC 
Description of problem:
Not sure what caused this

Version-Release number of selected component:
evolution-data-server-3.28.3-1.fc28

Additional info:
reporter:       libreport-2.9.5
backtrace_rating: 4
cmdline:        /usr/libexec/evolution-source-registry
crash_function: g_wakeup_new
executable:     /usr/libexec/evolution-source-registry
journald_cursor: s=bc64b0deac69405ca8d741b6730dac86;i=5d3072;b=5d70b12ce1b34c1a9b7e05d6df73d3f7;m=f8fb47dc21;t=570c0feb00412;x=cc77da3fdd5d3db6
kernel:         4.17.2-200.fc28.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1001

Truncated backtrace:
Thread no. 1 (8 frames)
 #4 g_wakeup_new at gwakeup.c:161
 #5 g_main_context_new at gmain.c:656
 #6 g_dbus_connection_send_message_with_reply_sync at gdbusconnection.c:2138
 #7 g_dbus_connection_call_sync_internal at gdbusconnection.c:5950
 #8 g_dbus_connection_call_with_unix_fd_list_sync at gdbusconnection.c:6299
 #9 g_dbus_proxy_call_sync_internal at gdbusproxy.c:2870
 #10 g_dbus_proxy_call_sync at gdbusproxy.c:3062
 #11 0x00007f0086cf436b in

Potential duplicate: bug 1394206
Comment 1 Brian J. Murrell 2018-07-12 11:37:31 UTC 
Created attachment 1458353 [details]
File: backtrace
Comment 2 Brian J. Murrell 2018-07-12 11:37:32 UTC 
Created attachment 1458354 [details]
File: cgroup
Comment 3 Brian J. Murrell 2018-07-12 11:37:33 UTC 
Created attachment 1458355 [details]
File: core_backtrace
Comment 4 Brian J. Murrell 2018-07-12 11:37:34 UTC 
Created attachment 1458356 [details]
File: cpuinfo
Comment 5 Brian J. Murrell 2018-07-12 11:37:36 UTC 
Created attachment 1458357 [details]
File: dso_list
Comment 6 Brian J. Murrell 2018-07-12 11:37:37 UTC 
Created attachment 1458358 [details]
File: environ
Comment 7 Brian J. Murrell 2018-07-12 11:37:38 UTC 
Created attachment 1458359 [details]
File: limits
Comment 8 Brian J. Murrell 2018-07-12 11:37:39 UTC 
Created attachment 1458360 [details]
File: maps
Comment 9 Brian J. Murrell 2018-07-12 11:37:41 UTC 
Created attachment 1458361 [details]
File: mountinfo
Comment 10 Brian J. Murrell 2018-07-12 11:37:42 UTC 
Created attachment 1458362 [details]
File: open_fds
Comment 11 Brian J. Murrell 2018-07-12 11:37:43 UTC 
Created attachment 1458363 [details]
File: proc_pid_status
Comment 12 Milan Crha 2018-07-13 06:02:10 UTC 
Thanks for a bug report. According to the backtrace:
> Creating pipes for GWakeup: Too many open files
it looks like there had been opened too many file descriptors. There used to be similar issues in the past, but they had been fixed some time ago. If there are actions/steps which increase (and do not free) opened file descriptors, then those can be found for example by running:
   $ lsof -p `pidof evolution` | wc -l
before the action and then after it. Redirecting the lsof output to a file
   $ lsof -p `pidof evolution` >/tmp/before
and similarly for "after":
   $ lsof -p `pidof evolution` >/tmp/after
and then compare the two:
   $ diff -u /tmp/before /tmp/after
will show what changed.
Comment 13 Milan Crha 2018-07-13 06:04:00 UTC 
Oops, except this is not evolution, but evolution-source-registry process. My fault, I'm sorry. The fixes I've been talking about were done in evolution, thus it had been unrelated to your issue.
Comment 14 Brian J. Murrell 2018-07-13 11:04:29 UTC 
$ lsof -p `pidof evolution-source-registry` | tail
evolution 2523 brian   87u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   88u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   89u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   90u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   91u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   92u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   93u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   94u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   95u  a_inode               0,13         0    10694 [eventfd]
evolution 2523 brian   96u  a_inode               0,13         0    10694 [eventfd]
$ lsof -p `pidof evolution-source-registry` | grep eventfd | wc -l
86

for a data point.

86 open eventfds is not huge in the context of the 1024 nofiles soft limit, but perhaps it's excessive for what the process does regardless.  I guess you will have to be the judge of that.
Comment 15 Milan Crha 2018-07-17 12:35:32 UTC 
Thanks for the update. I tried to call 'Refresh' on the collection backend, which basically re-populates the sources for collections, and every time I do that it increases the eventfd counter by one, thus there is leaking something. It might no the noticeable from the beginning, but the longer you have the evolution-source-registry running and the more the machine goes online and offline for such collection backends, the more file handles are leaked, which reaches the limit after some time.

I'm searching what exactly causes the leak.
Comment 16 Milan Crha 2018-07-18 14:04:37 UTC 
This was tricky, because evolution-data-server didn't do anything wrong, as far as I can tell. The problem was that EWebDAVDiscover (used to find out CalDAV calendars/memo lists/task lists and CardDAV books) wanted to receive a proxy ESource, for which it requires either ESourceRegistry or ESourceRegistryServer. As the EWebDAVDiscover API didn't have any such argument it always created its own ESourceRegistry instance, which had been dropped shortly afterwards. This drop also meant that the internal GDBus object scheduled its own events on the ESourceRegistry's thread main_context (to unsubscribe D-Bus signals), but this main_context didn't have enough time to flush the queue, which resulted in a leak of the main_context, because the events in it referenced it (similar to circular reference). The fix was to flush the queue before closing the thread and to reorganize the ESourceRegistry's dispose() function, thus the flush is done on filled main_context.

While I've been in it, some structures unnecessarily long held a GCancellable object, which also means a file descriptor, thus I fixed that as well.

As the last, but not least, I added a new API for the EWebDAVDiscover to be able to provide also a function to reference an ESource, thus there is no need to create the ESourceRegistry instance, thus it is quicker and more effective now.

Created commit 98f133652 in eds master (3.29.90+) [1]
Created commit ed6149387 in eds gnome-3-28 (3.28.5+)

[1] https://gitlab.gnome.org/GNOME/evolution-data-server/commit/98f133652