Description of problem: This happens to me for some time (definitely before upgrade to F30). Evolution is leaking a lot. I have 1 hour uptime (and I started evolution right after start) and the evolution process takes 1,9 GiB (2053 MB of reserved memory). This was about 500 MB 40 minutes ago. I have three accounts (two gmail and one IMAP). Gmail ones has also calendar and everything connected to gmail. Version-Release number of selected component (if applicable): evolution-3.32.0-1.fc30.x86_64 How reproducible: I don't check it everytime but when I do it takes looooot of memory Steps to Reproduce: 1. Start evolution with some accounts set 2. Wait 3. Actual results: Evolution takes excessive amount of memory Expected results: Shouldn't leak over time Additional info:
Thanks for a bug report. Is there any evidence Evolution is leaking memory, or you only think that? Could you run evolution from a terminal as: $ GIGACAGE_ENABLED=0 evolution and see whether anything improves, please? The gigacage is a thing of the WebKitGTK+, not Evolution, thus anything what uses WebKitGTK+ might be "affected". To verify whether your environment really leaks a lot of memory you can run evolution under valgrind with these commands (the gigacage disable is needed to have valgrind started, without it it aborts due to too large memory block being requested by the WebKitGTK+): $ export GIGACAGE_ENABLED=0 $ G_SLICE=always-malloc valgrind --leak-check-full --num-callers=30 --show-leak-kinds=definite evolution &>log.txt then try to use it for some time. I wrote "try to", because all the memory checking valgrind does takes significant time and uses a lot of CPU, thus everything is more than noticeably slower. To have the log useful it's a good idea to install debuginfo packages for evolution-data-server and evolution at least (definitely do __not__ install debuginfo for webkit2gtk3). You should also close the application gracefully, otherwise the log will not contain the correct data.
Hi, I run it with valgrind and GAGACAGE_ENABLED=0 for a while and it still seems to eat memory a lot. You can look at output from valgrind in attachments.
Created attachment 1551396 [details] Output of valgrind
Hi, was data in comment 3 enough or should I send something else? I just stored memory usage over time for last 24 hours. And in first 9 hours it raised to 5 GB and then it stop. So for last 15 hours the memory usage is still around 5 GB. It's still quite a lot for such application (which is supposed to run whole time).
I'm sorry for the delay, I'm currently busy with some other things. I've this in my todo. I'll get to you soon, maybe by the end of this week, or the next week the latest, I hope. I agree that 5GB is quite a lot of memory for the mail application and it should not do it. What is your webkit2gtk3 version, please?
Thank you for answer. I have webkit2gtk3-2.24.0-1.fc30.x86_64 installed. I'd like to add to comment 4, that the memory raises just when I use the application (it has 6 GB now after another 2 hours).
By using it, do you mean just reading mails in various folders, or also writing and sending mails? Eventually also using different parts of the Evolution, not only the Mail part?
I had a brief look into your log and it shows some weird things (at least weird to me), like this: > e_util_is_running_gnome (e-misc-utils.c:3466) is there quite many times, but the function is supposed to evaluate the value once and not re-do it again. The most leaking part comes from glib-networking, at least according to the valgrind, but maybe it's a false positive. Do you recall whether evolution stopped gracefully when you've been running it under valgrind? I'll install Fedora 30 and will try it here. It'll be easier than chasing you for the logs when I can make them on my own. My use case would not be the same as yours, but I can at least identify some common leaks.
Just from observation - the usage raises when during hours I read mails often. I wrote few mails (really just about 5). And I barely use another functionality. Once or twice I went to Calendar tab. Yes, evolution ended gracefully when I run it under valgrind. Just now I run evolution for 4 hours with updated webkit (webkit2gtk3-2.24.1-1.fc30.x86_64) and the memory usage raised from 150 MB in first 6 minutes, 1.5 GB in hour and half to 3 GB in 4 hours. I was just reading mails in different folders and accounts (2 gmails and 1 IMAP).
I'm afraid this is related to glib2, at least according to my tests. It's hard to downgrade glib in Fedora 30 (due to dependencies), thus I tried this in Fedora 28 in the other way. When I ran evolution with the glib2-2.60.0-3, the resulting statistic was: ==31761== LEAK SUMMARY: ==31761== definitely lost: 3,445,506 bytes in 125,551 blocks ==31761== indirectly lost: 7,547,818 bytes in 158,791 blocks ==31761== possibly lost: 7,506,950 bytes in 7,084 blocks ==31761== still reachable: 53,620,749 bytes in 488,402 blocks ==31761== of which reachable via heuristic: ==31761== length64 : 25,854,950 bytes in 50,413 blocks ==31761== newarray : 2,832 bytes in 97 blocks ==31761== suppressed: 0 bytes in 0 blocks When I downgrade it to glib2-2.56.4-1 (note I didn't do anything else, I only downgraded glib2 packages, aka glib2, glib2-devel and glib2-doc), the resulting statistic was like this: ==32751== LEAK SUMMARY: ==32751== definitely lost: 26,571 bytes in 26 blocks ==32751== indirectly lost: 59,700 bytes in 2,363 blocks ==32751== possibly lost: 7,380,534 bytes in 3,278 blocks ==32751== still reachable: 68,420,179 bytes in 773,541 blocks ==32751== of which reachable via heuristic: ==32751== length64 : 25,854,957 bytes in 50,413 blocks ==32751== newarray : 2,752 bytes in 92 blocks ==32751== suppressed: 0 bytes in 0 blocks while I basically only ran evolution, clicked Send/Receive, and closed it in both cases. --------------------------------------------------------------------------------------------------------------- I wanted to try with something smaller, for which I picked gedit-3.28.1-1 (for simplicity with the test machine I used). I used this command: $ G_SLICE=always-malloc valgrind --leak-check=full --num-callers=30 --show-leak-kinds=definite gedit and when it started I pressed: a b c <Alt+F4> <Tab> <Space> The results with glib2-2.56.4-1 are: ==871== LEAK SUMMARY: ==871== definitely lost: 29,134 bytes in 39 blocks ==871== indirectly lost: 65,307 bytes in 2,711 blocks ==871== possibly lost: 7,011 bytes in 74 blocks ==871== still reachable: 4,217,229 bytes in 51,976 blocks ==871== of which reachable via heuristic: ==871== length64 : 14,496 bytes in 213 blocks ==871== newarray : 2,432 bytes in 72 blocks ==871== suppressed: 0 bytes in 0 blocks The results with glib2-2.60.0-3 are: ==1466== LEAK SUMMARY: ==1466== definitely lost: 115,442 bytes in 2,228 blocks ==1466== indirectly lost: 200,862 bytes in 5,603 blocks ==1466== possibly lost: 6,051 bytes in 55 blocks ==1466== still reachable: 3,900,214 bytes in 47,193 blocks ==1466== of which reachable via heuristic: ==1466== length64 : 14,496 bytes in 213 blocks ==1466== newarray : 2,432 bytes in 72 blocks ==1466== suppressed: 0 bytes in 0 blocks which is significantly more. The only change between the two runs was a call to `dnf update ./glib2-*.rpm` (which updated glib2, glib2-devel and glib2-doc). Nothing else changed. I'm moving this to glib2 and I'll also file an upstream bug for further investigation.
I filled this one upstream: https://gitlab.gnome.org/GNOME/glib/issues/1749
The upstream bug contains a hint how to workaround the issue with the GHashTable. The new valgrind command would look like this: $ export GIGACAGE_ENABLED=0 $ G_SLICE=always-malloc valgrind --leak-check=full --num-callers=30 --show-leak-kinds=definite --aspace-minaddr=0x100000000 evolution &>log.txt and if you've installed also glib2-devel package, you can add an argument to valgrind: --suppressions=/usr/share/glib-2.0/valgrind/glib.supp to hide some of the known leaks (mostly global memory pools, I guess). I tried it here and it prints more sane result (even without the suppressions argument). By the way, do you read messages in the Mail window (beside the message list) or you read them in a separate window, like double-click on them? I noticed one leak here: ==2692== 196 (72 direct, 124 indirect) bytes in 1 blocks are definitely lost in loss record 26,184 of 28,337 ==2692== at 0x10083880B: malloc (vg_replace_malloc.c:309) ==2692== by 0x1014FEDC8: g_malloc (in /usr/lib64/libglib-2.0.so.0.6000.1) ==2692== by 0x1015171E5: g_slice_alloc (in /usr/lib64/libglib-2.0.so.0.6000.1) ==2692== by 0x10151780D: g_slice_alloc0 (in /usr/lib64/libglib-2.0.so.0.6000.1) ==2692== by 0x101368E45: g_settings_schema_source_lookup (in /usr/lib64/libgio-2.0.so.0.6000.1) ==2692== by 0x1045B4144: web_view_initialize (e-web-view.c:1274) which is called whenever a new window is created. It's nothing significant, but it can pile up during the time. I fixed it with [1] for evolution 3.33.2+ and 3.32.3+. I made a scratch build [2] for you (it depends on 3.32.2, which is in updates-testing currently) and I'd like to ask you to test with it. Do not forget to install also the debuginfo/debugsource packages, thus the valgrind output is usable. Thanks in advance. [1] https://gitlab.gnome.org/GNOME/evolution/commit/5535ba1393 [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=34742877
Created attachment 1568261 [details] Output of valgrind - 3.32.2-1 I tested with build from comment 12 (evolution-3.32.2-1.1.fc30.x86_64). I run it with valgrind for a while (without suppressions argument) and it still leaks a bit. When I closed the app it took about 1.1 GB of memory. I then run it without valgrind and it takes about 3.3 GB after one day of normal usage (it got to 3 GB in just 4 hours). I don't read emails in separate window but in the main window (beside list of mails) - I use classic view and 'show message preview'. Only time I open new window is when I write a mail (but I wrote about 2 mails during testing).
Thanks for the update. Looking into the log, its final statistic says: ==23190== LEAK SUMMARY: ==23190== definitely lost: 61,996 bytes in 293 blocks ==23190== indirectly lost: 46,878 bytes in 1,712 blocks ==23190== possibly lost: 6,269,956 bytes in 1,677 blocks ==23190== still reachable: 37,090,902 bytes in 533,194 blocks ==23190== of which reachable via heuristic: ==23190== length64 : 1,394,334 bytes in 5,748 blocks ==23190== newarray : 3,072 bytes in 112 blocks ==23190== suppressed: 0 bytes in 0 blocks I see some leaks from libEGL: ==23190== 5,385 bytes in 255 blocks are definitely lost in loss record 30,011 of 30,375 ==23190== at 0x10083880B: malloc (vg_replace_malloc.c:309) ==23190== by 0x10471B91E: strdup (in /usr/lib64/libc-2.29.so) ==23190== by 0x120DC59F6: ??? ==23190== by 0x120DC57A4: ??? ==23190== by 0x120D79FC8: ??? ==23190== by 0x10A550761: ??? (in /usr/lib64/libGLdispatch.so.0.0.0) ==23190== by 0x10A550F87: __glDispatchMakeCurrent (in /usr/lib64/libGLdispatch.so.0.0.0) ==23190== by 0x1061B7FA3: ??? (in /usr/lib64/libEGL.so.1.1.0) ==23190== by 0x10313A701: ??? (in /usr/lib64/libwebkit2gtk-4.0.so.37.37.3) though the caught backtrace suggests it's done only once, when the web view is created. There are plenty of these: ==23190== 7,282 (512 direct, 6,770 indirect) bytes in 1 blocks are definitely lost in loss record 30,104 of 30,375 ==23190== at 0x10083AD19: realloc (vg_replace_malloc.c:836) ==23190== by 0x10564D3D8: ??? (in /usr/lib64/libfontconfig.so.1.12.0) ==23190== by 0x10564D9F8: ??? (in /usr/lib64/libfontconfig.so.1.12.0) which is semi-known. And then a set of these: ==23190== 3,575 (448 direct, 3,127 indirect) bytes in 8 blocks are definitely lost in loss record 29,872 of 30,375 ==23190== at 0x10149CFAF: g_type_create_instance (in /usr/lib64/libgobject-2.0.so.0.6000.2) which may or may not claim real GObject-s. The log itself doesn't show any significant memory leak, from my point of view. It can be due to the arguments, where the command I gave you shows only memory leaks about which valgrind believes are "real memory leaks" (aka the definitely lost memory). Those "still reachable" are blocks which have reference within some other object or global memory. According to my experience, the global caches can grow, but evolution uses some limits (even with objects needed to view folder content, it's held in memory for 5 minutes and freed if unused), while definitely lost memory can grow ad infinity, especially when it's in a code part which is called often. Some objects can be reachable, because certain "free global cache" code didn't run properly at the application exit for some reason. That said, having more than 35MB still reachable at the application exit is suspicious and can be related to not freed memory of some global object. I do not recall whether there had been any major change in this regard since evolution 3.30.x, the global objects are handled the same way as in 3.30.x, as far as I know. It's possible some code change caused an issue, I do not have any evidence in either way. I'll try to find out other ways to check for memory leaks and I'll test them here. By the way, as you mostly read mails, are there many HTML messages with external or attached images? And if there are any, do you show them by default (Edit->Preferences->Mail Preferences->HTML Messages tab->Loading remote content section) or on demand, if at all? The valgrind doesn't suggest there is any leak in this regard, but I know there had been some in the past and it's possible the messages you viewed during the run under valgrind didn't catch other parts of the code with messages you viewed without valgrind.
I recompiled glib2 with --buildtype=debug option added and then ran: $ export GOBJECT_DEBUG=objects $ evolution and in printed at the end of the application all the alive GObject-s (it's a bit tricky due to fork() and the WebKit-related processes to be covered too). The result was that after viewing 6 mails, I've got a list of 1981 stale objects. Those related to the libcamel part (which is used for the Mail view) were only 23 (all of them with ref_count=1), from which 6 being message infos, which are those shown in the message list. After some testing I realized that those not freed are also those viewed, which sounds like a clue. For a compare, Gtk-related stale objects at the end of the application were 819 in this run and those related to evolution and evolution-data-server (type name with prefix 'E' or 'Gal') were 441. Some of these can be related, because some objects in the evolution code are held/referenced by a GtkWidget subclass. Let's aim on the message infos first. If you read a log of messages, it can mean a lot of memory in a long run.
s/a log of messages/a lot of messages/
(In reply to Milan Crha from comment #15) > Let's aim on the message infos first. I fixed that leak with commit [1], for 3.33.2+ and 3.32.3+. Here [2] is a scratch build with it. I currently do not see anything more repetitive in the logs, but I might overlook something (more or less obvious) too, thus it would be good if you could give it a try. For the record, the leak happened in a way that if the message had not been already downloaded (where new messages do not use to be), then before the download started the message info had been received to check the message size, but it was not unreferenced, which led to a situation where the info had been in memory until the end of the application. This fixes it and makes them freed once they are not needed any more (like after ~5 minutes of being away in the UI from the folder they belong to). A funny part of this is that each error path in that function did unreference the message info as expected, only the "everything works as expected" path didn't do it. [1] https://gitlab.gnome.org/GNOME/evolution-data-server/commit/d7abfa369 [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=34848390
Created attachment 1568957 [details] Output of valgrind - e-d-s 3.32.2-1.1 I tested with build from comment 17. Not sure if something changed - the memory usage still raises a lot over time.
Thanks for the update. What was the running time for the log, please? It doesn't show anything significant (what would repeat in the time), except of this one: ==411== 128 bytes in 4 blocks are definitely lost in loss record 24,259 of 27,070 ==411== at 0x10083880B: malloc (vg_replace_malloc.c:309) ==411== by 0x101517DC8: g_malloc (in /usr/lib64/libglib-2.0.so.0.6000.2) ==411== by 0x1015301E5: g_slice_alloc (in /usr/lib64/libglib-2.0.so.0.6000.2) ==411== by 0x1014E22F6: g_ptr_array_sized_new (in /usr/lib64/libglib-2.0.so.0.6000.2) ==411== by 0x1014E23A2: g_ptr_array_new_with_free_func (in /usr/lib64/libglib-2.0.so.0.6000.2) ==411== by 0x11CF26666: imapx_parse_fetch (camel-imapx-utils.c:2121) ==411== by 0x11CF0C5E9: imapx_untagged_fetch (camel-imapx-server.c:1113) ==411== by 0x11CF0AD2F: imapx_untagged (camel-imapx-server.c:2228) ==411== by 0x11CF0E6C3: imapx_step (camel-imapx-server.c:2619) ==411== by 0x11CF0E6C3: camel_imapx_server_process_command_sync (camel-imapx-server.c:4095) ==411== by 0x11CF12E2C: imapx_server_fetch_changes (camel-imapx-server.c:5336) It's a minor leak, only 128 bytes, but if your receive a lot of messages, then it can grow in time. The other things looks non-significant and/or known. I do not know whether the fontconfig leaks are repetitive or a one time thing only; these are there for quite long time now. The other used memory you face might involve some global cache, or something like that, I'm afraid. By the way, where do you check the memory usage, in the System Monitor? Do you run GNOME, or other desktop environment. If GNOME, is it Wayland or Xorg version? I've evolution running for almost two hours and it says: Memory 375.5 MiB Virtual Memory 5.9 GiB Resident Memory 371.8 MiB I plan to upgrade to Fedora 30 soon, after which I might get better (more recent) numbers.
Okay, I saw resident memory usage being increased while moving between folders in the Mail view, thus I added some debug prints on places which were related, but it didn't help to claim anything. My IMAP account has set both Trash and Junk folders as real folders, which also means that moving away from any folder immediately frees memory used by the message info-s of that folder (I have it confirmed with the added debug prints). The memory allocated by sqlite (according to sqlite3_memory_used()) is approximately at 31MB here, with no increment over time. I also enabled more information from valgrind and it did not reveal anything usable (or I didn't notice anything suspicious in the log at least). It can mean that moving between folders also adds some memory into some global (or object's) cache, which is freed on exit, thus it's not noticed by valgrind. I still do not know.
I added some debug prints into glib2 and evolution-data-server, in a hope to discover weak points, but I'm not much sure what to think out of it. My reproducer was to enter an IMAP folder with more than 187K messages, then move away from it. The IMAP account is configured to use real Trash and Junk folders (it matters, because if one of them is virtual, the folder is left in the memory till the end of the application, though the message infos are freed after ~5 minutes, if not needed; with both real folders the folder is freed as soon as it's not needed, like when moving away from it). My debug prints in glib2 included memory allocation statistics for memory allocated through glib itself. When I enter the folder, the memory allocated by glib increases by ~150MB, which is reflected in the Resident Memory usage of the 'evolution' process in the System Monitor. When I move away from that folder, the memory allocated through glib decreases by the same amount, for me to ~25MB in total. The resident memory usage doesn't decrease that much, but when I enter the large folder again, the Resident Memory usage doesn't increase by 150MB, but less, which makes me believe some memory is reused. I do not know why the memory is not reused in a better way. Can it be due to large fragmentation? I'm afraid evolution(-data-server) cannot do much about it, but I do not know these low level things (may it come down to kernel?), thus I'm open for suggestions. These are only guesses, from what I see. I'll try to write a little application as a proof of the idea.
Created attachment 1575394 [details] simple reproducer This is a simple reproducer, a mem.cpp file. The first line contains a comment with a command line to compile and run it. It tries to simulate memory allocation as done by Evolution (or, in this case, more like the libcamel from the evolution-data-server), which means many smaller blocks, thus quite fragmented memory. It contains also a way to allocate large blocks. My observation is as follows (the resident and virtual memory are more or less in sync, thus when I say 'memory', I mean both): a) * allocate 4 large blocks (letter 'b' 4 times) - the memory in the system monitor increases accordingly (there is used more memory due to tracking structures) * free half of the allocated objects (letter 'h') - the memory usage decreases in the system monitor accordingly * free everything (letter 'f') - again, the memory usage decreases almost to zero (I suspect the left memory is also held by the tracking structs). b) * allocate many small blocks (letter 'a' once) - the memory increases accordingly * free half of the blocks (letter 'h') - the freed memory is not the same as that in the system monitor. The in-program claims that 45MB are allocated, but the system monitor shows more than 70MB memory being used * free everything (letter 'f') - memory usage in the system monitor decreased to a bit more than 44MB, which can be slightly diverged with the tracking structures, but not that much for sure * allocated one big block (letter 'b') - memory usage increases accordingly * free all (letter 'f') - memory usage decreases accordingly, but still ends on 44MB, even the application freed all its allocated blocks c) * use letters "abaaaahbababbah" (it can be copy&pasted into the terminal) - the result is that the application claims 364MB being allocated, but the system monitor says 650MB * free everything (letter 'f') - the application says 0MB allocated, but the system monitor has left 406MB My guess from the above is that the system doesn't like many small chunks of memory and doesn't free them accordingly. It even doesn't know how to "glue them together" and reuse them, at least when allocating a large block in the end of the c) leads to a new allocation of it, though the process has more than 400MB of the free memory (in small freed chunks). I can be wrong with my guess, it's just what it looks like from the use of the test program. I also noticed that malloc can switch to mmap when the allocated block is too large, thus it's possible the large block data allocated by this test application is not the same as the small chunks allocation.
I just noticed malloc_stats(), which prints the statistic of the memory used by malloc and the values there correspond with the resident memory reported by the system monitor. It looks like the behaviour of not freeing assigned resident memory (not that from mmap) is expected behaviour.
I've been trying to follow this procedure to get to the bottom of what looks like memory leaks in my evolution environment but it seems quite impractical. First of all, evolution is very painful to even try to use while running with valgrind so the hopes of reproducing the full array of activities one normally does are quite low. But much more impractical is that fact that now, even 24 hours after I've "File->Quit"ted the evolution window, and despite the fact that the main window has disappeared, the process has not exited and printed the summary of allocations, leaks, etc. It has not updated the stderr/stdout since quitting even. It has beem chewing up the full capacity of a core for those 24 hours, but nothing seems to be actually happening. Is this normal?
(In reply to Brian J. Murrell from comment #24) > Is this normal? No, it's not. Do you have webkit2gtk3 debuginfo installed? That's one of the biggest I know of and it takes a lot of time to parse it (to gdb, valgrind,...). I do not know whether it's related or not, I also do not expect it being a culprit on quit, it's more visible on startup.
I do, but as is typical during system upgrades (i.e. F30->F31) the debug* RPMs didn't get upgraded: webkit2gtk3-2.26.1-1.fc31.x86_64 webkit2gtk3-debuginfo-2.24.4-1.fc30.x86_64 webkit2gtk3-debugsource-2.24.4-1.fc30.x86_64 webkit2gtk3-jsc-2.26.1-1.fc31.x86_64 But this taking a long time after exit happened with F30 also where those debug* packages would have been properly aligned. How long should valgrind take to do process after evolution is exited?
From our chat on IRC, I would not let valgrind chew my CPU for longer than 10-15 minutes, definitely not for a day or two.
Created attachment 1632662 [details] valgrind output I ended up having to kill -KILL the valgrind process. The output it accumulated is attached.
Created attachment 1632789 [details] valgrind output Another run, with only evolution, evolution-rss and e-d-s debuginfo installed. Had to kill this with a -KILL again so I don't know of how much use it will be. Would be good to know why valgrind is not exiting properly when evolution is exited.
Created attachment 1633030 [details] valgrind output Latest valgrind after evolution-rss update. Still had to kill -KILL it to get it to stop.
You are right that the log is not that great, when valgrind could not close as it should. The last log shows the issues from evolution-rss being gone, at least the major one (there is one more, but only about uninitialized value in mktime call, which can just create things differently in some cases than earlier with the same arguments).
I made series of commits which would help with memory usage in the mail view, the biggest part related to the content of the message list. These changes will be part of the upcoming 3.35.90 development release, thus 3.36.0. I currently do not have much idea what to focus on next, also because some memory increase is caused by memory allocator (comment #22). I'm closing this in favour of 3.36.0.