Bug 1476329 - Deadlock on exit under e-categories.c:finalize_categories()
Deadlock on exit under e-categories.c:finalize_categories()
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: evolution-data-server (Show other bugs)
7.4
Unspecified Unspecified
unspecified Severity high
: rc
: ---
Assigned To: Milan Crha
Desktop QE
:
Depends On:
Blocks: 1477211
  Show dependency treegraph
 
Reported: 2017-07-28 12:06 EDT by Oliver Haessler
Modified: 2018-04-10 12:36 EDT (History)
4 users (show)

See Also:
Fixed In Version: evolution-data-server-3.22.7-7.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-10 12:35:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
29801.txt (3.77 KB, text/plain)
2017-07-31 06:21 EDT, Oliver Haessler
no flags Details
29335.txt (3.77 KB, text/plain)
2017-07-31 06:22 EDT, Oliver Haessler
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0912 None None None 2018-04-10 12:36 EDT

  None (edit)
Description Oliver Haessler 2017-07-28 12:06:55 EDT
Description of problem:
After running my notebook for ~ 1 week withour reboot, I run out of mmemory (including swap). I have 16GB RAM and 8GB Swap.
I checked the processes with: 

ps aux  | awk '{print $6/1024 " MB\t\t" $11}'  | sort -n

and on htop, and found:

1.97266	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.04297	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.12109	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.125	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.17969	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.18359	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.21484	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.25391	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.25781	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.28516	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.35938	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
2.36719	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
4.14062	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
4.60156	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
5.17969	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
5.99609	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
6.05078	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
6.22266	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
7.11719	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
8.28125	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
8.71484	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
10.2266	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
13.2969	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
14.2734	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
16.4375	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
24.5352	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
65.2344	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
102.445	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
116.438	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
121.754	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
124.457	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
145.469	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
156.531	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
164.168	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
166.887	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
168.828	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
169.066	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
169.805	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
170.168	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
170.504	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
184.777	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
187.465	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
190.117	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
190.426	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
190.43	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
190.492	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
190.727	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
190.766	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
190.801	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.102	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.586	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.691	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.711	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.711	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.879	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.906	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.973	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.98	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.988	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.992	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
191.996	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
192.008	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
192.184	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"
214.148	"MB    /usr/libexec/evolution-addressbook-factory-subprocess"

The process lines looked like this (example):
19793  0.0  0.1 1739860 25120 ?       Sl   Jul25   0:16 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx5357x47 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/5357/47

After killing /usr/libexec/evolution-addressbook-factory-subprocess I had 6GB RAM free and ~6GB Swap


Version-Release number of selected component (if applicable):
evolution-3.22.6-10.el7.x86_64
evolution-data-server-3.22.7-6.el7.x86_64
evolution-ews-3.22.6-6.el7.x86_64
evolution-help-3.22.6-10.el7.noarch
evolution-mapi-3.22.6-1.el7.x86_64

Steps to Reproduce:
1. Let computer run (suspend/resume) over some time

Actual results:
process eats a lot of RAM

Expected results:
well.. do not eat all my ram :D

Additional info:
I use Gnome Online Accounts with a company Gmail account.
Comment 2 Milan Crha 2017-07-31 04:36:41 EDT
Thanks for a bug report. If I understand that `ps` command properly, then it reports that you have that many evolution-addressbook-factory-subprocess processes running, which is rather odd, because evolution-addressbook-factory runs one for each factory type (the --factory argument in the `ps` output), without "duplicates", and uses it until each client connecting to it is closed/freed.

Could the suspend/resume caused any misbehaviour here? I'll try to reproduce and will return back to you when I know more.
Comment 3 Oliver Haessler 2017-07-31 05:45:11 EDT
It might be a suspend/resume issue.. so over the weekend I have this again now:

ps aux | grep /usr/libexec/evolution-addressbook-factory-subprocess | grep -v grep

ohaessle   561  1.1  1.3 1674348 221592 ?      Sl   11:30   0:09 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx537x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/537/2
ohaessle 18506  0.0  1.3 1674348 221584 ?      Sl   Jul28   0:10 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx18493x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/18493/2
ohaessle 18997  0.0  1.3 1674348 221588 ?      Sl   Jul28   0:18 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx18973x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/18973/2
ohaessle 22163  0.0  1.3 1674344 221592 ?      Sl   Jul28   0:17 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx22145x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/22145/2
ohaessle 23460  0.0  1.3 1674348 221548 ?      Sl   Jul28   0:16 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx23435x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/23435/2
ohaessle 28155  0.1  1.3 1674352 221512 ?      Sl   10:19   0:08 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx28133x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/28133/2
ohaessle 28748  0.2  1.3 1674352 221676 ?      Sl   10:28   0:09 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx28737x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/28737/2
ohaessle 28993  0.2  1.3 1674384 221616 ?      Sl   10:29   0:10 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx28984x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/28984/2
ohaessle 29335  0.2  1.3 1674344 221552 ?      Sl   10:33   0:09 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx29321x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/29321/2
ohaessle 29801  0.4  1.3 1674388 221648 ?      Sl   10:37   0:19 /usr/libexec/evolution-addressbook-factory-subprocess --factory google --bus-name org.gnome.evolution.dataserver.Subprocess.Backend.AddressBookx29782x2 --own-path /org/gnome/evolution/dataserver/Subprocess/Backend/AddressBook/29782/2
Comment 4 Milan Crha 2017-07-31 06:09:48 EDT
I tried it here, by installing suspend-button extension in the GNOME Shell in my virtual machine (VM) with RHEL, thus I did suspend it and when I woke the machine up there was no duplicated addressbook subprocess for the Google book. I didn't change networking or anything like that, it was the VM.

Could you grab a backtrace of any of the two running subprocesses, please? I'd like to see what they are doing. Ideally to pick the latest and the oldest, but it can be tricky to do due to the PID number not increasing monotonically. Getting a backtrace of evolution-addressbook-factory would be also interesting. Please install debuginfo package for evolution-data-server (and only for it), to get better backtrace. Make sure the debuginfo package version will match the version of the main package.

You can get the backtrace with command like this:
   $ gdb --batch --ex "t a a bt" -pid=PID &>bt.txt
where the PID is replaced with the actual process ID. Please check the bt.txt for any private information, like passwords, email address, server addresses,... I usually search for "pass" at least (quotes for clarity only). You might want to rename the bt.txt based on the process you capture the backtrace for.
Comment 5 Oliver Haessler 2017-07-31 06:21 EDT
Created attachment 1306998 [details]
29801.txt
Comment 6 Oliver Haessler 2017-07-31 06:22 EDT
Created attachment 1307000 [details]
29335.txt
Comment 7 Oliver Haessler 2017-07-31 06:22:39 EDT
added to backlogs for the 2 newest PIDs
Comment 8 Milan Crha 2017-07-31 10:52:46 EDT
Thanks for the update. It shows that the process is stuck, when saving categories on exit. I do not know why it doesn't strike here, but it is a valid issue also in the upstream code.

I fixed it upstream with this commit:
https://git.gnome.org/browse/evolution-data-server/commit/?id=35ae7bc
which landed for 3.25.90+ and 3.24.5+.

Here is a test build with it included for RHEL 7.4:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13746532
It requires restart after being installed, thus the background process will use the new bits.

I believe a workaround would be to run evolution in the Contacts view and select all of the Google address books you might see there, thus for example from a terminal:

   $ evolution -c contacts

and then select the books with a mouse. After it wait until the books are fully populated. Then the categories changes will be saved to the disk, and you might see them in the Show: combo box as well (above the list with the contacts), but I'd like to test the fix first, thus the workaround is mentioned here just in case it won't work.
Comment 9 Oliver Haessler 2017-08-08 03:07:16 EDT
After testing a while, I can confirm that the patch is working.

Do we know when we can expect the patch to be in RHEL 7.4, so I can start deploying it on workstations?
Comment 10 Milan Crha 2017-08-21 12:04:06 EDT
(In reply to Oliver Haessler from comment #9)
> After testing a while, I can confirm that the patch is working.

Thanks for testing and confirmation.

> Do we know when we can expect the patch to be in RHEL 7.4, so I can start
> deploying it on workstations?

It missed 7.4, but it's approved for 7.5.
Comment 15 errata-xmlrpc 2018-04-10 12:35:59 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0912

Note You need to log in before you can comment on or make changes to this bug.