Bug 1354074 - Possible big memory leak in packagekit
Summary: Possible big memory leak in packagekit
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: PackageKit
Version: 29
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: Richard Hughes
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords: Reopened
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-09 02:43 UTC by Marcelo Ricardo Leitner
Modified: 2018-11-29 12:16 UTC (History)
17 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-11-28 14:25:44 UTC


Attachments (Terms of Use)
/proc/../smaps (255.52 KB, text/plain)
2016-07-09 11:55 UTC, Marcelo Ricardo Leitner
no flags Details
/proc/.../maps (40.13 KB, text/plain)
2016-07-09 11:56 UTC, Marcelo Ricardo Leitner
no flags Details
valgrind output (2.93 MB, text/plain)
2016-07-14 11:10 UTC, Marcelo Ricardo Leitner
no flags Details

Description Marcelo Ricardo Leitner 2016-07-09 02:43:39 UTC
Description of problem:
Current status is:
$ uptime
 23:40:19 up 4 days,  4:42,  1 user,  load average: 0,36, 0,42, 0,57
yet:
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                               
 1577 root      20   0 1073368 434596  17800 S   0,0  3,6   0:32.10 /usr/libexec/packagekitd              
 1989 mrl       20   0 2017828 322332  57916 S   4,3  2,7 249:21.72 /usr/bin/gnome-shell                  

This is sorted by the biggest memory users in my system. I don't like but I can understand gnome-shell using that much but seems packagekit is just using too much memory.

I'm using F24, updated from a F23.

Version-Release number of selected component (if applicable):
PackageKit-1.1.1-3.fc24.x86_64

How reproducible:
No idea, just noticed this.

Steps to Reproduce:
1.Don't know
2.
3.

Actual results:
High memory usage

Expected results:
Normal memory usage

Additional info:
I'll keep it running as is in case you need some info from it.

Comment 1 Marcelo Ricardo Leitner 2016-07-09 11:55 UTC
Created attachment 1177841 [details]
/proc/../smaps

$ grep 'Rss.*[0-9]\{5,9\} ' smaps 
Rss:               47984 kB
Rss:               63376 kB
Rss:               65020 kB
Rss:               65364 kB
Rss:               13748 kB
Rss:               65472 kB
Rss:               22224 kB
Rss:               40792 kB
Rss:               55796 kB

Comment 2 Marcelo Ricardo Leitner 2016-07-09 11:56 UTC
Created attachment 1177842 [details]
/proc/.../maps

Comment 3 Kalev Lember 2016-07-09 12:02:47 UTC
Could you run packagekitd under valgrind please for a few days and see what it reports?

Something like,
dnf debuginfo-install PackageKit
killall packagekitd
valgrind --log-file=/tmp/packagekitd-valgrind-log --leak-check=full /usr/libexec/packagekitd

Comment 4 Marcelo Ricardo Leitner 2016-07-11 16:42:43 UTC
This is how it was now, before restarting:
root      1577  0.0  3.5 1030508 427796 ?      Ssl  Jul04   1:51 /usr/libexec/packagekitd
It didn't grow anymore since bug report.

So I did a simple restart, just to check how it comes back:
root     20289  1.5  0.1 533464 18684 ?        Ssl  13:40   0:00 /usr/libexec/packagekitd

Now it's running under valgrind as you requested, lets see. 
root     20388 37.2  0.9 465944 109328 pts/1   Sl+  13:40   0:07 valgrind --log-file=/tmp/packagekitd-valgrind-log --leak-check=full /usr/libexec/packagekitd

Comment 5 andylembke 2016-07-14 04:54:35 UTC
I have the same problem. Memory usage seems to rise each time packagekitd checks for updates. After a few days without restarting you have a few hundred megabytes.
Also downloading and installing updates seems to be much slower than using dnf on a command line.

Comment 6 andylembke 2016-07-14 04:58:02 UTC
(In reply to andylembke from comment #5)
> I have the same problem. Memory usage seems to rise each time packagekitd
> checks for updates. After a few days without restarting you have a few
> hundred megabytes.
> Also downloading and installing updates seems to be much slower than using
> dnf on a command line.

I'm on a fresh Fedora 24 installation btw.

Comment 7 Marcelo Ricardo Leitner 2016-07-14 11:10 UTC
Created attachment 1179752 [details]
valgrind output

It already looked like:
root     20388  0.0  1.9 620380 233160 pts/1   Sl+  Jul11   3:30 valgrind --log-file=/tmp/packagekitd-valgrind-log --leak-check=full /usr/libexec/packagekitd

it doubled.

Valgrind output has some unresolved symbols but I have the debuginfos installed.

Interesting that valgrind actually say they memory is still reachable?? THe leaks are minors and the 'still reachable' ones are quite comparable to the memory usage.

Comment 9 Marcelo Ricardo Leitner 2016-08-10 15:12:16 UTC
Cool, thanks. If you want I can give them a try if you prepare a package, btw.

Comment 10 Kalev Lember 2016-08-10 16:32:44 UTC
Sure, http://koji.fedoraproject.org/koji/taskinfo?taskID=15204070 -- it's a master snapshot that uses new libdnf and has a bunch of other new code and may crash or not work properly in all cases. Testing very welcome :)

Thanks!

Comment 11 Marcelo Ricardo Leitner 2016-08-10 16:52:25 UTC
Alright. Safety helmet if on, thanks for the advice. :)
Pkgs are installed,
root     25824  1.5  0.1 531632 18992 ?        Ssl  13:49   0:00 /usr/libexec/packagekitd

After hitting a command not found situation:
root     25824  0.8  0.6 774988 77208 ?        Ssl  13:49   0:01 /usr/libexec/packagekitd

After hitting it again, it didn't change a thing. Now I'll leave it be for a while.

Comment 12 Marcelo Ricardo Leitner 2016-08-13 20:25:11 UTC
Now it looks like:
root      1549  0.0  1.7 944628 208212 ?       Ssl  Aug11   0:14 /usr/libexec/packagekitd

I didn't install any update. It checked, it warned be that there are updates, but I didn't do them.

Comment 13 Marcelo Ricardo Leitner 2016-08-15 11:57:21 UTC
root      1549  0.0  2.6 970336 312088 ?       Ssl  Aug11   0:33 /usr/libexec/packagekitd

$ rpm -q PackageKit
PackageKit-1.1.4-0.0.20160810.fc24.kalev10.x86_64

Laptop was suspended for most of the time during this weekend.

Comment 14 Kalev Lember 2016-08-15 17:42:22 UTC
Yeah, looks like it's still slowly going up. I'll try to poke at this more when I get back from GUADEC.

Comment 15 Marcelo Ricardo Leitner 2016-09-01 12:24:46 UTC
$ ps aux | grep package
root     23764  0.0  3.2 1068912 389416 ?      Ssl  Ago26   1:26 /usr/libexec/packagekitd

$ uptime
 09:22:41 up 11 days, 17:40,  1 user,  load average: 0,26, 0,29, 0,41

But I had a reboot in between the reports, forgot to save the initial values.

Comment 16 Fedora End Of Life 2016-11-25 09:24:49 UTC
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 17 Fedora End Of Life 2016-12-20 21:09:33 UTC
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 18 pokorny.jan.94 2017-07-31 13:17:23 UTC
Hello,

I am using Fedora 25 a still facing common memory problem.

After 3 days packagekitd proccess consumes 477.5 MB.

uname -a: Linux localhost.localdomain 4.11.10-200.fc25.x86_64 #1 SMP Wed Jul 12 19:04:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Comment 19 Jonas Ådahl 2018-01-16 03:25:13 UTC
Reopening. This doesn't seem to be fixed in F27. I just saw it take 800MB of RES for no apparent reason. In fact, I tend to go and kill it on a weekly basis as I see it often eating more and more memory over time.

Comment 20 Alan Jenkins 2018-02-12 10:50:59 UTC
/subscribe

Packagekitd appears idle e.g. in pkmon, but it takes slightly more RAM than thunderbird.

$ ps aux | grep package
root      1445  0.0  4.5 1020784 362308 ?      Ssl  Feb07   0:46 /usr/libexec/packagekitd

This was after 5 days.

$ ls -ld /proc/$(pgrep ackag)
dr-xr-xr-x. 9 root root 0 Feb  7 09:08 /proc/1445
$ date
Mon 12 Feb 10:25:21 GMT 2018

After restarting (and exercising with pkcon refresh force + pkcon get-updates), the daemon process shows significantly smaller.

$ ps aux | grep package
root     22107  4.1  1.2 850696 102976 ?       Ssl  10:28   0:04 /usr/libexec/packagekitd

Though I'm still not madly happy to have 100M resident for an always-running daemon when idle.  Swapping does not tend to go well!

It seems a slightly surprising.  When I think about the always-on part of packagekit, what comes to mind is the appdata metadata which was specifically created for packagekit.  So you'd think it could use an specialized cache for this, which could be made efficient and/or be backed by disk files.  Disk files *at least* get synchronized to disk more aggressively than process memory, which makes it less painful to page them out when someone else needs the RAM.

I don't think we have a good reason to keep the distro-specific package manager loaded fully into RAM all the time.

Comment 21 Trae Santiago 2018-08-19 06:33:00 UTC
Hello,

It appears I'm having this bug as well, on two different machines with significantly differing hardware configurations. I've removed PackageKit for now, but I found it eating nearly 8GB of RAM idling on both machines.

fpaste --sysinfo from one of them:
https://paste.fedoraproject.org/paste/ps~emjYBjmQ64PnToih10Q

Screenshot:
https://photos.app.goo.gl/i1aYAxbiu4UKA7LQ6

As well, after some cursory reading, I'm wondering why PackageKit is running in the background at all; it is intentionally designed so that it does not run unless needed. Taken from https://www.freedesktop.org/software/PackageKit/pk-intro.html :

Being system activated means that it's only being run when the user is using a text mode or graphical tool, and quits when it's no longer being used. This means we don't delay the boot sequence or session startup and don't consume memory when not being used.

So just my 2¢, but I this should be fixed by making sure PackageKit is not running when not actively in use, and the memory leak fixes should come after.

Comment 22 Volker Sobek 2018-09-05 21:43:33 UTC
There is a commit which fixes this issue in the official repo [0], but there has not been a new release since.

For now, commenting out

#ShutdownTimeout=300

in /etc/PackageKit/PackageKit.conf will shut down packagekitd after 5 minutes. Notice that #ShutdownTimeout=300 is also *not* the default used when the line is not commented out, the default was changed to 0 in an earlier commit mentioned in the commit message of commit [0] without updating PackageKit.conf accordingly. The commit message also mentions potential problems when using dnf or rpm alongside PackageKit when using a timeout.

[0] https://github.com/hughsie/PackageKit/commit/0c84d71509e851db20445c747529bd7d3724f081

Comment 23 Volker Sobek 2018-09-05 22:16:42 UTC
Sorry, I should have been more precise: The commit I mentioned dosen't fix a memory leak, but it does at least ensure that packagekit shuts down after a timeout by default. The memory leak might still be around anyway.

(In reply to Volker Sobek from comment #22)
> There is a commit which fixes this issue in the official repo [0], but there
> has not been a new release since.
> 
> For now, commenting out
> 
> #ShutdownTimeout=300
> 
> in /etc/PackageKit/PackageKit.conf will shut down packagekitd after 5
> minutes. Notice that #ShutdownTimeout=300 is also *not* the default used
> when the line is not commented out, the default was changed to 0 in an
> earlier commit mentioned in the commit message of commit [0] without
> updating PackageKit.conf accordingly. The commit message also mentions
> potential problems when using dnf or rpm alongside PackageKit when using a
> timeout.
> 
> [0]
> https://github.com/hughsie/PackageKit/commit/
> 0c84d71509e851db20445c747529bd7d3724f081

Comment 24 Ryan 2018-09-27 22:30:31 UTC
Just to tag in here, this bug appears to have gotten worse in F29. Like everyone else here has mentioned, I've seen this bug in previous versions but it always took a few days to build up and I have no real idea how to really help so I've not piled on.

That being said, in the last two days of using F29, I've had packagekit climb to over ~300 MB in around 6 hours of usage without doing any installs or even opening software center. After the first day of seeing that I even made sure I disabled the new automatic updates options in Software Center, thinking that may help.  

Even more concerning though, is that today - after making sure I manually restarted the packagekit.service before puting the computer to sleep - the packagekit daemon grew from it's initial ~4MB (last I looked at it before going to sleep) to  ~158MB (took a look as soon as I booted up). This happened, while the computer was supposed to be 'sleeping', locked and with the lid closed. 

I'd be happy to try to play guinea pig and test fixes to this if I can, but I may need some basic hand holding as I've never done it before.

Comment 25 Scott Mattan 2018-11-27 12:43:16 UTC
I just wanted to add my voice as well, I see packagekitd taking up 600~800MB of RES within 2 days.  As with Ryan, I would also be happy to test potential fixes.

Comment 26 Scott Mattan 2018-11-27 12:43:52 UTC
I just wanted to add my voice as well, I see packagekitd taking up 600~800MB of RES within 2 days.  As with Ryan, I would also be happy to test potential fixes.

Comment 27 Ben Cotton 2018-11-27 13:28:30 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 28 Scott Mattan 2018-11-27 13:48:17 UTC
Richard, If you could, please update this to Fedora 29 as the problem appears to still exist.

Comment 29 Marcelo Ricardo Leitner 2018-11-28 14:17:21 UTC
(In reply to Scott Mattan from comment #28)
> Richard, If you could, please update this to Fedora 29 as the problem
> appears to still exist.

I am not using PackageKit since then, btw.

Scott, which package are you using?

Comment 30 Kalev Lember 2018-11-28 14:25:44 UTC
packagekitd in F29 is automatically shutting down after a few minutes of inactivity. This should make sure that it doesn't start eating a lot of memory over time.

Comment 31 Marcelo Ricardo Leitner 2018-11-28 14:43:10 UTC
That's more a workaround than a fix.
Can we really assume that it will have enough inactivity in all cases?
Considering that the issue may lay on dependency handling, wouldn't it use less memory on its iterations by fixing this?

Comment 32 Kalev Lember 2018-11-28 14:54:19 UTC
Yes, agreed that it's a workaround, but that's the best we can do when underlying libraries leak.

Comment 33 Marcelo Ricardo Leitner 2018-11-28 16:06:26 UTC
If we know which library it is, we can move the bug to it too.

Comment 34 Scott Mattan 2018-11-29 12:16:12 UTC
(In reply to Marcelo Ricardo Leitner from comment #29)
> (In reply to Scott Mattan from comment #28)
> > Richard, If you could, please update this to Fedora 29 as the problem
> > appears to still exist.
> 
> I am not using PackageKit since then, btw.
> 
> Scott, which package are you using?

Richard, I apologize, seems I have Fedora 28 installed on this machine.

It is running PackageKit-1.1.10-1.fc28.x86_64

That being said, I think Ryan from comment #24 posted that he was using Fedora 29 with the same issue.


Note You need to log in before you can comment on or make changes to this bug.