Description of problem: For about a minute or so with 10 minutes interval yumBackend.py runs on my system raising cpu load to 100%. This started to happen since last time i updated my system. These are the packages that got updated that day (not even packagekit what makes thing stranger to me). Oct 24 09:20:49 Updated: sssd-client-1.6.2-4.fc16.x86_64 Oct 24 09:20:49 Updated: libipa_hbac-1.6.2-4.fc16.x86_64 Oct 24 09:20:50 Updated: 1:NetworkManager-glib-0.9.1.90-4.git20110927.fc16.x86_64 Oct 24 09:20:50 Installed: libunwind-0.99-2.20110424git1e10c293.fc16.x86_64 Oct 24 09:20:50 Installed: google-perftools-1.8-1.fc16.x86_64 Oct 24 09:20:51 Installed: binutils-2.21.53.0.1-2.fc16.x86_64 Oct 24 09:20:52 Installed: ceph-0.31-3.fc16.x86_64 Oct 24 09:20:52 Updated: 2:qemu-common-0.15.1-1.fc16.x86_64 Oct 24 09:20:53 Updated: 2:qemu-system-x86-0.15.1-1.fc16.x86_64 Oct 24 09:20:54 Updated: soprano-2.7.2-1.fc16.x86_64 Oct 24 09:21:07 Updated: selinux-policy-3.10.0-46.fc16.noarch Oct 24 09:21:10 Updated: selinux-policy-targeted-3.10.0-46.fc16.noarch Oct 24 09:21:12 Updated: konversation-1.3.1-5.fc16.x86_64 Oct 24 09:21:12 Updated: 2:qemu-kvm-0.15.1-1.fc16.x86_64 Oct 24 09:21:12 Updated: 2:qemu-img-0.15.1-1.fc16.x86_64 Oct 24 09:21:12 Updated: 1:NetworkManager-0.9.1.90-4.git20110927.fc16.x86_64 Oct 24 09:21:13 Updated: sssd-1.6.2-4.fc16.x86_64 Oct 24 09:21:13 Updated: libmtp-1.1.1-1.fc16.x86_64 Oct 24 09:21:13 Updated: dracut-013-16.fc16.noarch Oct 24 09:21:14 Updated: mdadm-3.2.2-12.fc16.x86_64 Oct 24 09:21:14 Updated: 1:smartmontools-5.42-1.fc16.x86_64 Oct 24 09:21:14 Updated: 3:ypbind-1.33-7.fc16.x86_64 Oct 24 09:21:14 Updated: libdrm-2.4.26-2.fc16.x86_64 Oct 24 09:21:14 Updated: chrony-1.26-3.20110831gitb088b7.fc16.x86_64 Oct 24 09:21:15 Updated: cln-1.3.2-2.fc16.x86_64 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I've been seeing similar symptoms ever since upgrading from Fedora 14 to 15. Recently, some users have reported similar symptoms on #fedora-kde.
Do you have any graphic front-end with auto update software enabled? Like KPackagekit? It could be that program that triggers autoupdate every 10 minutes. For 100% CPU usage you should wait for more expert people than me. Which CPU do you have? On IRC I read you have a dual core, is it right?
The auto update is clearly triggered by a GUI frontend (KPackageKit in my case, I don't know what the reporter is using). Now if it really does run every 10 minutes, that's excessive, it's supposed to be once per day by default, or once per hour maximum. (I have it set to hourly, and I don't notice it running more often than that.) But the real problem here is the blatant CPU abuse.
Yes, I use KDE, so I guess it's to do with running "apper" (new name for KPackageKit). The last time yumBackend.py ran, it ran at close to 100% CPU for over 15 minutes. I haven't noticed how often it fires, but it definitely runs for a long time at very high resource usage when it does kick off. Is this a KDE-only issue? I see that the original reporter has several KDE-related packages installed so I suspect maybe he's running KDE too ... ?
When it's running at 100%, what does "pkmon" report?
I just started apper and pressed "check for updates" to try to trigger this behaviour. apper claims it's "Generating package lists", yumBackend.py is running at 90+% CPU, and here's the output from pkmon: Transactions: 1 /943_bbedcccb_data /943_bbedcccb_data status setup /943_bbedcccb_data percentage -1 /943_bbedcccb_data role refresh-cache /943_bbedcccb_data allow_cancel 1 /943_bbedcccb_data status refresh-cache /943_bbedcccb_data percentage 2 /943_bbedcccb_data percentage 6 /943_bbedcccb_data percentage 22 /943_bbedcccb_data percentage 38 /943_bbedcccb_data percentage 46 /943_bbedcccb_data percentage 58 /943_bbedcccb_data percentage 95 /943_bbedcccb_data percentage 100 daemon locked=1 /943_bbedcccb_data status finished /943_bbedcccb_data allow_cancel 0 /943_bbedcccb_data status generate-package-list /943_bbedcccb_data percentage -1 Transactions: 1 /943_bbedcccb_data 2 /944_dabcacad_data /944_dabcacad_data status wait /944_dabcacad_data allow_cancel 1 /944_dabcacad_data percentage -1 /944_dabcacad_data role get-distro-upgrades I let this run for about 10 minutes with the progress bar spinning back and forth in apper and no further output from pkmon.
Update: it finally finished! Here's the full pkmon output -- note that this was about 15 minutes of 100% CPU from start to finish. Transactions: 1 /943_bbedcccb_data /943_bbedcccb_data status setup /943_bbedcccb_data percentage -1 /943_bbedcccb_data role refresh-cache /943_bbedcccb_data allow_cancel 1 /943_bbedcccb_data status refresh-cache /943_bbedcccb_data percentage 2 /943_bbedcccb_data percentage 6 /943_bbedcccb_data percentage 22 /943_bbedcccb_data percentage 38 /943_bbedcccb_data percentage 46 /943_bbedcccb_data percentage 58 /943_bbedcccb_data percentage 95 /943_bbedcccb_data percentage 100 daemon locked=1 /943_bbedcccb_data status finished /943_bbedcccb_data allow_cancel 0 /943_bbedcccb_data status generate-package-list /943_bbedcccb_data percentage -1 Transactions: 1 /943_bbedcccb_data 2 /944_dabcacad_data /944_dabcacad_data status wait /944_dabcacad_data allow_cancel 1 /944_dabcacad_data percentage -1 /944_dabcacad_data role get-distro-upgrades /943_bbedcccb_data percentage 0 /943_bbedcccb_data percentage 50 /943_bbedcccb_data status finished /943_bbedcccb_data exit code: success Transactions: 1 /944_dabcacad_data daemon locked=0 /944_dabcacad_data status query updates-changed daemon locked=1 /944_dabcacad_data status finished /944_dabcacad_data exit code: success Transactions: [none] daemon locked=0 Transactions: 1 /945_eecdebbb_data /945_eecdebbb_data status setup /945_eecdebbb_data percentage -1 /945_eecdebbb_data role get-updates Transactions: 1 /945_eecdebbb_data 2 /946_bacacdda_data /946_bacacdda_data status wait /946_bacacdda_data allow_cancel 1 /946_bacacdda_data percentage -1 /946_bacacdda_data role get-distro-upgrades /945_eecdebbb_data allow_cancel 1 /945_eecdebbb_data status info daemon locked=1 /945_eecdebbb_data status finished /945_eecdebbb_data allow_cancel 0 /945_eecdebbb_data exit code: success Transactions: 1 /946_bacacdda_data daemon locked=0 /946_bacacdda_data status setup /946_bacacdda_data allow_cancel 0 /946_bacacdda_data allow_cancel 1 /946_bacacdda_data status query daemon locked=1 /946_bacacdda_data status finished /946_bacacdda_data allow_cancel 0 /946_bacacdda_data exit code: success Transactions: [none] daemon locked=0 Transactions: 1 /947_eebbacce_data /947_eebbacce_data status setup /947_eebbacce_data percentage -1 /947_eebbacce_data role get-distro-upgrades /947_eebbacce_data allow_cancel 1 /947_eebbacce_data status query daemon locked=1 /947_eebbacce_data status finished /947_eebbacce_data allow_cancel 0 /947_eebbacce_data exit code: success Transactions: [none] daemon locked=0
Mary, which one of these commands causes the 100% CPU for you? pkcon get-updates pkcon get-distro-upgrades pkcon refresh-cache Can you also tell me roughly how long each command took please. Thanks. Richard.
(In reply to comment #8) > pkcon get-updates Takes about 10-15 seconds, 100% CPU but for such a short time who cares :) > pkcon get-distro-upgrades < 2 seconds > pkcon refresh-cache (I assume you meant "pkcon refresh"?) This seems to be the culprit. Specifically, the "Generating package lists" part of the transaction ate CPU for over 12 minutes. I do have a fair number of repositories configured (see below) -- but still this shouldn't make things be this slow. Enabled Dropbox Dropbox Repository Enabled adobe-linux-i386 Adobe Systems Incorporated Enabled fedora Fedora 15 - x86_64 Enabled fedora-chromium Chromium web browser and deps Enabled kde kde Enabled kde-i686 kde Enabled kde-i686-testing kde-testing Enabled kde-i686-unstable kde-unstable Enabled kde-testing kde-testing Enabled kde-unstable kde-unstable Enabled rpmfusion-free RPM Fusion for Fedora 15 - Free Enabled rpmfusion-free-updates RPM Fusion for Fedora 15 - Free - Updates Enabled rpmfusion-nonfree RPM Fusion for Fedora 15 - Nonfree Enabled rpmfusion-nonfree-updates RPM Fusion for Fedora 15 - Nonfree - Updates Enabled texlive TeX Live Enabled updates Fedora 15 - x86_64 - Updates Enabled updates-testing Fedora 15 - x86_64 - Test Updates
> pkcon get-updates 6% !!! load, for about 2-3 seconds. > pkcon get-distro-upgrades Fatal error: GetDistroUpgrades not yet supported by backend > pkcon refresh-cache Command failed: Option 'refresh-cache' is not supported Well, after update i haven't notice any strange behavior in this regard.
I'm hitting the same issue on fully updated Fedora 15/x86_64. pkcon get-updates hands at 100% while waiting in queue with 100% CPU usage. Once I looked, I found: $ ps -AH | grep $(cat /var/run/yum.pid) -A2 -B2 8861 pts/1 00:00:00 grep 6120 ? 00:00:04 packagekitd 8625 ? 00:01:38 yumBackend.py <defunct> 8710 ? 00:00:00 yumBackend.py ... packagekitd seems to be spawning yumBackend's without killing the old ones. After killall -9 packagekitd yumBackend.py, I managed to run get-updates just fine: $ time pkcon get-updates Getting updates [=========================] Waiting in queue [=========================] Starting [=========================] Getting information [=========================] There are no updates available at this time. real 0m5.816s user 0m0.012s sys 0m0.009s ... But then systemctld (?) restarted packagekitd and I got strait back to yumBackend.py eating 100% CPU and pkcon hanging while waiting in queue. $ ps -AH | grep $(cat /var/run/yum.pid) -A2 -B2 8561 ? 00:00:00 apper-sentinel 8953 ? 00:00:00 packagekitd 8985 ? 00:01:13 yumBackend.py Killing packagekitd and yumBackend (again) helps once again: $ time pkcon get-distro-upgrades Getting upgrades [=========================] Waiting in queue [=========================] Starting [=========================] There are no upgrades available at this time. real 0m0.759s user 0m0.005s sys 0m0.009s For the time being I'll simply disable packagekitd. - Gilboa
The issue in comment #11 seems to be different than mine -- get-updates completes very quickly for me, with no blocking, but refresh (in particular, the "Generating package lists" part of refresh) takes ages.
(In reply to comment #11) > ... packagekitd seems to be spawning yumBackend's without killing the old ones. > After killall -9 packagekitd yumBackend.py, I managed to run get-updates just > fine: This annoyed me enough a few weeks ago that I looked into the code and figured out the various ways that packagekit can leak children. I know what to do to circumvent the problem, I just need to get back to it and get it done.
(In reply to comment #9) > (I assume you meant "pkcon refresh"?) > This seems to be the culprit. Specifically, the "Generating package lists" part > of the transaction ate CPU for over 12 minutes. It might be interesting to profile this with sysprof, or similar.
Latest observation. I get this problem only when i wake laptop up from suspend. Meaning that when i reboot and work i do not have this problem. If i suspend laptop and continue work later problem occurs.
So it looks like 'pkcon refresh' got slower between F14 and F15. Boot to runlevel 3 and run pkcon refresh once to get packagekitd started and everything downloaded. Then 'time pkcon refresh'. Fedora 14 Live image + updated rpm, yum, PackageKit: # rpm -q yum rpm PackageKit rpm-4.8.1-7.fc14.x86_64 yum-3.2.28-7.fc14.noarch PackageKit-0.6.12-4.fc14.x86_64 # time pkcon refresh real 0m6.908s user 0m0.008s sys 0m0.007s Fedora 15 Live image, as released: # rpm -q yum rpm PackageKit rpm-4.9.0-6.fc15.x86_64 yum-3.2.29-4.fc15.noarch PackageKit-0.6.14-2.fc15.x86_64 # time pkcon refresh real 2m10.322s user 0m0.324s sys 0m9.083s for (unfair) comparison, 'yum clean all' followed by 'yum check-update' takes about 40 seconds all in. I don't know whether there's a yum command that's equivalent to pkcon refresh.
Hello: I see this behaviour since I upgraded from F15 -> F16. I see similar behaviour of 'pkcon refresh' as also pointed out in Comments #11 and #12 - It takes ages in the "Generating package lists" stage.
In my case (with F16) the specific command causing the problems appears to be: /usr/bin/python /usr/share/PackageKit/helpers/yum/yumBackend.py get-packages unknown
# time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Enabled fedora Fedora 16 - x86_64 Enabled updates Fedora 16 - x86_64 - Updates real 1m1.584s user 0m0.103s sys 0m0.092s
(In reply to comment #18) > In my case (with F16) the specific command causing the problems appears to be: > /usr/bin/python /usr/share/PackageKit/helpers/yum/yumBackend.py get-packages > unknown I have the same (annoying) problem on my F16 laptop.
I use gkrellm to monitor my system and I see one of my four cores shoot up to 100% every few minutes and it looks like this may be the cause. [root@apex ~]# time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Enabled Google Google - x86_64 Enabled fedora Fedora 16 - x86_64 Enabled fedora-chromium Chromium web browser and deps Enabled updates Fedora 16 - x86_64 - Updates real 1m27.940s user 0m0.174s sys 0m0.210s
Same problem, F16, KDE, apper-sentinel kicks off packagekitd when runs the yum backend -- over and over and over ad infinitum, about every 10 minutes. yum backend has the cpu at around 97%, and the disks are chattering away merrily. Solution: remove apper rpm.
This is also happening on X86 platforms. PackageKit-yum-0.6.21-1.fc16.i686 From top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27323 root 20 0 112m 63m 8004 R 95.9 1.6 2:36.97 yumBackend.py
The same issue here: Fedora 16 (x86_64) KDE, upgraded from F15. Installed packages: PackageKit.x86_64 0.6.21-1.fc16 @updates PackageKit-glib.x86_64 0.6.21-1.fc16 @updates PackageKit-gstreamer-plugin.x86_64 0.6.21-1.fc16 @updates PackageKit-qt.x86_64 0.6.21-1.fc16 @updates PackageKit-yum.x86_64 0.6.21-1.fc16 @updates PackageKit-yum-plugin.x86_64 0.6.21-1.fc16 @updates yum.noarch 3.4.3-7.fc16 @updates yum-langpacks.noarch 0.2.4-2.fc16 @updates yum-metadata-parser.x86_64 1.1.4-5.fc16 @anaconda-0 yum-plugin-changelog.noarch 1.1.31-2.fc16 @anaconda-0 yum-plugin-ps.noarch 1.1.31-2.fc16 @anaconda-0 yum-plugin-security.noarch 1.1.31-2.fc16 @anaconda-0 yum-plugin-show-leaves.noarch 1.1.31-2.fc16 @anaconda-0 yum-plugin-verify.noarch 1.1.31-2.fc16 @anaconda-0 yum-presto.noarch 0.7.1-1.fc16 @anaconda-0 yum-utils.noarch 1.1.31-2.fc16 @anaconda-0 The yumBackend.py starts to go wild after random time from several minutes to several hours 6, 7 or more after start-up. Then it is being triggered/started periodically. Re-boot is a temporary workaround until it happens again. $ top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10957 root 20 0 435m 112m 8720 R 98.1 1.9 1:12.42 /usr/bin/python /usr/share/PackageKit/helpers/yum/yumBackend.py get-packages unknown The period is from about 4 to 6 minutes and it runs from about 1.5 to 2 minutes. I used a simple script to track the yumBackend.py process, S-Start, F-Finish, Ex: Tue Jan 3 19:26:07 CET 2012 S: Tue Jan 3 19:28:20 CET 2012 F: Tue Jan 3 19:29:54 CET 2012 S: Tue Jan 3 19:34:20 CET 2012 F: Tue Jan 3 19:35:54 CET 2012 S: Tue Jan 3 19:40:21 CET 2012 F: Tue Jan 3 19:41:58 CET 2012 S: Tue Jan 3 19:46:19 CET 2012 F: Tue Jan 3 19:47:53 CET 2012 S: Tue Jan 3 19:52:19 CET 2012 F: Tue Jan 3 19:53:56 CET 2012 $ time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Enabled adobe-linux-x86_64 Adobe Systems Incorporated Enabled fedora Fedora 16 - x86_64 Enabled rpmfusion-free RPM Fusion for Fedora 16 - Free Enabled rpmfusion-free-updates RPM Fusion for Fedora 16 - Free - Updates Enabled rpmfusion-nonfree RPM Fusion for Fedora 16 - Nonfree Enabled rpmfusion-nonfree-updates RPM Fusion for Fedora 16 - Nonfree - Updates Enabled updates Fedora 16 - x86_64 - Updates real 1m31.943s user 0m0.092s sys 0m0.116s The most time consuming part is the "Generating package lists". In parallel to the pkcon I ran pkmon: $ pkmon Transactions: [none] network status=wifi Transactions: 1 /2765_cdddedac_data /2765_cdddedac_data status setup /2765_cdddedac_data percentage -1 /2765_cdddedac_data role refresh-cache /2765_cdddedac_data status refresh-cache /2765_cdddedac_data allow_cancel 1 /2765_cdddedac_data percentage 0 /2765_cdddedac_data percentage 95 /2765_cdddedac_data percentage 100 daemon locked=1 /2765_cdddedac_data status finished /2765_cdddedac_data allow_cancel 0 /2765_cdddedac_data status generate-package-list /2765_cdddedac_data percentage -1 /2765_cdddedac_data percentage 0 /2765_cdddedac_data status finished /2765_cdddedac_data exit code: success Transactions: [none] daemon locked=0 updates-changed Transactions: 1 /2766_caacacbe_data /2766_caacacbe_data status setup /2766_caacacbe_data percentage -1 /2766_caacacbe_data role get-updates /2766_caacacbe_data status info /2766_caacacbe_data allow_cancel 1 daemon locked=1 /2766_caacacbe_data status finished /2766_caacacbe_data allow_cancel 0 /2766_caacacbe_data exit code: success Transactions: [none] daemon locked=0
I am having this problem too, but Apper icon is not appearing... What kind of infos could I give you?
<aol>Me too.</aol> F16, KDE. Every 10 minutes, about 1 minute is spent in generate-package-list with CPU at 100%.
+1 any solution without remove apper ?
Tomorrow I will do some tries, I will let you know about my experience
What is everyone with this issue's Check for update frequency? I was having this problem as well. As far as everything I've tested through, on my machine it seems to be a problem with the appr setting of 'Check for updates' = Hourly. I ran with 'Check for updates' = Monthly for a while without this issue, then switched to daily for a short period and haven't yet experienced it. My guess would be the resource utilization is yum checking for updated every 10 minutes or so, instead of hourly.
The frequency of the checks has nothing to do with the resource utilization and is filed separately as bug #752564.
Same here 100% cpu utilisation, stalls at Generating package lists. 13 minutes to complete is unacceptable. root@fedora ~# time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Scanning applications [=========================] Generating package lists [=========================] Enabled fedora Fedora 16 - x86_64 Enabled internode-fedora Internode Fedora 16 - x86_64 Enabled internode-updates Internode Fedora 16 - x86_64 - Updates Enabled updates Fedora 16 - x86_64 - Updates real 13m8.564s user 0m0.750s sys 0m8.613s ---- and in parallel: root@fedora ~# pkmon Transactions: [none] daemon connected=1 network status=wired Transactions: 1 /987_ddbdbaca_data /987_ddbdbaca_data status setup /987_ddbdbaca_data percentage -1 /987_ddbdbaca_data role refresh-cache /987_ddbdbaca_data allow_cancel 1 /987_ddbdbaca_data status refresh-cache /987_ddbdbaca_data percentage 0 /987_ddbdbaca_data percentage 11 /987_ddbdbaca_data percentage 22 /987_ddbdbaca_data percentage 66 /987_ddbdbaca_data percentage 95 /987_ddbdbaca_data percentage 100 daemon locked=1 /987_ddbdbaca_data status finished /987_ddbdbaca_data allow_cancel 0 /987_ddbdbaca_data status generate-package-list /987_ddbdbaca_data percentage -1 Transactions: 1 /987_ddbdbaca_data 2 /988_aabacaec_data /988_aabacaec_data status wait /988_aabacaec_data allow_cancel 1 /988_aabacaec_data percentage -1 /988_aabacaec_data role get-updates /987_ddbdbaca_data status scan-applications /987_ddbdbaca_data status generate-package-list /987_ddbdbaca_data percentage 0 /987_ddbdbaca_data percentage 50 /987_ddbdbaca_data status finished /987_ddbdbaca_data exit code: success Transactions: 1 /988_aabacaec_data updates-changed daemon locked=0 /988_aabacaec_data status setup /988_aabacaec_data allow_cancel 0 /988_aabacaec_data allow_cancel 1 /988_aabacaec_data status info daemon locked=1 /988_aabacaec_data status finished /988_aabacaec_data allow_cancel 0 /988_aabacaec_data exit code: success Transactions: [none] daemon locked=0 Transactions: 1 /990_ddcebecb_data /990_ddcebecb_data status setup /990_ddcebecb_data percentage -1 /990_ddcebecb_data role get-updates /990_ddcebecb_data status info /990_ddcebecb_data allow_cancel 1 daemon locked=1 /990_ddcebecb_data status finished /990_ddcebecb_data allow_cancel 0 /990_ddcebecb_data exit code: success Transactions: [none] daemon locked=0 daemon connected=0 daemon connected=1
Looks like this was actually the same as bug #752897, as I suspected from the beginning. After upgrading to rpm-4.9.1.2-4.fc15, KPackageKit completes its verifications blazingly fast again. (The kernel also got upgraded, to 2.6.42.3-2.fc15.) Does anybody still see this slowness after upgrading to rpm-4.9.1.2-5.fc16 resp. rpm-4.9.1.2-4.fc15? If not, we can close this as a duplicate of bug #752897.
Uhm, nevermind… It looks like it just hadn't done the step which takes forever ("refreshing package lists") yet, grrr! :-( (I triggered a manual refresh now and it's slow as ever.) So this bug is still not fixed (neither the RPM upgrade nor the kernel upgrade helped).
F16, KDE, same problem yumBackend.py going randomly nuts. In fact suspected some upgrade issue, and reinstalled. Helped only briefly, now the symptom is reoccurring. Now back at "while [ 1 ]; do killall yumBackend.py; sleep 5s; done" to keep machine usable.
+1 Georg. I'm doing the same thing. :(
Removing google-chrome and all packages from google repo seems to solve the problem.
That can't be it. I don't have any packages from Google's repo installed and I'm seeing this bug.
Same here. No Google repo in use here before or after the re-installation and I keep having the problem.
yum remove google-* was the command used. No more 99% cpu usage by yumBackend.py. Yum removes also many maven-* packages and some other packages that I have forgotten. How can I query the last removed packages? Hoping this can help.
YumBackend.py came back to re-use 99% of cpu after some days. So there's no relation with the google repo as I suggest in my previous post.
well but disable external repos like : google , adobe-linux-i386, adobe-linux-x86_64 and others, just leaving rpmfusion helps here .
Is this bug being actively worked? It sure effects lots of systems and reflects badly on Fedora.
Still waiting for PK folks to respond to 'pkmon' output/feedback. feel free to give your own 'pkmon' output when yumBackend.py is going crazy, hopefully the additional information will help.
pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Downloading file lists [=========================] Generating package lists ... stuck awhile (a couple minutes so far), add $ pkmon Transactions: [none] daemon connected=1 network status=wired Transactions: 1 /14476_ddedbcbb_data /14476_ddedbcbb_data status setup /14476_ddedbcbb_data percentage -1 /14476_ddedbcbb_data role refresh-cache /14476_ddedbcbb_data status refresh-cache /14476_ddedbcbb_data allow_cancel 1 /14476_ddedbcbb_data percentage 0 /14476_ddedbcbb_data percentage 15 /14476_ddedbcbb_data status download-filelist /14476_ddedbcbb_data percentage 39 /14476_ddedbcbb_data percentage 45 /14476_ddedbcbb_data percentage 51 /14476_ddedbcbb_data percentage 57 /14476_ddedbcbb_data percentage 63 /14476_ddedbcbb_data percentage 69 /14476_ddedbcbb_data percentage 95 /14476_ddedbcbb_data percentage 100 daemon locked=1 /14476_ddedbcbb_data status finished /14476_ddedbcbb_data allow_cancel 0 /14476_ddedbcbb_data status generate-package-list /14476_ddedbcbb_data percentage -1 stuck... here
high cpu load for a couple of minutes at: status generate-package-list percentage -1
Do you guys also get 100% cpu if you issue the command "pkcon get-packages > /dev/null" ?
Yep. (Fully update F16/x86_64) From top: 5189 root 20 0 468m 143m 8772 R 99.9 1.2 1:14.58 yumBackend.py ^^^^
Yes
(In reply to comment #48) > Yes If you save this as test-yum.py: #!/usr/bin/python import yum yb = yum.YumBase() yb.doConfigSetup(errorlevel=0, debuglevel=0) yb.conf.cache = 0 print len(yb.rpmdb) print len(yb.pkgSack) And then run it as root, does it also take an age to finish?
1st invocation # time ./test-yum.py 1541 33826 real 0m24.070s user 0m1.607s sys 0m0.355s Consequent invocations # time ./test-yum.py 1541 33826 real 0m0.701s user 0m0.609s sys 0m0.081s 24seconds is much better than 2minutes.
No it does not. Run 1: time ./test-yum.py 3025 27564 real 0m5.731s user 0m4.734s sys 0m0.457s Run 2: time ./test-yum.py 3025 27564 real 0m5.474s user 0m4.700s sys 0m0.453s
time ./test-yum.py 1574 27604 real 0m3.352s user 0m1.742s sys 0m0.390s time ./test-yum.py 1574 27604 real 0m2.852s user 0m1.670s sys 0m0.358s time ./test-yum.py 1574 27604 real 0m3.341s user 0m1.715s sys 0m0.386s
# time ./test-yum.py 1574 28970 real 0m0.869s user 0m0.756s sys 0m0.105s # time ./test-yum.py 1574 28970 real 0m0.868s user 0m0.759s sys 0m0.100s # time ./test-yum.py 1574 28970 real 0m0.867s user 0m0.757s sys 0m0.102s
4375 root 20 0 950m 143m 8812 R 99.3 3.7 1:11.66 yumBackend.py after finish update and when I check to check updates, stays on "refreshing package cache" , generating package list . after pass that is quick .
yum on its own works just fine, so this is all because PackageKit does not respect yum settings and starts sticking its nose where no one asks! Wouldn't that be nice to just look where yum looks? Too much creativity on PackageKit part, IMHO.
(In reply to comment #55) > yum on its own works just fine, so this is all because PackageKit does not > respect yum settings no , I have the right to check for updates specially after update all system. I think two windows and ask two times for updates , kpackagekit , take much more time, that really should, because seems each process waits for the other.
(In reply to comment #55) > yum on its own works just fine, so this is all because PackageKit does not > respect yum settings and starts sticking its nose where no one asks! How useful, thanks for that comment. Basically, after doing a system update, PackageKit creates a list of all the software that is installed and available and saves it to a file. This is used to make the auto-complete input fields work in the various GNOME and KDE UIs. If this isn't important to you, just disable the functionality. If you open /etc/PackageKit/PackageKit.conf and change the line to read "UpdatePackageList=false" It seems asking yum to return a list of all the packages in all the repos takes a huge amount of time for some people. I'm assuming people are not running custom yum.conf or .repo files with custom low timeouts, so I'm not sure what the problem could be.
In my opinion, PackageKit is a pain and I would not install it if there were not so many dependencies. Any process, I don't care what it is, that consumes all available CPU is nasty and either needs to be written to be well behaved or it needs to be abandoned. Disabling the functionality is one thing, but making a package required by so many other packages and installing it because it is "required" with no option to not install is, in my opinion, arrogant of the writers.
(In reply to comment #58) > but making a package required by so many other packages Err, the upstream maintainers add it as a dep, not me. > to not install is, in my opinion, arrogant of the writers. arrogance: "Having or revealing an exaggerated sense of one's own importance or abilities." I don't see how providing a service that people have used is arrogant. Please do not add insult maintainers in bugzilla, as that's a very quick way to get the maintainer to start ignoring a bug. KTHXBYE.
I appreciate all the maintainers including you Richard. My comment was aimed at comment 56 by Sergio which is a great way to lose users of Fedora. It came across to me as arrogant. Maybe English is not Sergio's native language and that is why his comment came across that way.
(In reply to comment #60) > I appreciate all the maintainers including you Richard. > > Maybe English is not Sergio's > native language and that is why his comment came across that way. yeah , I also not write well in my native language. This bug report is try find why packagekit and yumBackend.py sometimes gets 100% load and takes too long. I 'm saying and pointing when I try force check updates this bug report happens , may be we are dealing with more than one bug. But seems to me related with this bug report and those comment , we are free to criticize. and Packagetkit is a great tool. Best regards,
"... providing a service ..." Oh, please. You are way over your head. Okay, I installed flash player. And you are shooting me in the head for that. Thanks, but no thanks!
Lets all cool our jets a bit. I know it is hard to work on a bug and to feel under appreciated for doing something for free. I have been in the same situation with software I was developing. I think everyone that has commented has done so in good faith. I also believe that we all appreciate the work done by Richard and all the developers and maintainers. This bug is irritating. I am looking at 100% CPU right now caused by yumBackend.py. If it only used 10% of the CPU, then it would be more tolerable. I really don't expect heavy CPU loads except when doing video rendering or some other computational heavy lifting. I'm not a python guy, so I do not know if python and python scripts are CPU intensive by nature. Maybe it is the nature of the interpreter; I would hope not, but I do not know. If I wrote a Java program that was chewing up this much CPU, I would be looking for something in a tight loop or some intensive math computation. Profile and figure out where it was spending its time. But python, I haven't a clue. I know there is nothing in what I have written to help diagnose this bug. My only hope is to get everyone to calm down. Richard, sorry if I offended you. Again, my comment was not directed in your direction. Sergio, I hope we are cool. I speak and write no other language (except computer languages) so you are doing well. Tomorrow is a new dawn and I hope we will approach life and this bug with cooperation and not confrontation.
This is a bug reporting tool, not a discussion board. Stop wasting everyone time, you-generally are spamming everyone email with your unwanted and "out of topic" comments.
Is everyone who is seeing this using KDE? And do we all have one or more kde-redhat repositories enabled? Just wondering ...
(In reply to comment #65) > Is everyone who is seeing this using KDE? I do. My machine is very fast, so yumBackend.py only eats one of my CPUs for 2mn, and not for 10mn, but I understand why people are complaining on less powerful machines. > And do we all have one or more kde-redhat repositories enabled? I do. apper-0.7.1-1.fc16.x86_64 kde-workspace-4.8.1-2.fc16.x86_64 Apper is for F16 updates, and my KDE is from fedora-kde48.
I have the same problem since I switch to KDE. When doing some strace on the yumBackend.py I have this kind of errors: access("/var/cache/yum/x86_64/16/rpmfusion-free-updates/c17a1a4d74a20f525389ca4332749f42968aa1e59c765696d2a1e2fa9d28ecf9-primary.sqlite-journal", F_OK) = -1 ENOENT (No such file or directory) fstat(13, {st_mode=S_IFREG|0644, st_size=1553408, ...}) = 0 access("/var/cache/yum/x86_64/16/rpmfusion-free-updates/c17a1a4d74a20f525389ca4332749f42968aa1e59c765696d2a1e2fa9d28ecf9-primary.sqlite-wal", F_OK) = -1 ENOENT (No such file or directory) And tcpdump also shows some traffic to DNS and http port, this is bad because I'm behind a proxy. Is yum.conf ignored by yumBackend.py? Is it able to pass thru a proxy?
Created attachment 569687 [details] Result of `pkmon -v | tee ~/pkmon-v.log` I attach a log of `pkmon -v` where we see PackageKit run 'get-updates' every 6 minutes, on my machine. I stopped the log after five of such cycles. We can see in the timings that each get-updates needs about 2 minutes to finish. During those two minutes, `top` shows yumBackend.py consumes 100% of one of my CPUs most of the time.
> And do we all have one or more kde-redhat repositories enabled? No, I don't have kde-redhat repos enabled.
I use kde but do not have any kde repos enabled.
I use fresh installation of Fedore-KDE x64_86; nothing from kde-redhat. I install koji kernel using rpm once in a while, i.e. not even in updates-testing. I installed Adobe Plash Player using rpm directly; again not in any repo. I do use "yum update" directly too, just to mess up with KP I guess :). So I have a few packages not in any repo. Is this crazy? I do not think so and you does not give me any grief. Dear developers, You deserve to be yelled at because this bug/behavior is so old and so annoying: bug 473379, bug 474492, bug 590984, bug 773193. It looks like you never paid attention to those bugs: they seemed to resolve themselves without your input. I guess they never actually did resolve. It is official this bug is 3 damn years old. Please get your act together!
(In reply to comment #71) > You deserve to be yelled at... I'll ignore that. Is anyone seeing this bug that isn't using KDE? It looks to me that some thing in KDE is triggering a full RefreshCache every few minutes, rather than once a week which is what GNOME does. Richard.
This is a Fedora 16 i686 KDE install Enabled repos: adobe-linux-i386.repo fedora-updates.repo fedora.repo google-chrome.repo rpmfusion-free-updates.repo rpmfusion-free.repo rpmfusion-nonfree-updates.repo rpmfusion-nonfree.repo
(In reply to comment #72) > Is anyone seeing this bug that isn't using KDE? It looks to > me that some thing in KDE is triggering a full RefreshCache every few minutes, > rather than once a week which is what GNOME does. I also use kde, I though that could be kpackagekit but that no exist any more. When I'm searching for that, I found /var/lib/yum have things from fc12, shouldn't be clean up ? I clean it up (moving folder), and seems no changes. after update the system with apper, I force check updates and root 3477 3251 99 22:14 ? 00:01:12 /usr/bin/python /usr/share/PackageKit/helpers/yum/yumBackend.py get-packages unknown stuck strace -p 3477 shows nothing for a very long time, the process are waiting for what ? yum clean all yum update is much more faster.
There is indeed a bug in Apper that causes it to check for updates too often. This is filed separately in bug #752564. However, checking daily or even hourly is a valid setting (IMHO, weekly is not anywhere near frequent enough) and it is taking way too long.
(In reply to comment #75) > There is indeed a bug in Apper that causes it to check for updates too often. > This is filed separately in bug #752564. Kevin, the problem is not Apper check too often. The problem is when Apper check, take much much more time (with yumBackend.py loads CPU to 100%) than: yum clean all , yum update , why ?
I'm under the impression it has to do with sorting groups. I start from the point that I do not know if Apper does a `yum clean all' before every check. But wouldn't be that by mistake it sorts groups even when no new updates are available?
I'm not using KDE and I see what looks like the same problem. Here are two consecutive 'time pkcon refresh' runs: [root@hoho0 user1]# time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Enabled fedora Fedora 16 - x86_64 Enabled updates Fedora 16 - x86_64 - Updates real 2m19.452s user 0m0.227s sys 0m0.277s [root@hoho0 user1]# time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Enabled fedora Fedora 16 - x86_64 Enabled updates Fedora 16 - x86_64 - Updates real 2m18.572s user 0m0.203s sys 0m0.293s [root@hoho0 user1]# It is too bad that subsequent runs of refresh don't use any previous information. Perhaps previous information is useless and a full refresh is always necessary? Looking at the system monitor: I have a dual core system. The full load core switches about 8 seconds in, then at 28 seconds additional, then it switches a few more times at larger intervals until it finishes. A glance at the python code might explain the switches. Is it a bad algorithm, or just a lot of work?
Running on another system, also not KDE gives: [root@hoho6 user1]# time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Scanning applications [=========================] Generating package lists [=========================] Enabled adobe-linux-x86_64 Adobe Systems Incorporated Enabled fedora Fedora 15 - x86_64 Enabled updates Fedora 15 - x86_64 - Updates real 1m31.433s user 0m0.057s sys 0m0.088s [root@hoho6 user1]# This is a faster 4 core system with 8G of memory. The full load core also jumps around, starting at about 5 seconds, then an additional 20 seconds, then it switches among the 4 cores at larger intervals.
Another system - FC9, no KDE, updated to the limits of FC9, but clearly it lacks some features: [root@hoho4 html]# uname -a Linux hoho4.chidig.com 2.6.27.25-78.2.56.fc9.x86_64 #1 SMP Thu Jun 18 12:24:37 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux [root@hoho4 html]# [root@hoho4 html]# time pkcon refresh refresh-cache [ ] (0%) Command failed: PolicyKit authorization failure real 0m0.078s user 0m0.004s sys 0m0.006s [root@hoho4 html]# This system just runs and runs, no memory leaks, have seen uptime better than 722 days, etc.
time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Scanning applications [=========================] Generating package lists [=========================] Enabled fedora Fedora 16 - x86_64 Enabled rpmfusion-free RPM Fusion for Fedora 16 - Free Enabled rpmfusion-free-updates RPM Fusion for Fedora 16 - Free - Updates Enabled rpmfusion-free-updates-testing RPM Fusion for Fedora 16 - Free - Test Updates Enabled rpmfusion-nonfree RPM Fusion for Fedora 16 - Nonfree Enabled rpmfusion-nonfree-updates RPM Fusion for Fedora 16 - Nonfree - Updates Enabled rpmfusion-nonfree-updates-testing RPM Fusion for Fedora 16 - Nonfree - Test Updates Enabled updates Fedora 16 - x86_64 - Updates real 4m31.911s user 0m0.349s sys 0m0.425s but 99% of time is on "Generating package lists" , and if I by mistake, do other yum something real time goes to 10 minutes. Yes this is the bug, nobody wants waits 4 minutes for "Generating package lists" .
time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Generating package lists [=========================] Enabled fedora Fedora 16 - x86_64 Enabled updates Fedora 16 - x86_64 - Updates real 2m54.845s user 0m0.298s sys 0m0.306s Most of the time bouncing in "Generating package lists".
time pkcon get-packages > /dev/null real 2m48.012s user 0m4.891s sys 0m0.508s
So, the get-packages CPU hog has been bugging me for a while now, so I went and wrapped yumBackend.py:get_packages by cProfile. I'm attaching two resulting profiles (first one from pkcon refresh, second auto-triggered by Apper; both basically identical). Aggregate timing stats look like this: ncalls tottime percall cumtime percall filename:lineno(function) 2 846.347 423.174 1240.173 620.086 /usr/lib/python2.7/site-packages/packagekit/filter.py:72(get_package_list) 185698453 391.350 0.000 391.694 0.000 /usr/share/PackageKit/helpers/yum/yumFilter.py:132(_pkg_compare) 4652 104.324 0.022 105.003 0.023 /usr/lib/python2.7/site-packages/yum/rpmsack.py:83(_get_hdr) 4654 66.193 0.014 66.225 0.014 /usr/lib/python2.7/site-packages/yum/rpmsack.py:1207(_get_packages) So apparently, when "Generating package lists", the backend spends 7 minutes in get_package_lists with CPU pegged at 100 %, and during that time it executes _pkg_compare 93 million (!!!) times. Given that function calls in Python aren't all that fast, no wonder that it takes this long. AFAICS the culprit is the following block: # check there are not available versions in the package list # that are older than the installed version package_list = self.package_list self.package_list = [] for pkg, state in package_list: add = True; if state is INFO_AVAILABLE: for pkg_tmp, state_tmp in self.package_list: if state_tmp is not INFO_INSTALLED: continue rc = self._pkg_compare(pkg, pkg_tmp) # don't add if the same as the installed package # or a downgrade to the existing installed package if rc == 0 or rc == -1: add = False break if add: self.package_list.append((pkg, state)) Basically, this is an O(N^2) CPU abuse just for filtering out old packages. This should probably be rewritten in a more friendly way (e.g. first add the package names to a set and use that for a fast (hashed) lookup, call _pkg_compare only on packages with same names). I'm not that of an Python programmer, but will try to implement something better in a week or two. However, anyone willing to do that sooner is more than welcome.
Created attachment 570946 [details] cProfile log from pkcon refresh
Created attachment 570947 [details] cProfile log from an automated Apper update check
Is there any way to outsource this to "yum check-update"?
hi, (suggestion / idea ) apper have two distinct areas: updates and installed software. when we are in updates, can apper not "Generating package lists", in updates we don't need it. When we go to "installed software" ok with "Generate package lists" before.
We've shown that pkcon is retarded when it comes to generating package lists. Why do you want to work around this in apper? Apper is fine.
One workaround is editing /etc/PackageKit/PackageKit.conf, and set: UpdatePackageList=false
(In reply to comment #90) > One workaround is editing /etc/PackageKit/PackageKit.conf, and set: > UpdatePackageList=false For me that it ! Thanks, time pkcon refresh Refreshing cache [=========================] Waiting for authentication [=========================] Starting [=========================] Refreshing software list [=========================] Scanning applications [=========================] Generating package lists [=========================] Enabled VirtualBox Fedora Virtual Box 4.1.x Enabled fedora Fedora 16 - x86_64 Enabled rpmfusion-free RPM Fusion for Fedora 16 - Free Enabled rpmfusion-free-updates RPM Fusion for Fedora 16 - Free - Updates Enabled rpmfusion-free-updates-testing RPM Fusion for Fedora 16 - Free - Test Updates Enabled rpmfusion-nonfree RPM Fusion for Fedora 16 - Nonfree Enabled rpmfusion-nonfree-updates RPM Fusion for Fedora 16 - Nonfree - Updates Enabled rpmfusion-nonfree-updates-testing RPM Fusion for Fedora 16 - Nonfree - Test Updates Enabled updates Fedora 16 - x86_64 - Updates real 0m1.732s user 0m0.017s sys 0m0.005s
Created attachment 575045 [details] Algorithm patch for filter.py This patch replaces the quadratic monstrosity with an approximately linear equivalent. On my system (40297 installed+available packages) this gets the time spent in get_package_list to 1.3 s from the initial ~600 s. In other words, pkcon refresh now spends just about 20 s CPU time instead of 8 minutes. To test, just do as root: cd /usr/lib/python2.7/site-packages/packagekit/ patch < pk-yum-filter.patch Tested on top of PackageKit-0.6.21-2.fc16.x86_64
Looks good Tomáš, but one (possibly newbie) question: Why use a collections object rather than a plain old dict?
(In reply to comment #93) > Looks good Tomáš, but one (possibly newbie) question: Why use a collections > object rather than a plain old dict? Just because it looks nicer (and maybe it's even faster, I haven't checked) this way - defaultdict creates the inner lists automatically on first use, so I can just go and .append() without checking. Of course, if you wish, it can easily be replaced by: installed_dict = dict() for pkg, state in self.package_list: if state is INFO_INSTALLED: pkg_name = self._pkg_get_name(pkg) if pkg_name in installed_dict: installed_dict[pkg_name].append(pkg) else installed_dict[pkg_name] = [pkg]
I've pushed this to master: commit 21ce80d11bba01230a8a3307ad7e21fbe0bb4685 Author: Tomáš Trnka <tomastrnka> Date: Wed Apr 4 12:17:45 2012 +0100 python: Speed up get_package_list() This patch replaces the quadratic monstrosity with an approximately linear equivalent. On my system (40297 installed+available packages) this gets the time spent in get_package_list to 1.3 s from the initial ~600 s. In other words, pkcon refresh now spends just about 20 s CPU time instead of 8 minutes. Signed-off-by: Richard Hughes <richard> If there are no regressions (I couldn't see any after testing for a few mins) then I'll backport this into PACKAGEKIT_0_6_X which is for F16. I'll do a combined update in a few days time if you're waiting for Fedora packages.
Thanks for the patch. It reduces the pkcon refresh run time from ~100sec to 8sec, for ~27k packages. No side effects.
Remaining question is: Does it have to run every 6 minutes?
No, that's bug #752564.
PackageKit-0.6.22-1.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/PackageKit-0.6.22-1.fc16
Package PackageKit-0.6.22-1.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing PackageKit-0.6.22-1.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-5750/PackageKit-0.6.22-1.fc16 then log in and leave karma (feedback).
I confirm that PackageKit.x86_64 0.6.22-1.fc16 fixes the issue and works fine. Thanks.
PackageKit-0.6.22-1.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.
Can the fix be backported to F15? I can build a patched 0.6.17 if you don't want to upgrade F15 to 0.6.22 and don't have time to do the backport.
(In reply to comment #103) > Can the fix be backported to F15? I can build a patched 0.6.17 if you don't > want to upgrade F15 to 0.6.22 and don't have time to do the backport. You have my blessing to patch 0.6.17.fc15 with the backported patch. Thanks dude.
Hmmmph, this fails to build because of xulrunner: pk-main.c:543:1: error: conflicting types for 'NP_GetMIMEDescription' /usr/include/xulrunner-2/npfunctions.h:307:24: note: previous declaration of 'NP_GetMIMEDescription' was here I haven't touched the xulrunner-related code at all, it must be due to a xulrunner change.
Looks like I also need a74d58a2adf03b520fcd1ce3f134ef8fdc1d1fc2 "Make the browser plugin compile with newer versions of xulrunner". It's trivial (just changes a char * to a const char *).
(In reply to comment #106) > Looks like I also need a74d58a2adf03b520fcd1ce3f134ef8fdc1d1fc2 "Make the > browser plugin compile with newer versions of xulrunner". It's trivial (just > changes a char * to a const char *). Yup, xulrunner changed API without bumping soname. That commit makes sense to drag along too.
PackageKit-0.6.17-2.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/PackageKit-0.6.17-2.fc15
PackageKit-0.6.17-2.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.