Bug 865020

Summary: doesn't clean up yumdb
Product: [Fedora] Fedora Reporter: udo <udovdh>
Component: yumAssignee: Fedora Packaging Toolset Team <packaging-team>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 21CC: brian, daniel.julien, ffesti, james.antill, jzeleny, maxamillion, packaging-team, pg7724, rtc, tim.lauridsen, vchepkov
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-03 06:13:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description udo 2012-10-10 16:04:40 UTC
Description of problem: yum does not clean up the yumdb, not even with a yum clean all.


Version-Release number of selected component (if applicable):
yum-3.4.3-29.fc17.noarch

How reproducible:
Use Fedora for a few years. 
Do a `yum clean all`.
See /var/lib/yum/yumdb/*

Actual results:
A history thingie about packages that aren't on the system anymore for at least a few years.

Expected results:
More uptodate database cleanup so that it is reasonable to have the info. after a fedora upgrade we can forget about older packages.

Additional info:

Comment 1 udo 2012-10-10 16:09:56 UTC
See https://bugzilla.redhat.com/show_bug.cgi?id=624964
Over here yumdb is 125 megabytes.
That costs and takes time when restoring as well. especially all those tiny files.
Blocksize is 4096 so the yumdb wastes quite a lot.

E.g.:

[root@surfplank2 g]# cd 7729f8921cf67a9a5796f0851441ee50ac3ac7bc-gimp-2.8.2-1.fc17-x86_64
[root@surfplank2 7729f8921cf67a9a5796f0851441ee50ac3ac7bc-gimp-2.8.2-1.fc17-x86_64]# ls -al
total 124
drwxr-xr-x.    2 root root  4096 Aug 31 15:24 .
drwxr-xr-x.  642 root root 86016 Oct  4 10:26 ..
-rw-r--r--. 2502 root root     3 Aug 11  2010 changed_by
-rw-r--r--.    1 root root    64 Aug 31 15:24 checksum_data
-rw-r--r--.   16 root root     6 Aug 31 15:24 checksum_type
-rw-r--r--.   16 root root     6 Aug 31 15:24 command_line
-rw-r--r--.  883 root root     7 Jan 14  2011 from_repo
-rw-r--r--.   15 root root    10 Aug 31 15:24 from_repo_revision
-rw-r--r--.   15 root root    10 Aug 31 15:24 from_repo_timestamp
-rw-r--r--. 2250 root root     4 Aug  2  2010 reason
-rw-r--r--. 2335 root root     2 Jul  6 14:48 releasever
[root@surfplank2 7729f8921cf67a9a5796f0851441ee50ac3ac7bc-gimp-2.8.2-1.fc17-x86_64]# du -sk .
40	.

That is 40 KB for 112 bytes of data? Make it 128 for a few nulls here and there.
That is 0.3% efficieny.
And then there is the lack of usability for the info beyond the current Fedora version.
Multiply by the millions of yum users and tadaa! you must have shares in seagate, hitachi, samsung, etc.

Comment 2 Zdeněk Pavlas 2012-10-11 11:04:42 UTC
This script works for me.  "yum clean yumdb" would be nice, still.
Mine yumdb is about 20MB, something I can live with..

#! /bin/bash
declare -A INS
for pkg in `rpm -qa`; do
  INS[$pkg]=1
done
for path in /var/lib/yum/yumdb/?/*; do
  pkg=${path#*-}
  pkg=${pkg%-*}.${pkg##*-}
  [ -z ${INS[$pkg]} ] && rm -rf $path
done

Comment 3 udo 2012-10-11 11:15:58 UTC
Zdeněk,

Thanks for the script but this does not solve the problems for the end user.
As I described the user interface should also clean out the yumdb with the `clean all` or at least provide a specific command to do so.
Furthermore the storage method is highly inefficient as I demonstrated.

On top of all that yum still complains when I installed/removed a package using rpm, as if that is abnormal, a sin or worse.
This issue has been discussed before but why must yum have another way or maybe even multiple ways of storing info that links very much to the rpm db.
Why can't yum not have a database like rpm's for all info that is related to rpm but not in the rpm database. That would be a lot more efficient.
All this aside of the actual reason for having a database for yum.

Yum just needs to find all dependencies when installing, removing, etc.
No further information should be stored and a simple log file should suffice.

Comment 4 Zdeněk Pavlas 2012-10-11 11:48:57 UTC
I think there's a reason why Yum does not provide similar functionality: Removing uninstalled packages from yumdb likely breaks "yum history" commands (this also rules out rpmdb piggybacking as yumdb storage).

There's no doubt the current yumdb implementation is inefficient space-wise.  But (from my POV as a Yum user) this does not matter that much- I'd rather see faster and more reliable MD updating, faster depsolving etc.

Comment 5 Fedora End Of Life 2013-04-03 16:58:05 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Comment 6 Paul Grabbe 2013-12-18 15:15:03 UTC
This issue persists in Fedora 20. After an upgrade via fedup I see the packages in my yumdb are all from prior releases starting with fc17. I don't see anything from fc20 there. 

I had done "yum clean all" before checking.

Comment 7 Henry Pfeil 2013-12-26 17:49:07 UTC
If I may be so bold as to jump in here, 
 `find /var/lib/yum/yumdb/ -name "*fc17*" | wc -l`
yields 2932 folders. However, 
 `yumdb search releasever 17`
shows only three packages currently installed.

`rpm -qa | grep fc17` says there are six packages left over from fc17. I checked, there are no more-recent versions for xfce4-doc, faad2 (+dev,libs), tango-icon-scheme or libfreebob.

(I know this isn't the place to start ranting about yum's reverse-dependency-hell. When you remove a package, it should remove only those files that package installed, not something installed by another package. I tried to remove an old perl module, but yum thought it would be a smart idea to delete the entire Xorg, KDE, XFCE, etc. trees, including bash. One file and my entire installation goes away. Fortunately the yes/no prompt came to the rescue.)

Comment 8 Fedora End Of Life 2015-01-09 17:25:06 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 9 Fedora End Of Life 2015-05-29 08:47:44 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 10 Brian J. Murrell 2015-05-31 16:01:23 UTC
Same here:

$ rpm -q fedora-release
fedora-release-21-2.noarch
$ find /var/lib/yum/yumdb -name \*.fc20-\* | wc -l
2541
$ find /var/lib/yum/yumdb -name \*.fc21-\* | wc -l
781
$ find /var/lib/yum/yumdb -name \*.fc19-\* | wc -l
26

Is there a reason I need previous Fedora release packages in my yumdb?

Comment 11 James Antill 2015-06-03 06:13:49 UTC
Yum will delete these entries if yum removes the package. fedup doesn't use yum though.

You can use "yumdb undeleted" to see the old yumdb entries, but as of F22 dnf is now the main packaging software so it makes little sense to wrkaround fedup in yum.

Comment 12 udo 2015-06-03 13:02:08 UTC
dnf is said to be the new yum.
it is at 1.0.0 and it doesn't even shwo us teh most basic stuff.
So how are we to be convinced that dnf will never do the stuff described in this bug?