Bug 671377 - livecd-creator should remove /var/lib/rpm/__db* from images
livecd-creator should remove /var/lib/rpm/__db* from images
Product: Fedora
Classification: Fedora
Component: spin-kickstarts (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Bruno Wolff III
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2011-01-21 08:11 EST by Matthias Clasen
Modified: 2011-02-12 17:04 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2011-02-12 17:04:34 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Matthias Clasen 2011-01-21 08:11:27 EST
Panu points out that these have no place on the live cd:

Panu> will fix, but whatever creates the rpmdb on the image should actually eliminate the __db* files before creating the iso
Panu> the __db* stuff contains highly host-specific db locking information, cached data etc, and should not be copied around
Comment 1 Bruno Wolff III 2011-01-21 08:55:53 EST
Note that we still want live usb devices to be able to run yum or rpm.
Comment 2 Panu Matilainen 2011-01-21 09:04:14 EST
Whether the media is read-only or not is not relevant for the __db* files: they get created as needed, but they should not be copied from one system to another, much less included in any images.
Comment 3 Bruno Wolff III 2011-01-21 09:14:33 EST
yum and rpm should work to some extent even on read only media, though the changes won't persist. For usb devices, you can do permanent upgrades using the overlay if you created one on the device.

Does this problem show up when doing live installs? I wouldn't expect there to be a problem with having those files for a live image even when you won't normally be updating, as the live image is the host. Even when doing live installs, wouldn't the copy be valid, since you'll be installing to the same architecture and the same package set.
Comment 4 Bruno Wolff III 2011-01-21 09:15:36 EST
Isn't creating them on the fly going to make running yum on a live image pretty slow?
Comment 5 Bruno Wolff III 2011-01-21 09:33:59 EST
Another note is that if you do an update of a live usb aren't those files going to be recreated and again used for the install process if you do a live install from an updated live usb?
Comment 6 Panu Matilainen 2011-01-21 10:03:43 EST
I don't know the exact context where Matthias' problem came up, or much about live usb etc specifics. But you do not ever want to include the __db* files on installation images of any kind, period. Besides the potential problems they can cause, on read-only media they do NOTHING but waste space as the environment needs to be writable (even for read-only access to the actual database) to be usable. When it's not writable, a private (ie in-memory) environment is used.

I thought I had a sense of deja-vu wrt this, and indeed there's this in livecd-tools yuminst.py already:

    def _cleanupRpmdbLocks(self, installroot):
        # cleans up temporary files left by bdb so that differing
        # versions of rpm don't cause problems
        for f in glob.glob(installroot + "/var/lib/rpm/__db*"):

Makes me wonder what has made this to resurface, is there some new method to create images which bypasses this or something...
Comment 7 Bruno Wolff III 2011-01-21 10:50:29 EST
Live images aren't just installation media, they are also running systems with overlays so they are updatable even when the media is read only.

Doing the removal in yuminst.py seems like the right place to deal with this, though the way installs work you probably also should guard against updates running concurrently with the install. (It may in fact do that; I don't know.)
Comment 8 Panu Matilainen 2011-01-25 05:40:45 EST
Well, the __db* files belong to the writable overlay part along with other updatable parts of the system. Not in distributed images.
Comment 9 Bruno Wolff III 2011-01-25 09:11:11 EST
I disagree. I agree that is a good idea not to copy them for a live install. They could be in some intermediate state at that time if something using rpm/yum is running while the copy is happening. I see no reason that they can't be part of the live image. Removing them from a live image has performance implications. Recreating those files takes a significant amount of time and for cases where the update is done on an in memory overlay, you need to rebuild them again after rebooting if you need them again.
If they  really don't work on live images, I would expect that they also wouldn't work after normal systems reboot and that aren't currently getting rebuilt after every reboot.
Comment 10 Panu Matilainen 2011-01-25 09:45:39 EST
You know, those files get removed on each system boot anyway... so again, you're literally doing nothing but wasting space by including the __db* environment files on images.
Comment 11 Bruno Wolff III 2011-01-25 10:57:56 EST
No I didn't know that. That explains why sometimes yum appears to be much slower than usual. Given that, I have no objection to stripping those files out.
Comment 12 Bruno Wolff III 2011-01-25 11:40:37 EST
I'll look at doing this if no one beats me to it. These files take up a significant amount of space when uncompressed. Even compressed on the live image, its probably a noticeable amount of space.
I won't remove the code in liveinst.py that also removes these files, as that will still be needed if yum has been run on the live image.
Comment 13 Bruno Wolff III 2011-01-26 00:32:57 EST
I started taking a look at this and there is already code that appears to try to remove the db files after yum install is done, but it doesn't work. It might be something obvious to people who are more used to python than I am. I'm going to see if the code is actually getting called and if so what exact path it is trying to unlink.
Comment 14 Bruno Wolff III 2011-01-26 00:49:20 EST
It looks like the files do get properly removed part way through building the iso. So I suspect that something is recreating them again after that step.
Comment 15 Bruno Wolff III 2011-01-26 00:54:43 EST
The files are back by the time the man page stuff is done. I notice that a package list is printed out inbetween. I bet whatever generates the package list is what is causing the problem.
Comment 16 Bruno Wolff III 2011-01-26 00:58:37 EST
It looks like the problem is really in the live-base ks file. I'll be testing that.
Comment 17 Bruno Wolff III 2011-01-26 02:00:17 EST
I added a command to remove the db files in the ks files that ran rpm in the %post section. This change is upstream, but not in a released spin-kickstarts package. The nightlies should start using the fixed version within a day or so.
Comment 18 Bruno Wolff III 2011-02-12 17:04:34 EST
This should be fixed with spin-kickstarts-0.15.1-1.fc15 which is in rawhide and this version or the one from the mass rebuild should show up in F15 as soon as that is refreshed.

Note You need to log in before you can comment on or make changes to this bug.