Bug 524982 - rebuilding .rpm from .drpm is 10x too slow
Summary: rebuilding .rpm from .drpm is 10x too slow
Alias: None
Product: Fedora
Classification: Fedora
Component: yum-presto
Version: rawhide
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Jonathan Dieter
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2009-09-22 21:29 UTC by John Reiser
Modified: 2009-10-07 18:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2009-10-07 18:11:49 UTC
Type: ---

Attachments (Terms of Use)

Description John Reiser 2009-09-22 21:29:53 UTC
Description of problem: Rebuilding the new .rpm from the .drpm and the old .rpm is too slow by a factor of at least ten (10).  Also, it is not parallelized with the download of the remaining .drpm files.  In common cases, the effect is to make the whole process slower than downloading the whole new .rpm, even at 1.5Mb/s DSL.

Updating rawhide from yesterday to today via "yum update": yum identified 138 packages totalling 151MB.  [That's about 17 to 19 minutes at 1.5Mb/s.]  Presto fetched 206kB of metadata, gave up on the *.noarch files (including 20MB evolution-help and 15MB gnome-games-help [what a waste!]), and in the end claimed to reduce the download from 150MB to 71MB, but also claimed that was only a 26% reduction.  After finishing the download, then presto took HALF AN HOUR of 100% CPU time to rebuild .rpms.  There are marvelous inventions called compilers which produce machine code.  Please use one, or get an algorithm that is not O(n*n).  Also, downloading is a light CPU load, so even a uni-processor machine can get good gains by using a rebuild thread in parallel with a download thread.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. "yum update" with yum-presto installed and enabled
Actual results: Rebuilding new .rpm from .drpm and old .rpm is very slow (tens of minutes for 100MB of new .rpm), and takes nearly 100% CPU time.

Expected results: Rebuilding new .rpm is quick, at least ten times as fast as currently, and is parallelized with remaining downloads.

Additional info:

Comment 1 Jonathan Dieter 2009-09-23 03:58:47 UTC
The problem is that in rebuilding our rpms, we have to recompress them with the same compression settings as the original rpm.  With xz compression, that means creating the rpm from the deltarpm is *over* 10x slower than with gzip, not because of our algorithm, but because of xz's algorithm.

The building of the deltarpms is parallelized with the download of deltarpms, but cannot be parallelized with the download of regular rpms because we have to actually *create* all the real rpms before allowing yum to download the remaining ones (otherwise, the md5 checksum error would result in yum failing to work, rather than redownloading the rpm).

Comment 2 Jonathan Dieter 2009-10-07 09:15:06 UTC
There have been reports that this has been fixed (by changing xz's default compression level in rpm).  Have you seen any change?

Comment 3 John Reiser 2009-10-07 18:11:49 UTC
Yes, today's and yesterday's "yum update" (with yum-presto enabled) have good wall-clock speed.  yum-presto still gives good to very good reduction in download size, although not the very good to excellent reduction of a couple weeks ago.

Note You need to log in before you can comment on or make changes to this bug.