Bug 524982

Summary:	rebuilding .rpm from .drpm is 10x too slow
Product:	[Fedora] Fedora	Reporter:	John Reiser <jreiser>
Component:	yum-presto	Assignee:	Jonathan Dieter <jonathan>
Status:	CLOSED RAWHIDE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	medium	Docs Contact:
Priority:	low
Version:	rawhide	CC:	james.antill, jonathan, tim.lauridsen
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2009-10-07 18:11:49 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Reiser 2009-09-22 21:29:53 UTC

Description of problem: Rebuilding the new .rpm from the .drpm and the old .rpm is too slow by a factor of at least ten (10). Also, it is not parallelized with the download of the remaining .drpm files. In common cases, the effect is to make the whole process slower than downloading the whole new .rpm, even at 1.5Mb/s DSL.

Updating rawhide from yesterday to today via "yum update": yum identified 138 packages totalling 151MB. [That's about 17 to 19 minutes at 1.5Mb/s.] Presto fetched 206kB of metadata, gave up on the *.noarch files (including 20MB evolution-help and 15MB gnome-games-help [what a waste!]), and in the end claimed to reduce the download from 150MB to 71MB, but also claimed that was only a 26% reduction. After finishing the download, then presto took HALF AN HOUR of 100% CPU time to rebuild .rpms. There are marvelous inventions called compilers which produce machine code. Please use one, or get an algorithm that is not O(n*n). Also, downloading is a light CPU load, so even a uni-processor machine can get good gains by using a rebuild thread in parallel with a download thread.

Version-Release number of selected component (if applicable):
yum-presto-0.6.1-1.fc12.noarch.rpm

How reproducible:

Steps to Reproduce:
1. "yum update" with yum-presto installed and enabled
2.
3.

Actual results: Rebuilding new .rpm from .drpm and old .rpm is very slow (tens of minutes for 100MB of new .rpm), and takes nearly 100% CPU time.

Expected results: Rebuilding new .rpm is quick, at least ten times as fast as currently, and is parallelized with remaining downloads.

Additional info:

Comment 1 Jonathan Dieter 2009-09-23 03:58:47 UTC

The problem is that in rebuilding our rpms, we have to recompress them with the same compression settings as the original rpm.  With xz compression, that means creating the rpm from the deltarpm is *over* 10x slower than with gzip, not because of our algorithm, but because of xz's algorithm.

The building of the deltarpms is parallelized with the download of deltarpms, but cannot be parallelized with the download of regular rpms because we have to actually *create* all the real rpms before allowing yum to download the remaining ones (otherwise, the md5 checksum error would result in yum failing to work, rather than redownloading the rpm).

Comment 2 Jonathan Dieter 2009-10-07 09:15:06 UTC

There have been reports that this has been fixed (by changing xz's default compression level in rpm).  Have you seen any change?

Comment 3 John Reiser 2009-10-07 18:11:49 UTC

Yes, today's and yesterday's "yum update" (with yum-presto enabled) have good wall-clock speed.  yum-presto still gives good to very good reduction in download size, although not the very good to excellent reduction of a couple weeks ago.