Description of problem: When processing large numbers of package updates, deltarpm reassembly can sometimes take longer than downloading the full packages. It would be excellent if yum would detect multi-processor/multi-core capability and use multiple threads to reassemble deltarpms. This would speed up updates substantially. Version-Release number of selected component (if applicable): yum-3.2.29-4.fc15.noarch How reproducible: Every time Steps to Reproduce: 1. yum update Actual results: Delta RPM reassembly happens one at a time Expected results: It would be preferable if it processed N rpms simultaneously, based on the available number of CPUs. Additional info:
I've just kicked off a build in Rawhide that should do just this. Thanks, Lars Bjønnes for the patch.
On a related note, Tom Horsley claims in https://lists.fedoraproject.org/pipermail/test/2012-May/108238.html that yum-presto appears to wait until all the drpms are fetched before starting rebuilding. Is this true, and does this patch solve that as well (since each drpm gets rebuilt in a new thread)?
(In reply to comment #2) > On a related note, Tom Horsley claims in > https://lists.fedoraproject.org/pipermail/test/2012-May/108238.html that > yum-presto appears to wait until all the drpms are fetched before starting > rebuilding. Is this true, and does this patch solve that as well (since each > drpm gets rebuilt in a new thread)? Yum-presto has had the rebuild in a separate thread almost from the beginning, specifically so the rebuild would be happening during drpm download. If that's not actually happening, someone should open a bug, but, at least from what I've seen, it's still working correctly. To test, do a large update, and then check where the deltarpm rebuild progress bar starts from once the deltarpms finish downloading. If it starts at any point higher than 0%, then it's working correctly.
Is it planned to get the threaded version of yum-presto into F16/F17?
The multi-threaded build version is in Fedora 17 (just checked and it's in the last two entries in the changelog), but will not be pushed to Fedora 16 as I believe it would violate the Fedora update policy. Multi-threaded downloads (which I don't think is what you're asking about) will only be available in Fedora 18. I'm going to go ahead and close this as currentrelease, but if you have a strong argument for pushing this to Fedora 16, I'd be happy to hear it.
Are you sure it's in F17? The changelog for yum-presto-0.9.0-1.fc18 has * Tue May 22 2012 Jonathan Dieter <jdieter> - 0.8.0-1 - Enable multi-threaded rebuild on systems with more than one CPU core (thanks Lars Gullik Bjønnes for the patch) (#701711) but yum-presto-0.7.3-1.fc17 (the current F17 version) was built in Koji on March 29 and has no such entry.
*Sigh* Color me stupid. I have a locally built version of yum-presto-0.8.1 installed on my laptop, which is what I checked. Sorry for the confusion. Unfortunately, I still think I shouldn't push it to F17 as the change was pretty invasive. I could be talked round, though, if you really think it's worth it.
In what sense is it invasive? Also, could you at least make a Koji build for F17 so people willing to risk it can try it out? If you do, I'll post a link to the test list. The single-threaded version seems to have lost ground in recent years since individual CPUs stopped getting faster, and there are some worrisome changes being made right now, such as the F18 DVD defaulting to downloading updated packages in full rather than using the original DVD versions (though one can check a box to prevent that if one knows where it is and that it's necessary).
I just installed yum-presto-0.9.0-1.fc18.noarch on my F17 box. No other dependencies pulled in, I'll keep an eye on it to see if it behaves. (Unfortunately, I don't have multicore hardware so I can't test if it gets the expected linear speedup.)
The changes are invasive in that they were done throughout yum-presto's code. I don't think there are any bugs, but bugs in yum-presto have nasty consequences, which is why I tend to be really paranoid. I've just kicked off builds for F16 and F17, but I'm definitely not putting the F16 ones into -testing, and I'm probably won't for F17 unless I get lots of "this is working perfectly" feedback.
Thanks! I installed the F17 Koji build. https://lists.fedoraproject.org/pipermail/test/2012-October/110857.html
Jonathan: Due to the fact that applydeltarpm maxes out my one CPU, I thought there would be a linear speedup (factor of N for N cores). One response to my post above got about a factor of 1 on 2 cores, the other about a factor of 2 on 4 cores. How much of a speedup have you been getting on your hardware, relative to the number of cores?
Probably about a factor of 2, but I don't have an SSD. I have noticed the numbers flipping around quite a bit more with the new code (i.e. goes from 2.1M/s to 500K/s and then back in the space of 30 seconds), but I haven't gotten around to checking on the reason yet. I'm assuming it's because my bottleneck is the hard drive. I did just run a test on my system, downgrading libreoffice and then updating it. I noticed that I only got the speed increase for the first few seconds of building, and then it went down to one core. The reason for this is that libreoffice-core is much larger than any of the other rpms, and applydeltarpm took far longer to build libreoffice-core than the other rpms combined. I'm not sure how often the above situation occurs, but I definitely wouldn't be claiming a linear increase given the huge variation in rpm size and the fact that, at least on rotating disks, applydeltarpm can be bound by the speed of the disk. I did read through the emails in the link above, and would comment that the reason that yum-presto has slowed over the years is that Fedora has changed from gzip compression to xz, and then the xz compression level increased by one.
OK, thanks for the explanation to slowdown of delta rebuild thru time. Today another update of 85MB size arrived with the kernel a firefox. The rebuils speed variability is really high, from 300kB-1MB here on SSD/Athlon X2 4400. Maybe would be nice to print the average at the end as for downloads, so we have a better number to compare, even the number will never be accurate.
Jonathan: I've been using the F17 Koji build on 3 single-core machines without any issues. You could put it in 17 updates-testing and set the karma high if you want.