Red Hat Bugzilla – Bug 461995
Severe performance regression in yum 3.2.19 when parsing exclude= lines.
Last modified: 2014-01-21 18:06:12 EST
Created attachment 316477 [details] Yum configuration which triggers the performance regression Description of problem: From laptop.org bug: http://dev.laptop.org/ticket/8395 The OLPC yum configuration uses an exclude= line with a large number (400 or so) of packages named. (We maintain a stable repository of packages shipped in our image, but let end-users install packages we don't ship using the f-9 repository; we exclude the packages we ship from the f-9 repo to ensure that users don't inadvertently pick up an incompatible update from f-9.) In yum-3.2.17-2.fc9.noarch this worked fine. We recently picked up yum 3.2.19-3.fc9 as part of the fedora key transition. In 3.2.19, yum appears to hang. Debugging, I found that it works fine if you remove the excludes= line, but you can reproduce the hang by creating an excludes line of (say) 1000 entries -- it doesn't matter what they are, I used 'abcd-1', 'abcd-2', etc since that was easy to generate by script. This seems to be a significant performance regression in 3.2.19. Version-Release number of selected component (if applicable): yum 3.2.19-3.fc9 How reproducible: always Steps to Reproduce: 1. Take any working yum repository configuration 2. Add an excludes= line with about a hundred random package names. The named packages don't need to exist; I've used 'abcd-1', 'abcd-2', etc. 3. Try to install some package: 'yum install SDL-devel' (for example) Actual results: Yum appears to hang after printing: "Excluding Packages from Fedora 9 - i386 - Updates Newkey" Expected results: Correct installation of the desired package. Additional info: I've attached the actual repo configuration used by OLPC in case that matters, but I expect that the recipe above will be easier to use to reproduce.
> 2. Add an excludes= line with about a hundred random package names. > The named packages don't need to exist; I've used 'abcd-1', 'abcd-2', etc. Ok, so these are full package names? On 3.2.17 we "gave up" in a number of situations where there was a large number of sql arguments (because SQLite failed), and did it all in python. In 3.2.18 we moved to always doing it in SQL but doing it multiple times (in groups of 73). Can you give a rough number for the amount of packages you are excluding?
It's package names - and about 2200 of them in the list I have. The patterns in excludePackages is what is taking up all the time. If you comment out the patterns then the process completes quickly b/c of what you described above. I think the easiest solution would be to do a check of the length of the patterns we're passing in. If it is > say 4x the MAX argument limit of sqlite then get the complete list and whittle it down w/parsePackages
This is a test fix, it'll probably change a bit before it goes upstream, but if you really need it quick it should solve your problem: http://people.redhat.com/jantill/yum/patches/sqlite-limit.patch
Thanks, James. We've downgraded to yum-3.2.17-2.fc9.noarch for the moment, so our need is not urgent. Is it worth throwing your patch in our development repo anyway to get testing, or is it too dissimilar to what might land for that to be helpful?
So, I've been looking at sqlite code tonight and I can't figure out where the idea of a limit on the number of glob/like patterns came from. All I've been able to find is that there is a limit on the number of bytes in all the arguments. But not the count of the arguments. So, now I'm beginning to wonder why PATTERNS_MAX exists in that way at all.
Scott, 3.2.19 has a good number of improvements over 3.2.17 - you'd be better off if you're heading for a release to do 3.2.19 + that patch.
This has been fixed upstream now, and should be in 3.2.20 etc.