Bug 461995 - Severe performance regression in yum 3.2.19 when parsing exclude= lines.
Summary: Severe performance regression in yum 3.2.19 when parsing exclude= lines.
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: yum
Version: 9
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Seth Vidal
QA Contact: Fedora Extras Quality Assurance
URL: http://dev.laptop.org/ticket/8395
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-09-11 19:43 UTC by C. Scott Ananian
Modified: 2014-01-21 23:06 UTC (History)
5 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2008-11-07 18:25:02 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Yum configuration which triggers the performance regression (6.25 KB, application/octet-stream)
2008-09-11 19:43 UTC, C. Scott Ananian
no flags Details

Description C. Scott Ananian 2008-09-11 19:43:08 UTC
Created attachment 316477 [details]
Yum configuration which triggers the performance regression

Description of problem:

From laptop.org bug: http://dev.laptop.org/ticket/8395

The OLPC yum configuration uses an exclude= line with a large number
(400 or so) of packages named.  (We maintain a stable repository of
packages shipped in our image, but let end-users install packages we
don't ship using the f-9 repository; we exclude the packages we ship
from the f-9 repo to ensure that users don't inadvertently pick up an
incompatible update from f-9.)

In yum-3.2.17-2.fc9.noarch this worked fine.  We recently picked up
yum 3.2.19-3.fc9 as part of the fedora key transition.  In 3.2.19, yum
appears to hang.

Debugging, I found that it works fine if you remove the excludes=
line, but you can reproduce the hang by creating an excludes line of
(say) 1000 entries -- it doesn't matter what they are, I used
'abcd-1', 'abcd-2', etc since that was easy to generate by script.

This seems to be a significant performance regression in 3.2.19.

Version-Release number of selected component (if applicable): yum 3.2.19-3.fc9


How reproducible: always


Steps to Reproduce:
1. Take any working yum repository configuration
2. Add an excludes= line with about a hundred random package names. The named packages don't need to exist; I've used 'abcd-1', 'abcd-2', etc.
3. Try to install some package: 'yum install SDL-devel' (for example)
  
Actual results:

Yum appears to hang after printing:
"Excluding Packages from Fedora 9 - i386 - Updates Newkey"

Expected results:
Correct installation of the desired package.

Additional info:

I've attached the actual repo configuration used by OLPC in case that matters, but I expect that the recipe above will be easier to use to reproduce.

Comment 1 James Antill 2008-09-11 21:24:16 UTC
> 2. Add an excludes= line with about a hundred random package names.
>  The named packages don't need to exist; I've used 'abcd-1', 'abcd-2', etc.

 Ok, so these are full package names?
 On 3.2.17 we "gave up" in a number of situations where there was a large number of sql arguments (because SQLite failed), and did it all in python. In 3.2.18 we moved to always doing it in SQL but doing it multiple times (in groups of 73).

 Can you give a rough number for the amount of packages you are excluding?

Comment 2 seth vidal 2008-09-11 21:32:58 UTC
It's package names - and about 2200 of them in the list I have.

The patterns in excludePackages is what is taking up all the time.

If you comment out the patterns then the process completes quickly b/c of what you described above.

I think the easiest solution would be to do a check of the length of the patterns we're passing in. If it is > say 4x the MAX argument limit of sqlite then get the complete list and whittle it down w/parsePackages

Comment 3 James Antill 2008-09-11 22:01:15 UTC
 This is a test fix, it'll probably change a bit before it goes upstream, but if you really need it quick it should solve your problem:

http://people.redhat.com/jantill/yum/patches/sqlite-limit.patch

Comment 4 C. Scott Ananian 2008-09-12 04:01:59 UTC
Thanks, James.  We've downgraded to yum-3.2.17-2.fc9.noarch for the moment, so our need is not urgent.  Is it worth throwing your patch in our development repo anyway to get testing, or is it too dissimilar to what might land for that to be helpful?

Comment 5 seth vidal 2008-09-12 04:38:03 UTC
So, I've been looking at sqlite code tonight and I can't figure out where the idea of a limit on the number of glob/like patterns came from. All I've been able to find is that there is a limit on the number of bytes in all the arguments. But not the count of the arguments. So, now I'm beginning to wonder why PATTERNS_MAX exists in that way at all.

Comment 6 seth vidal 2008-09-12 04:38:45 UTC
Scott, 
 3.2.19 has a good number of improvements over 3.2.17 - you'd be better off if you're heading for a release to do 3.2.19 + that patch.

Comment 7 James Antill 2008-09-12 14:54:20 UTC
 This has been fixed upstream now, and should be in 3.2.20 etc.


Note You need to log in before you can comment on or make changes to this bug.