Bug 461995

Summary:

Severe performance regression in yum 3.2.19 when parsing exclude= lines.

Product:

[Fedora] Fedora

Reporter:

C. Scott Ananian <cscott>

Component:

yum

Assignee:

Seth Vidal <skvidal>

Status:

CLOSED UPSTREAM

QA Contact:

Fedora Extras Quality Assurance <extras-qa>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

CC:

ffesti, james.antill, katzj, pmatilai, tim.lauridsen

Target Milestone:

---

Target Release:

---

Hardware:

All

OS:

Linux

URL:

http://dev.laptop.org/ticket/8395

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2008-11-07 18:25:02 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Yum configuration which triggers the performance regression	none

Description C. Scott Ananian 2008-09-11 19:43:08 UTC

Created attachment 316477 [details]
Yum configuration which triggers the performance regression

Description of problem:

From laptop.org bug: http://dev.laptop.org/ticket/8395

The OLPC yum configuration uses an exclude= line with a large number
(400 or so) of packages named.  (We maintain a stable repository of
packages shipped in our image, but let end-users install packages we
don't ship using the f-9 repository; we exclude the packages we ship
from the f-9 repo to ensure that users don't inadvertently pick up an
incompatible update from f-9.)

In yum-3.2.17-2.fc9.noarch this worked fine.  We recently picked up
yum 3.2.19-3.fc9 as part of the fedora key transition.  In 3.2.19, yum
appears to hang.

Debugging, I found that it works fine if you remove the excludes=
line, but you can reproduce the hang by creating an excludes line of
(say) 1000 entries -- it doesn't matter what they are, I used
'abcd-1', 'abcd-2', etc since that was easy to generate by script.

This seems to be a significant performance regression in 3.2.19.

Version-Release number of selected component (if applicable): yum 3.2.19-3.fc9


How reproducible: always


Steps to Reproduce:
1. Take any working yum repository configuration
2. Add an excludes= line with about a hundred random package names. The named packages don't need to exist; I've used 'abcd-1', 'abcd-2', etc.
3. Try to install some package: 'yum install SDL-devel' (for example)
  
Actual results:

Yum appears to hang after printing:
"Excluding Packages from Fedora 9 - i386 - Updates Newkey"

Expected results:
Correct installation of the desired package.

Additional info:

I've attached the actual repo configuration used by OLPC in case that matters, but I expect that the recipe above will be easier to use to reproduce.

Comment 1 James Antill 2008-09-11 21:24:16 UTC

> 2. Add an excludes= line with about a hundred random package names.
>  The named packages don't need to exist; I've used 'abcd-1', 'abcd-2', etc.

 Ok, so these are full package names?
 On 3.2.17 we "gave up" in a number of situations where there was a large number of sql arguments (because SQLite failed), and did it all in python. In 3.2.18 we moved to always doing it in SQL but doing it multiple times (in groups of 73).

 Can you give a rough number for the amount of packages you are excluding?

Comment 2 seth vidal 2008-09-11 21:32:58 UTC

It's package names - and about 2200 of them in the list I have.

The patterns in excludePackages is what is taking up all the time.

If you comment out the patterns then the process completes quickly b/c of what you described above.

I think the easiest solution would be to do a check of the length of the patterns we're passing in. If it is > say 4x the MAX argument limit of sqlite then get the complete list and whittle it down w/parsePackages

Comment 3 James Antill 2008-09-11 22:01:15 UTC

 This is a test fix, it'll probably change a bit before it goes upstream, but if you really need it quick it should solve your problem:

http://people.redhat.com/jantill/yum/patches/sqlite-limit.patch

Comment 4 C. Scott Ananian 2008-09-12 04:01:59 UTC

Thanks, James.  We've downgraded to yum-3.2.17-2.fc9.noarch for the moment, so our need is not urgent.  Is it worth throwing your patch in our development repo anyway to get testing, or is it too dissimilar to what might land for that to be helpful?

Comment 5 seth vidal 2008-09-12 04:38:03 UTC

So, I've been looking at sqlite code tonight and I can't figure out where the idea of a limit on the number of glob/like patterns came from. All I've been able to find is that there is a limit on the number of bytes in all the arguments. But not the count of the arguments. So, now I'm beginning to wonder why PATTERNS_MAX exists in that way at all.

Comment 6 seth vidal 2008-09-12 04:38:45 UTC

Scott, 
 3.2.19 has a good number of improvements over 3.2.17 - you'd be better off if you're heading for a release to do 3.2.19 + that patch.

Comment 7 James Antill 2008-09-12 14:54:20 UTC

 This has been fixed upstream now, and should be in 3.2.20 etc.