Bug 201136 - yum work increases exponentially after first download of primary.xml.gz into sqlite
Summary: yum work increases exponentially after first download of primary.xml.gz into ...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: sqlite
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Peter Jones
QA Contact:
URL:
Whiteboard: bzcl34nup
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2006-08-03 00:41 UTC by Paul Dickson
Modified: 2014-01-21 22:54 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2008-05-07 00:43:32 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Paul Dickson 2006-08-03 00:41:09 UTC
Description of problem:
Yum will work for several hours after unpacking primary.xml.gz and loading the
sqlite database.  This is for the development repository.  After completing, the
next repository is downloaded.  The extras-development, although 30% larger,
only takes a few minutes (less than 5).

If I do a "rm -f /var/cache/yum/development/primary*", the next run of yum loads
the repository it just a minute or two.  But on the next run, the extremely long
run reoccurs.

The time has been increasing.  A recently completed run took 516 minutes to run.

Version-Release number of selected component (if applicable):
yum-2.9.3-1
sqlite-3.3.6-1.1
python-sqlite-1.1.7-1.2.1

How reproducible:
Always

Steps to Reproduce:
1. rm -f /var/cache/yum/development/primary*
2. time yum update
3. Answer NO
4. rm -f /var/cache/yum/development/cachecookie
4. time yum update
5. Answer NO
  
Actual results:
First run will take 1 to 5 minutes.
Second run will 2.5 to 4.5 hours to complete.

Expected results:
All runs should take only a few minutes.

Additional info:
The problem only occurs on a 300 MHz Pentium II notebook.

On a 1.6GHz Pentium M notebook, the problen DOES NOT occur.

Comment 1 Paul Dickson 2006-08-10 18:09:49 UTC
Another (important) data point.  On this system, /var/cache/yum/development is
mounted via NFS.  No other directory under /var/cache/yum is mounted that way
and they are free of the increased processing.

So it's the merging of the sqlite database over NFS that is causing the slow down.

A quick workaround is to delete the sqlite database before each run of yum:
    rm -f /var/cache/yum/development/primary*

Worse comes to worse, I can change the NFS volume to the package directory.

Comment 2 Seth Vidal 2006-08-10 18:31:37 UTC
sqlite and nfs being slow-ish would make sense.

Comment 3 Paul Dickson 2006-08-28 16:46:36 UTC
My setup is that /var/cache/yum/development is a symlink to an NFS mount hierarchy.

To work around this problem, I deleted the symlink and created the directory
/var/cache/yum/development, and under that, I created two symlinks for headers
and packages.  This eliminates the 2 to 4 hour wait caused by sqlite.

Comment 4 Valdis Kletnieks 2006-11-17 15:42:03 UTC
NFS seems to be a red herring.  I'm seeing the same issue, when /etc/yum.conf
contains "cachedir=/usr/src/yum" and /usr/src is a local ext3 filesystem.  The
problem seems to be worse for yum-updatesd - yum itself usually manages to be
not-so-piggy.

The *first* run after removing *.sqlite usually chews about 45 seconds to a
minute of CPU, the next few seem to take on the order of 5-10 minutes, and after
4-5 iterations, it's up in the hours range.

Tools like vmstat and top and gkrellm indicate 95% and higher user-mode, only
2-3% system CPU, and almost no actual I/O.  It's looking like a poor scaling
algorithm somewhere inside sqlite - like it's inserting stuff into the DB based
on a hash function, but it's a *poor* hash, so instead of choosing one of 10K
hash buckets with short 2-3 entry linked lists, it's colliding every time and
there's a 30K entry linked list hanging off one bucket.

Just hypothesizing, but that's my guess.

Comment 5 Paul Nasrat 2007-06-01 12:34:43 UTC
sqlite 3.3.17 has some performance improvements - please check from latest in
rawhide and see if it's still poor.

Comment 6 Red Hat Bugzilla 2007-08-21 05:24:39 UTC
User pnasrat's account has been closed

Comment 7 Bug Zapper 2008-04-03 17:55:06 UTC
Based on the date this bug was created, it appears to have been reported
against rawhide during the development of a Fedora release that is no
longer maintained. In order to refocus our efforts as a project we are
flagging all of the open bugs for releases which are no longer
maintained. If this bug remains in NEEDINFO thirty (30) days from now,
we will automatically close it.

If you can reproduce this bug in a maintained Fedora version (7, 8, or
rawhide), please change this bug to the respective version and change
the status to ASSIGNED. (If you're unable to change the bug's version
or status, add a comment to the bug and someone will change it for you.)

Thanks for your help, and we apologize again that we haven't handled
these issues to this point.

The process we're following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp

We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.

Comment 8 Bug Zapper 2008-05-07 00:43:30 UTC
This bug has been in NEEDINFO for more than 30 days since feedback was
first requested. As a result we are closing it.

If you can reproduce this bug in the future against a maintained Fedora
version please feel free to reopen it against that version.

The process we're following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp


Note You need to log in before you can comment on or make changes to this bug.