Bug 353171 - non-existence of *.sqlite-journal not cached during install
non-existence of *.sqlite-journal not cached during install
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
All Linux
low Severity low
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2007-10-25 16:10 EDT by John Reiser
Modified: 2008-01-25 15:40 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2008-01-25 15:40:52 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description John Reiser 2007-10-25 16:10:10 EDT
Description of problem: The installer does not remember that sqlite journaling
is not in effect.  These filenames are looked up much too often:
  16469 /tmp/cache/anaconda-base-200710250923.x86_64/primary.sqlite-journal
   1064 /tmp/cache/anaconda-base-200710250923.x86_64/filelists.sqlite-journal
(the number is the number of times that the lookup failed.)

Version-Release number of selected component (if applicable):

How reproducible: always

Steps to Reproduce:
1. as soon as vtty2 becomes available:
strace -f -o '|gzip' -p <pid-of-anaconda>  > strace.out &
(you have to provide strace on a USB flash memory device before boot, and mount
the device yourself.)
2. check strace.out (63MB compressed, 1GB uncompressed) for ENOENT involving
Actual results:
  16469 /tmp/cache/anaconda-base-200710250923.x86_64/primary.sqlite-journal
   1064 /tmp/cache/anaconda-base-200710250923.x86_64/filelists.sqlite-journal

Expected results: Only two failing lookups (once per filename.)

Additional info:
Comment 1 Chris Lumens 2007-10-26 11:44:12 EDT
Is there a way we can tell sqlite that there's not going to be a journal file so
it should not bother looking for one over and over again?
Comment 2 Panu Matilainen 2007-12-21 07:03:34 EST
Checking for journal existence is part of sqlite locking mechanism (see
http://www.sqlite.org/lockingv3.html, "dealing with hot journals"). AFAICT the
only way to avoid checking for the journal is grabbing an exclusive lock on the
db (doable at least with "begin exclusive" SQL-statement) but I suppose that'd
have to be done on yum-level and that in turn would probably have some other,
not necessarily wanted, implications...
Comment 3 Jeremy Katz 2007-12-21 09:10:39 EST
Actually, I don't know why we wouldn't want that.  At the yum level, we already
grab a global lock and don't allow other (correct :-) API users to do operations
at the same time.  So locking the sqlite dbs also doesn't seem unfair.
Comment 4 Panu Matilainen 2007-12-22 09:36:05 EST
Well, a quick experiment of sticking the below sledgehammer-approach patch to
yum-metadata-parser cuts down the tests for journal existence on "yum
--enablerepo=updates-testing update" on my box ATM from 5488 to 7 :)

--- sqlitecachec.py.orig        2007-12-22 15:58:48.000000000 +0200
+++ sqlitecachec.py     2007-12-22 16:20:44.000000000 +0200
@@ -31,6 +31,9 @@
         con = sqlite.connect(filename)
         if sqlite.version_info[0] > 1:
             con.row_factory = sqlite.Row
+        cur = con.cursor()
+        cur.execute("pragma locking_mode = EXCLUSIVE")
+        del cur
         return con
     def getPrimary(self, location, checksum):

With sufficiently large number of accesses, it starts to even show up in
wallclock times: in apt-rpm usage patterns exclusive access to sqlite db cuts
something like ~7% from a full cache rebuild time. Yum's usage patterns are
wildly different but it can't hurt there either...
Comment 5 Seth Vidal 2008-01-25 15:40:52 EST
panu, thank you for the patch, I've applied it to y-m-p and I'm going to check
put it in rawhide shortly.

Note You need to log in before you can comment on or make changes to this bug.