Bug 353171 - non-existence of *.sqlite-journal not cached during install
Summary: non-existence of *.sqlite-journal not cached during install
Alias: None
Product: Fedora
Classification: Fedora
Component: yum   
(Show other bugs)
Version: rawhide
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Seth Vidal
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2007-10-25 20:10 UTC by John Reiser
Modified: 2008-01-25 20:40 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2008-01-25 20:40:52 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

Description John Reiser 2007-10-25 20:10:10 UTC
Description of problem: The installer does not remember that sqlite journaling
is not in effect.  These filenames are looked up much too often:
  16469 /tmp/cache/anaconda-base-200710250923.x86_64/primary.sqlite-journal
   1064 /tmp/cache/anaconda-base-200710250923.x86_64/filelists.sqlite-journal
(the number is the number of times that the lookup failed.)

Version-Release number of selected component (if applicable):

How reproducible: always

Steps to Reproduce:
1. as soon as vtty2 becomes available:
strace -f -o '|gzip' -p <pid-of-anaconda>  > strace.out &
(you have to provide strace on a USB flash memory device before boot, and mount
the device yourself.)
2. check strace.out (63MB compressed, 1GB uncompressed) for ENOENT involving
Actual results:
  16469 /tmp/cache/anaconda-base-200710250923.x86_64/primary.sqlite-journal
   1064 /tmp/cache/anaconda-base-200710250923.x86_64/filelists.sqlite-journal

Expected results: Only two failing lookups (once per filename.)

Additional info:

Comment 1 Chris Lumens 2007-10-26 15:44:12 UTC
Is there a way we can tell sqlite that there's not going to be a journal file so
it should not bother looking for one over and over again?

Comment 2 Panu Matilainen 2007-12-21 12:03:34 UTC
Checking for journal existence is part of sqlite locking mechanism (see
http://www.sqlite.org/lockingv3.html, "dealing with hot journals"). AFAICT the
only way to avoid checking for the journal is grabbing an exclusive lock on the
db (doable at least with "begin exclusive" SQL-statement) but I suppose that'd
have to be done on yum-level and that in turn would probably have some other,
not necessarily wanted, implications...

Comment 3 Jeremy Katz 2007-12-21 14:10:39 UTC
Actually, I don't know why we wouldn't want that.  At the yum level, we already
grab a global lock and don't allow other (correct :-) API users to do operations
at the same time.  So locking the sqlite dbs also doesn't seem unfair.

Comment 4 Panu Matilainen 2007-12-22 14:36:05 UTC
Well, a quick experiment of sticking the below sledgehammer-approach patch to
yum-metadata-parser cuts down the tests for journal existence on "yum
--enablerepo=updates-testing update" on my box ATM from 5488 to 7 :)

--- sqlitecachec.py.orig        2007-12-22 15:58:48.000000000 +0200
+++ sqlitecachec.py     2007-12-22 16:20:44.000000000 +0200
@@ -31,6 +31,9 @@
         con = sqlite.connect(filename)
         if sqlite.version_info[0] > 1:
             con.row_factory = sqlite.Row
+        cur = con.cursor()
+        cur.execute("pragma locking_mode = EXCLUSIVE")
+        del cur
         return con
     def getPrimary(self, location, checksum):

With sufficiently large number of accesses, it starts to even show up in
wallclock times: in apt-rpm usage patterns exclusive access to sqlite db cuts
something like ~7% from a full cache rebuild time. Yum's usage patterns are
wildly different but it can't hurt there either...

Comment 5 Seth Vidal 2008-01-25 20:40:52 UTC
panu, thank you for the patch, I've applied it to y-m-p and I'm going to check
put it in rawhide shortly.

Note You need to log in before you can comment on or make changes to this bug.