Bug 722481

+++ This bug was initially created as a clone of Bug #721170 +++

Description of problem:

 I'm pretty sure I've got the details correct, but atm. the only reproducer I have is running a 20 minute "pungi" command. Anyway, what appears to be happening is that we do this:

cached_ro_ts = yb.rpmdb.ts
action_ts    = yb.ts
# Install a bunch of things
for h in cached_ro_ts.dbMatch('name', 'blah'):
   print "CTS", h['name'], h['epochnum'], h['version'], h['release'], h['arch']

for h in action_ts.dbMatch('name', 'blah'):
   print "ATS", h['name'], h['epochnum'], h['version'], h['release'], h['arch']

new_ts = initReadOnlyTS(root=self.root)
for h in action_ts.dbMatch('name', 'blah'):
   print "NTS", h['name'], h['epochnum'], h['version'], h['release'], h['arch']

...and we _only_ get answers from "ATS".

--- Additional comment from james.antill on 2011-07-13 18:24:36 EDT ---

 Note that this apparently worked with F15 GA of rpm/yum/pungi ... but fails now with rawhide. As I said, the test takes like 20 minutes, and you'll need a bunch of space on a VM etc.

 I'm testing a horrendous workaround atm. which is to do:

diff --git a/yum/__init__.py b/yum/__init__.py
index e9bc1f9..73155b8 100644
--- a/yum/__init__.py
+++ b/yum/__init__.py
@@ -1564,6 +1564,7 @@ class YumBase(depsolve.Depsolve):
             resultobject.return_code):
             self.rpmdb.dropCachedData()
         else:
+            self.rpmdb._ts = self.ts
             self.rpmdb.dropCachedDataPostTransaction(list(self.tsInfo))
 
         self.plugins.run('posttrans')

...I'll be mildly surprised if this doesn't break something, but at least the rpm lookups will work. And I assume pungi/yum will quit soon enough after, anyway I'll see what happens in about 20 minutes :).

--- Additional comment from james.antill on 2011-07-13 20:44:49 EDT ---

Ok, so the rpmdb._ts should be rpmdb.ts ... but apart from that it "works" ... except then pungi/yum dies because it can't find "nfs-utils-lib", but that appears to not actually have got installed ... it has a txmbr, but no callback and rpm/yum can't find it after pungi fails.

--- Additional comment from pmatilai on 2011-07-14 01:44:48 EDT ---

The thing is, you shouldn't be using more than one transaction set on a single db from a process. There's no good reason to do so, and rpm (sometime in the future) is likely to prevent that forcefully. (I've said this on yum-devel too but might've been missed by everybody :)

--- Additional comment from james.antill on 2011-07-14 01:46:57 EDT ---

 Ok, so after going through the post F15 changes and not seeing what could be causing this ... I decided to check, and the F15 GA versions of yum and rpm also fail.

--- Additional comment from james.antill on 2011-07-14 01:47:54 EDT ---

 Yeh, I know, and patches accepted ;).

--- Additional comment from james.antill on 2011-07-14 01:49:57 EDT ---

 Also note that I tried doing "del self.ts" ... and that didn't change anything, so the summary here is wrong (or, at least, incomplete).

--- Additional comment from james.antill on 2011-07-14 14:14:53 EDT ---

 Ok, so I downgraded to:

yum from F14
rpm from F15

...and now it "works", but only because yum isn't doing the checks anymore. After pungi has finished nfs-utils-lib is still not installed.

--- Additional comment from james.antill on 2011-07-14 15:27:02 EDT ---

 Ok, so testing with the rawhide yum/rpm again ... if I do this:

diff --git a/yum/__init__.py b/yum/__init__.py
index 12c2bdd..8c573c2 100644
--- a/yum/__init__.py
+++ b/yum/__init__.py
@@ -1564,6 +1564,7 @@ class YumBase(depsolve.Depsolve):
             resultobject.return_code):
             self.rpmdb.dropCachedData()
         else:
+            self.rpmdb.ts = self.ts
             self.rpmdb.dropCachedDataPostTransaction(list(self.tsInfo))
 
         self.plugins.run('posttrans')
diff --git a/yum/rpmsack.py b/yum/rpmsack.py
index ef6fbd5..cb0d089 100644
--- a/yum/rpmsack.py
+++ b/yum/rpmsack.py
@@ -375,6 +375,9 @@ class RPMDBPackageSack(PackageSackBase):
             pkg = self.searchNevra(n, e, v, r, a)
             if not pkg:
                 # Wibble?
+                if True:
+                    print "ERROR: ****:", txmbr
+                    continue
                 self._deal_with_bad_rpmdbcache("dCDPT(pkg checksums): %s" %
                                                txmbr)
                 continue

...then pungi "succeeds", even though nfs-utils-lib isn't installed. Dennis, I'm guessing you'll want to do that locally while we try to get to the bottom of the two-ish bugs:


 1. For some reason nfs-utils-lib isn't being installed, even though it's in the transaction. This is almost certainly an rpm thing, and maybe a packaging thing ... but nobody has noticed before yum started complaining loudly.

 2. Something changed so that we can't see the effects of the "action ts" from the other tses we have ... appears to happen even if we close the action ts. Yum is at least partly to blame here because 3.2.28-5 in F14 seems to be fine, which is only like 9 months of changes :).
 There have been changes where we added more tses in that time (like to fix the auto_close snafu), and removed them (Panu patches :).
 Any help, ideas etc. would be welcome.



 To "test" this you need a rawhide VM with a few GB of free space, install pungi and then get:

http://www.and.org/pungi-test-data

...changing the URL to whatever mirror is best for you (for rawhide). Put the second part of the file in temp.ks, and run the first part. About 20 minutes later it'll fail.

--- Additional comment from james.antill on 2011-07-14 16:41:51 EDT ---

 Ok, so I've found the #2 problem ... and it's nothing to do with rpm or yum.

 What happens is that pylorax/yumhelper.py does:

installroot = joinpaths(self.ybo.conf.installroot, "installroot")
os.makedirs(installroot)
self.ybo.conf.installroot = installroot

...but this is _after_ the rpmdb has been setup and told that .conf.installroot is the previous value. So the "action ts" is looking in /path/oldroot/installroot and all the other tses are looking in /path/oldroot ... shocker that they don't see the same data.

 If you _really_ need to do this, we'll probably have to add a .changeInstallRoot() API ... but that's going to be painful.

--- Additional comment from james.antill on 2011-07-14 17:32:46 EDT ---

 Ok, just confirmed that doing this change to pylorax/yumhelper fixes #2 ...

# After: self.ybo.conf.installroot = installroot
self.ybo.rpmdb.root = installroot
self.ybo.rpmdb.ts.close()

...that's not a useful upstream fix, as self.ybo.rpmdb._persistdir and cachedir (and maybe other things) are still broken. But it at least confirms the problem.