Bug 147537 - rpm-python transaction callback in rpm 4.4 adding too many transaction counts
Summary: rpm-python transaction callback in rpm 4.4 adding too many transaction counts
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: rawhide
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Jeff Johnson
QA Contact: Mike McLean
Depends On:
Blocks: 147484 147574
TreeView+ depends on / blocked
Reported: 2005-02-08 22:11 UTC by Seth Vidal
Modified: 2007-11-30 22:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2005-02-10 20:36:49 UTC

Attachments (Terms of Use)
debug log of what's going on. (12.62 KB, text/plain)
2005-02-08 22:11 UTC, Seth Vidal
no flags Details

Description Seth Vidal 2005-02-08 22:11:59 UTC
For programs using the rpm transaction callback in python the transaction counts
increment too high for the callback. The counts don't seem to match.

So a little debugging.

RPMCALLBACK_INST_OPEN_FILE is being called 2 extra times as is CLOSE_FILE.

Debugging output attached from rpm.setVerbosity(rpm.RPMLOG_DEBUG)

This is for a yum transaction of a single pkg.

When I talked about it with paul he suggested it might have to do with the
rollback patches that James applied. 
Afaict, it affects anaconda and should affect up2date, too.

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
Actual results:

Expected results:

Additional info:

Comment 1 Seth Vidal 2005-02-08 22:11:59 UTC
Created attachment 110837 [details]
debug log of what's going on.

Comment 2 Jeff Johnson 2005-02-08 23:36:37 UTC
Nope, this is pre and post everything transaction
scripts, not rollbacks.

Hmmm, yah, there's simpler way to accomplish this
task by extracting needed scriptlet tags at other places
rather than re-reading the package (with the associated
open/close callbacks). That's  like a couple hours work
to change.

However, I'll have to check with Gustavo, as smartpm may be
depending on the behavior (just like anaconda does) that
he implemented as part of installing packages.

No matter what, pre/post everything transaction scriptlets
are here to stay, although it will be a while before anyone

Comment 3 Jeremy Katz 2005-02-08 23:53:53 UTC
The new behavior seems to be screwing with anaconda's progress display
too, fwiw.  Thus if Gustavo is counting on the changed behavior, we
(preferably) want to sit down and come up with something that will
work for everyone.

Comment 4 James Olin Oden 2005-02-09 01:19:56 UTC
Just so you know, if the pre-post transaction scripts ever needed to 
be used in anaconda (ok, if someone was using them in their own 
packages with their own distro mistakingly think this new feature 
would work the same in anaconda as elsewhere) its likely some changes 
to anaconda would need to be made.  The scriptlets are pulled from 
the headers of all the packages, but anaconda in genhdlist strips 
scriptlets out (maybe it would not hit these though).  At anyrate, 
its going to try to get those scriptlets, requiring it to suck in the 
headers from all rpms before the transaction runs and after it runs.
Since that info is on the media that means switching through all 
cdroms 3 times.  This is my hunch, but I pretty sure that is what 
would happen.

I think what would be more desirable would be to keep the scriptlets 
in the header list headers, and aquire them from there, but even that 
is one fix for anaconda, but if there are any other installers trying 
to use rpm its not such a good thing.

Ultimately, though I like the feature and can definately see uses, 
there are problems with it in anaconda.

Comment 5 Jeremy Katz 2005-02-09 01:47:13 UTC
hdlist is dying, only a matter of how soon.  Then again, I've been
telling Jeff that for probably two years now ;)  But there's actually
a plan for how it's going to happen now and I expect it to be done for

At which point, we'll have to figure something probably to deal with
these because I'm sure that by then someone will be trying to use it :-)

Comment 6 Jeff Johnson 2005-02-09 02:03:21 UTC
Hmmm, I swear I added a comment. Oh well.

I'll have a fix next 24 hours.

Until hdlist is actually gone, try doing
    rpm -q -HT
which will simplify the QA on hdlist immensely.
Yah, the path nends to be parameterized, only the
installed system path currently because I'm a lazy schmuck ;-)

Comment 7 Jeff Johnson 2005-02-09 14:01:26 UTC
My other AWOL comment was releavnt.

Th rpmlib callback needs to be reworked, known many years now,
to permit applications to register for what events they wish to
be informed about, that would have avoided this problem.

And the progress needs to be handled by polling a shared data
structure, not by pushing information through the existing
callback API (where a uint32_t wired in concrete is preventing
vixing disk accounting).

And the hoary and timeless errors that are passed from rpmlib
simply are not adequate to describe current problems, like
multilib color, or autorelocated directory rename failure,
or selinux file context failure, any more.

This is a very silly state of affairs indeed, repairing a
callback mechanism to behave as it has always behaved, when
that behavior is not adequate and useful any longer in many

But I shall have a fix today.

Comment 8 Jeff Johnson 2005-02-10 07:02:41 UTC
I've checked with Gustavo, who has no special semantics
attached to the pre- and pots- everything callback, so
opening a package header a 4th and a 3rd time are just
a sub-optimal implementation in "working" rpm-4.4.1 code
that will be addressed momentarily.

Now if the (also sub-optimal) reopening of the original
package a 2nd time can be excised, then life would be
just peachy, wouldn't it?


Comment 9 Jeff Johnson 2005-02-10 11:50:06 UTC
FIxed in rpm-4.4.1-0.20.

That should buy you enough time to fix yum and anaconda
pkgcnt++, as that code is borken if it depends on this
notify callback always being done exactly the same
     p->fd = ts->notify(p->h, RPMCALLBACK_INST_OPEN_FILE, 0, 0,
                        rpmteKey(p), ts->notifyData);

The callback is stateless, and supplies the rpmteKey(p)
correctly. Please fix anaconda and yum code to be more
robust, whatever yum or anaconda has been doing has
*always* been broken even if not seen before Monday past.

Reassigning to yum to start the change ...

Comment 10 Jeremy Katz 2005-02-10 20:03:28 UTC
Well, the callback did change for rpm 4.1 but that didn't really
percolate into the python bindings.  And even looking at things, I
don't see rpmteKey differing between the RPMCALLBACK_INST_OPEN_FILE
even if I was doing it from C.  So I'm not sure exactly how the
callback being stateless or not matters.

Also, using INST_OPEN_FILE to get these scriptlets seems like a
significant overload of the implied meaning of the 'what' flags. 
Especially for things which don't do the pre-install download of
everything (ie, anaconda) as you're now having to download everything
more than once _or_ predownload everything which isn't really feasible
in huge transaction sets.  

The "right" thing to do really seems to be more to me that what
shouldn't be RPMCALLBACK_INST_OPEN_FILE and should instead be
appropriately).  Then, cases where they can be handled the same, you
can add things to case statements.  But where the behavior does need
to differ, it can and you don't break anything that's already existing.

Comment 11 Jeff Johnson 2005-02-10 20:36:49 UTC
I disagre. The better fix is to not do the callback at all,
there is no reason to read (and verify signatures etc etc)
repeatedly when the values can be snarfed and saved when
the header is originally stripped of data.

That change is more major than what can be undertaken
for rpm-4.4.1. The current patch is adequate to avoid
the problem until after rpm-4.4.1 is released.

There are many other problems with the existing callback
scheme, that addding Yet Another Enum cannot fix.

Deferred until rpm-4.4.2.

Comment 12 Jeff Johnson 2005-02-10 21:02:51 UTC
field atm that happens to have the predefined semantic
of indeed, returning a file descriptor for the package.

Adding other enum values with the same semantic is
certainly possible, but requires all applications to
implement and, indeed, return an open file descriptor
for previously unknown callback what.

Changing all applications that use the callback to return
an open fd is more of a change than fixing the two applications
that happen to be counting packages as side effect of the
callback, that might perhaps be able to figger out
from other args that the package has already been counted.

That was the rationale for assigning to yum.

Anyways, no callback at all is the better fix, and that's
not gonna happen in rpm-4.4.1. The current change, to test
for presence of pre- and post-transaction tags, which exist
in no package at all, is adequate fix.

Note You need to log in before you can comment on or make changes to this bug.