Bug 147537 - rpm-python transaction callback in rpm 4.4 adding too many transaction counts
rpm-python transaction callback in rpm 4.4 adding too many transaction counts
Status: CLOSED DEFERRED
Product: Fedora
Classification: Fedora
Component: rpm (Show other bugs)
rawhide
All Linux
medium Severity medium
: ---
: ---
Assigned To: Jeff Johnson
Mike McLean
:
Depends On:
Blocks: 147484 147574
  Show dependency treegraph
 
Reported: 2005-02-08 17:11 EST by Seth Vidal
Modified: 2007-11-30 17:11 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-02-10 15:36:49 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
debug log of what's going on. (12.62 KB, text/plain)
2005-02-08 17:11 EST, Seth Vidal
no flags Details

  None (edit)
Description Seth Vidal 2005-02-08 17:11:59 EST
For programs using the rpm transaction callback in python the transaction counts
increment too high for the callback. The counts don't seem to match.

So a little debugging.

RPMCALLBACK_INST_OPEN_FILE is being called 2 extra times as is CLOSE_FILE.

Debugging output attached from rpm.setVerbosity(rpm.RPMLOG_DEBUG)

This is for a yum transaction of a single pkg.

When I talked about it with paul he suggested it might have to do with the
rollback patches that James applied. 
Afaict, it affects anaconda and should affect up2date, too.




Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Seth Vidal 2005-02-08 17:11:59 EST
Created attachment 110837 [details]
debug log of what's going on.
Comment 2 Jeff Johnson 2005-02-08 18:36:37 EST
Nope, this is pre and post everything transaction
scripts, not rollbacks.

Hmmm, yah, there's simpler way to accomplish this
task by extracting needed scriptlet tags at other places
rather than re-reading the package (with the associated
open/close callbacks). That's  like a couple hours work
to change.

However, I'll have to check with Gustavo, as smartpm may be
depending on the behavior (just like anaconda does) that
he implemented as part of installing packages.

No matter what, pre/post everything transaction scriptlets
are here to stay, although it will be a while before anyone
uses.
Comment 3 Jeremy Katz 2005-02-08 18:53:53 EST
The new behavior seems to be screwing with anaconda's progress display
too, fwiw.  Thus if Gustavo is counting on the changed behavior, we
(preferably) want to sit down and come up with something that will
work for everyone.
Comment 4 James Olin Oden 2005-02-08 20:19:56 EST
Just so you know, if the pre-post transaction scripts ever needed to 
be used in anaconda (ok, if someone was using them in their own 
packages with their own distro mistakingly think this new feature 
would work the same in anaconda as elsewhere) its likely some changes 
to anaconda would need to be made.  The scriptlets are pulled from 
the headers of all the packages, but anaconda in genhdlist strips 
scriptlets out (maybe it would not hit these though).  At anyrate, 
its going to try to get those scriptlets, requiring it to suck in the 
headers from all rpms before the transaction runs and after it runs.
Since that info is on the media that means switching through all 
cdroms 3 times.  This is my hunch, but I pretty sure that is what 
would happen.

I think what would be more desirable would be to keep the scriptlets 
in the header list headers, and aquire them from there, but even that 
is one fix for anaconda, but if there are any other installers trying 
to use rpm its not such a good thing.

Ultimately, though I like the feature and can definately see uses, 
there are problems with it in anaconda.
Comment 5 Jeremy Katz 2005-02-08 20:47:13 EST
hdlist is dying, only a matter of how soon.  Then again, I've been
telling Jeff that for probably two years now ;)  But there's actually
a plan for how it's going to happen now and I expect it to be done for
FC5.

At which point, we'll have to figure something probably to deal with
these because I'm sure that by then someone will be trying to use it :-)
Comment 6 Jeff Johnson 2005-02-08 21:03:21 EST
Hmmm, I swear I added a comment. Oh well.

I'll have a fix next 24 hours.

Until hdlist is actually gone, try doing
    rpm -q -HT
which will simplify the QA on hdlist immensely.
Yah, the path nends to be parameterized, only the
installed system path currently because I'm a lazy schmuck ;-)
Comment 7 Jeff Johnson 2005-02-09 09:01:26 EST
My other AWOL comment was releavnt.

Th rpmlib callback needs to be reworked, known many years now,
to permit applications to register for what events they wish to
be informed about, that would have avoided this problem.

And the progress needs to be handled by polling a shared data
structure, not by pushing information through the existing
callback API (where a uint32_t wired in concrete is preventing
vixing disk accounting).

And the hoary and timeless errors that are passed from rpmlib
simply are not adequate to describe current problems, like
multilib color, or autorelocated directory rename failure,
or selinux file context failure, any more.

This is a very silly state of affairs indeed, repairing a
callback mechanism to behave as it has always behaved, when
that behavior is not adequate and useful any longer in many
ways.

But I shall have a fix today.
Comment 8 Jeff Johnson 2005-02-10 02:02:41 EST
I've checked with Gustavo, who has no special semantics
attached to the pre- and pots- everything callback, so
opening a package header a 4th and a 3rd time are just
a sub-optimal implementation in "working" rpm-4.4.1 code
that will be addressed momentarily.

Now if the (also sub-optimal) reopening of the original
package a 2nd time can be excised, then life would be
just peachy, wouldn't it?

BWAHAHAHAHAHAHAHA!
Comment 9 Jeff Johnson 2005-02-10 06:50:06 EST
FIxed in rpm-4.4.1-0.20.

That should buy you enough time to fix yum and anaconda
pkgcnt++, as that code is borken if it depends on this
notify callback always being done exactly the same
forevermore:
     p->fd = ts->notify(p->h, RPMCALLBACK_INST_OPEN_FILE, 0, 0,
                        rpmteKey(p), ts->notifyData);

The callback is stateless, and supplies the rpmteKey(p)
correctly. Please fix anaconda and yum code to be more
robust, whatever yum or anaconda has been doing has
*always* been broken even if not seen before Monday past.

Reassigning to yum to start the change ...
Comment 10 Jeremy Katz 2005-02-10 15:03:28 EST
Well, the callback did change for rpm 4.1 but that didn't really
percolate into the python bindings.  And even looking at things, I
don't see rpmteKey differing between the RPMCALLBACK_INST_OPEN_FILE
even if I was doing it from C.  So I'm not sure exactly how the
callback being stateless or not matters.

Also, using INST_OPEN_FILE to get these scriptlets seems like a
significant overload of the implied meaning of the 'what' flags. 
Especially for things which don't do the pre-install download of
everything (ie, anaconda) as you're now having to download everything
more than once _or_ predownload everything which isn't really feasible
in huge transaction sets.  

The "right" thing to do really seems to be more to me that what
shouldn't be RPMCALLBACK_INST_OPEN_FILE and should instead be
something like RPMCALLBACK_PRE_OPEN_FILE and _POST_ (s/OPEN/CLOSE/
appropriately).  Then, cases where they can be handled the same, you
can add things to case statements.  But where the behavior does need
to differ, it can and you don't break anything that's already existing.
Comment 11 Jeff Johnson 2005-02-10 15:36:49 EST
I disagre. The better fix is to not do the callback at all,
there is no reason to read (and verify signatures etc etc)
repeatedly when the values can be snarfed and saved when
the header is originally stripped of data.

That change is more major than what can be undertaken
for rpm-4.4.1. The current patch is adequate to avoid
the problem until after rpm-4.4.1 is released.

There are many other problems with the existing callback
scheme, that addding Yet Another Enum cannot fix.

Deferred until rpm-4.4.2.
Comment 12 Jeff Johnson 2005-02-10 16:02:51 EST
RPMCALLBACK_INST_{OPEN,CLOSE}_FILE is the only what
field atm that happens to have the predefined semantic
of indeed, returning a file descriptor for the package.

Adding other enum values with the same semantic is
certainly possible, but requires all applications to
implement and, indeed, return an open file descriptor
for previously unknown callback what.

Changing all applications that use the callback to return
an open fd is more of a change than fixing the two applications
that happen to be counting packages as side effect of the
callback, that might perhaps be able to figger out
from other args that the package has already been counted.

That was the rationale for assigning to yum.

Anyways, no callback at all is the better fix, and that's
not gonna happen in rpm-4.4.1. The current change, to test
for presence of pre- and post-transaction tags, which exist
in no package at all, is adequate fix.

Note You need to log in before you can comment on or make changes to this bug.