For programs using the rpm transaction callback in python the transaction counts increment too high for the callback. The counts don't seem to match. So a little debugging. RPMCALLBACK_INST_OPEN_FILE is being called 2 extra times as is CLOSE_FILE. Debugging output attached from rpm.setVerbosity(rpm.RPMLOG_DEBUG) This is for a yum transaction of a single pkg. When I talked about it with paul he suggested it might have to do with the rollback patches that James applied. Afaict, it affects anaconda and should affect up2date, too. Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 110837 [details] debug log of what's going on.
Nope, this is pre and post everything transaction scripts, not rollbacks. Hmmm, yah, there's simpler way to accomplish this task by extracting needed scriptlet tags at other places rather than re-reading the package (with the associated open/close callbacks). That's like a couple hours work to change. However, I'll have to check with Gustavo, as smartpm may be depending on the behavior (just like anaconda does) that he implemented as part of installing packages. No matter what, pre/post everything transaction scriptlets are here to stay, although it will be a while before anyone uses.
The new behavior seems to be screwing with anaconda's progress display too, fwiw. Thus if Gustavo is counting on the changed behavior, we (preferably) want to sit down and come up with something that will work for everyone.
Just so you know, if the pre-post transaction scripts ever needed to be used in anaconda (ok, if someone was using them in their own packages with their own distro mistakingly think this new feature would work the same in anaconda as elsewhere) its likely some changes to anaconda would need to be made. The scriptlets are pulled from the headers of all the packages, but anaconda in genhdlist strips scriptlets out (maybe it would not hit these though). At anyrate, its going to try to get those scriptlets, requiring it to suck in the headers from all rpms before the transaction runs and after it runs. Since that info is on the media that means switching through all cdroms 3 times. This is my hunch, but I pretty sure that is what would happen. I think what would be more desirable would be to keep the scriptlets in the header list headers, and aquire them from there, but even that is one fix for anaconda, but if there are any other installers trying to use rpm its not such a good thing. Ultimately, though I like the feature and can definately see uses, there are problems with it in anaconda.
hdlist is dying, only a matter of how soon. Then again, I've been telling Jeff that for probably two years now ;) But there's actually a plan for how it's going to happen now and I expect it to be done for FC5. At which point, we'll have to figure something probably to deal with these because I'm sure that by then someone will be trying to use it :-)
Hmmm, I swear I added a comment. Oh well. I'll have a fix next 24 hours. Until hdlist is actually gone, try doing rpm -q -HT which will simplify the QA on hdlist immensely. Yah, the path nends to be parameterized, only the installed system path currently because I'm a lazy schmuck ;-)
My other AWOL comment was releavnt. Th rpmlib callback needs to be reworked, known many years now, to permit applications to register for what events they wish to be informed about, that would have avoided this problem. And the progress needs to be handled by polling a shared data structure, not by pushing information through the existing callback API (where a uint32_t wired in concrete is preventing vixing disk accounting). And the hoary and timeless errors that are passed from rpmlib simply are not adequate to describe current problems, like multilib color, or autorelocated directory rename failure, or selinux file context failure, any more. This is a very silly state of affairs indeed, repairing a callback mechanism to behave as it has always behaved, when that behavior is not adequate and useful any longer in many ways. But I shall have a fix today.
I've checked with Gustavo, who has no special semantics attached to the pre- and pots- everything callback, so opening a package header a 4th and a 3rd time are just a sub-optimal implementation in "working" rpm-4.4.1 code that will be addressed momentarily. Now if the (also sub-optimal) reopening of the original package a 2nd time can be excised, then life would be just peachy, wouldn't it? BWAHAHAHAHAHAHAHA!
FIxed in rpm-4.4.1-0.20. That should buy you enough time to fix yum and anaconda pkgcnt++, as that code is borken if it depends on this notify callback always being done exactly the same forevermore: p->fd = ts->notify(p->h, RPMCALLBACK_INST_OPEN_FILE, 0, 0, rpmteKey(p), ts->notifyData); The callback is stateless, and supplies the rpmteKey(p) correctly. Please fix anaconda and yum code to be more robust, whatever yum or anaconda has been doing has *always* been broken even if not seen before Monday past. Reassigning to yum to start the change ...
Well, the callback did change for rpm 4.1 but that didn't really percolate into the python bindings. And even looking at things, I don't see rpmteKey differing between the RPMCALLBACK_INST_OPEN_FILE even if I was doing it from C. So I'm not sure exactly how the callback being stateless or not matters. Also, using INST_OPEN_FILE to get these scriptlets seems like a significant overload of the implied meaning of the 'what' flags. Especially for things which don't do the pre-install download of everything (ie, anaconda) as you're now having to download everything more than once _or_ predownload everything which isn't really feasible in huge transaction sets. The "right" thing to do really seems to be more to me that what shouldn't be RPMCALLBACK_INST_OPEN_FILE and should instead be something like RPMCALLBACK_PRE_OPEN_FILE and _POST_ (s/OPEN/CLOSE/ appropriately). Then, cases where they can be handled the same, you can add things to case statements. But where the behavior does need to differ, it can and you don't break anything that's already existing.
I disagre. The better fix is to not do the callback at all, there is no reason to read (and verify signatures etc etc) repeatedly when the values can be snarfed and saved when the header is originally stripped of data. That change is more major than what can be undertaken for rpm-4.4.1. The current patch is adequate to avoid the problem until after rpm-4.4.1 is released. There are many other problems with the existing callback scheme, that addding Yet Another Enum cannot fix. Deferred until rpm-4.4.2.
RPMCALLBACK_INST_{OPEN,CLOSE}_FILE is the only what field atm that happens to have the predefined semantic of indeed, returning a file descriptor for the package. Adding other enum values with the same semantic is certainly possible, but requires all applications to implement and, indeed, return an open file descriptor for previously unknown callback what. Changing all applications that use the callback to return an open fd is more of a change than fixing the two applications that happen to be counting packages as side effect of the callback, that might perhaps be able to figger out from other args that the package has already been counted. That was the rationale for assigning to yum. Anyways, no callback at all is the better fix, and that's not gonna happen in rpm-4.4.1. The current change, to test for presence of pre- and post-transaction tags, which exist in no package at all, is adequate fix.