Is there any way to make it run the scripts it failed to run? yum-complete-transaction just uninstalled the old version of libxml2, and did nothing else.

correction: yum-complete-transaction removed the old libxml2.i386, but left the duplicate libxml2.x86_64 installed, along with a few others...

[root@macbook dwmw2]#  yum-complete-transaction 
Loaded plugins: refresh-packagekit
There are 1 outstanding transactions to complete. Finishing the most recent one
The remaining transaction had 3 elements left to run
--> Running transaction check
---> Package libxml2.i386 0:2.7.2-1.fc9 set to be erased
--> Finished Dependency Resolution

================================================================================
 Package          Arch          Version              Repository           Size 
================================================================================
Removing:
 libxml2          i386          2.7.2-1.fc9          installed            1.7 M

Transaction Summary
================================================================================
Install      0 Package(s)         
Update       0 Package(s)         
Remove       1 Package(s)         

Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing        : libxml2                                           [1/1] 

Removed: libxml2.i386 0:2.7.2-1.fc9
Cleaning up completed transaction file
[root@macbook dwmw2]# package-cleanup --dupes | grep -v bluez-libs
Setting up yum
1:wpa_supplicant-0.6.3-6.fc9.x86_64
1:wpa_supplicant-0.6.4-2.fc9.x86_64
libxml2-2.7.2-1.fc9.x86_64
libxml2-2.7.2-2.fc9.x86_64
1:faad2-libs-2.6.1-5.fc9.x86_64
1:faad2-libs-2.6.1-6.fc9.x86_64
libtirpc-0.1.7-20.fc9.x86_64
libtirpc-0.1.7-21.fc9.x86_64
mdadm-2.6.7.1-1.fc9.x86_64
mdadm-2.6.4-4.fc9.x86_64

From the top:
1. the place where you cite yum using too much memory is actually rpm using too much memory. During the transaction yum isn't doing much beyond reporting what rpm is doing. You can refile it w/rpm but I'm pretty sure it'll be a dupe.

2. what version of yum and yum-utils is this?

This happened to me also, top showed that yum was using 3G of RAM which had pushed everything into swap space crippling the system.

Sure it's rpm code running at that stage, but that says precisely zero about who used up all that memory. It's yum process space, and the memory can be consumed by anything at all that has been used up to to that point: yum itself, any library it used up to that point (including rpm yes), python itself...

Rpm's memory peak occurs at fingerprinting stage, ie at the "Preparing..." phase, but that memory is freed and the actual transaction execution doesn't use huge amounts of memory. Certainly not 2.5 - 3GB for such a small transaction.

Pointing fingers to any particular piece is moot without further evidence. If somebody can actually reproduce this semi-reliably, looking at memory consumption at yum's "y/N" prompt would give some clues. For real hard data, valgrind would be best (massif log for example)

Panu, 
Fair enough.

This is the size of yum w/f10 repos enabled on my i686 laptop when I run:

  yum groupinstall kde-desktop

Virt   RSS SHR
58284  46m 6072 


That's after depsolving, but before downloading pkgs or testing the transaction

downloading pkgs doesn't increase the memory size. (It has to download 90 pkgs (245M) for this transaction).

During the test transaction the memory use is:
Virt   RSS SHR
71072  57m 8324

During the transaction we end up at:
Virt   RSS SHR 
77024  63m 8468

So, not an unreasonable growth.

Now, the two issues in this bug report is:

1. the reporter is using rpm and yum from f9 - which doesn't benefit from some of the changes you and florian have made in the rpm 4.6 series

2. the reporter is on x86_64 and probably dealing with multiple kernels at the same time.


So, I don't think anyone was trying to point any more blame anywhere, just trying to explain that when the transaction occurs the memory use is not obviously coming from anything that is going on outside of the transaction itself.

Does this line-up with what you've seen?

The numbers above line up with the expectations in normal circumstances. On x86_64 you can add a hefty amount of memory to the numbers, a transaction involving kernels can easily go into 250+ megs range (I only use x86_64 so this is familiar territory). But that's a LONG way from the 2.5 gigs still. This isn't the first time people have reported outrageous memory use on a relatively small transaction, it's just that quite apparently most people (me included) are not seeing it so the tricky part is to find out what triggers it, and which of all the involved pieces uses up all the memory.

There are all sorts of not-so-obvious things affecting memory use: just as an example, when selinux is used, the memory used during transaction grows gradually by significant amount (can be tens of megs), yet it is not a leak: libselinux is apparently doing it's own caching of looked-up contexts and these all get freed on matchpathcon_fini() at end of transaction. When looking at top, all you see is rpm hogging more and more memory when the memory consumed indirectly by another library. All it takes is a mechanism like that in one of the many many pieces involved to get confused and you'll have a disaster such as this.

> During a routine yum update, I noticed the machine swapping heavily, and 'top' > showed that yum was using 2.5GiB of RAM.
>
> I let it continue, and eventually saw this...

This to me suggests that it was actually using 2.5 gigs in very early stages, way before getting anywhere near the transaction - David, am I reading that right?

(In reply to comment #7)
> This to me suggests that it was actually using 2.5 gigs in very early stages,
> way before getting anywhere near the transaction - David, am I reading that
> right?

I'm afraid I'm not sure. I'm kind of in the habit of starting yum, leaving it alone for ten minutes or so and then coming back to see if it's managed to do anything useful yet. I don't even remember for sure whether this was after I'd come back to it and hit 'Y' to let it continue.

Panu,
 Why does the machine swapping and top showing the process as using 2.5g of ram lead you to believe it was already in use pre-transaction?

Here's what I did for a test:
import yum
import time

my = yum.YumBase()
my.repos.enableRepo('*-testing')
my.repos.enableRepo('rawhide')
my.repos.populateSack(mdtype='filelists')
my.repos.populateSack(mdtype='otherdata')
# at this point the memory use is 42M
filecount = 0
for pkg in my.pkgSack:
    print pkg
    print len(pkg.filelist)
    filecount += len(pkg.filelist + pkg.dirlist)
print "Total pkgs: %d" % len(my.pkgSack)
print "Total files: %d" % filecount
print "Waiting here for 30s, look at mem"

time.sleep(30)


On my i686 the maximum memory for:
Total pkgs: 23953
Total files: 3901203
Virt   Res 
1075M 1.0G

So, if David had 24K filelists all in memory then, yes, it is possible it consumed this memory before the transaction. In normal activity yum never traverses all the filelists, it searches for specific files and only loads the filelists for those pkgs.

I bet if he can duplicate the behavior we'll find it is exploding in size during the transaction or the transaction test.

 Ok, I changed Seth's script a bit:

http://people.redhat.com/jantill/yum/commands/yum-load-everything.py

...and on x86_64, with the above as-is, I get:

Before yum: Peak[241.01MB] Size[241.01MB] RSS[14.93MB]
Before repos: Peak[244.79MB] Size[244.79MB] RSS[19.47MB]
pkgsack BEG: PKG only: Peak[245.75MB] Size[245.75MB] RSS[20.70MB]
Excluding Packages from Fedora 9 - x86_64 - Test Updates
Finished
pkg pkgs   1000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   2000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   3000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   4000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   5000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   6000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   7000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   8000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   9000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  10000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  11000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  12000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  13000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  14000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  15000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  16000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  17000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  18000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  19000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  20000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  21000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  22000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  23000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  24000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  25000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  26000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  27000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  28000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  29000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  30000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  31000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  32000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  33000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  34000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  35000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  36000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  37000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs  38000: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkgsack END: PKG only: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
rpmsack BEG: PKG only: Peak[332.82MB] Size[332.82MB] RSS[107.42MB]
pkg pkgs   1000: Peak[349.47MB] Size[349.47MB] RSS[124.23MB]
pkg pkgs   2000: Peak[349.47MB] Size[349.47MB] RSS[124.23MB]
rpmsack END: PKG only: Peak[349.47MB] Size[349.47MB] RSS[124.23MB]
MID: Peak[349.47MB] Size[349.47MB] RSS[124.23MB]
Total pkgs: 40865
pkgsack BEG: Peak[349.47MB] Size[349.47MB] RSS[124.23MB]
pkg pkgs   1000: Peak[395.72MB] Size[395.72MB] RSS[170.46MB]
pkg pkgs   2000: Peak[445.72MB] Size[445.72MB] RSS[220.43MB]
pkg pkgs   3000: Peak[489.72MB] Size[489.72MB] RSS[264.06MB]
pkg pkgs   4000: Peak[524.72MB] Size[524.72MB] RSS[298.99MB]
pkg pkgs   5000: Peak[557.72MB] Size[557.72MB] RSS[332.14MB]
pkg pkgs   6000: Peak[593.72MB] Size[593.72MB] RSS[367.87MB]
pkg pkgs   7000: Peak[642.72MB] Size[642.72MB] RSS[417.88MB]
pkg pkgs   8000: Peak[693.72MB] Size[693.72MB] RSS[468.32MB]
pkg pkgs   9000: Peak[733.72MB] Size[733.72MB] RSS[508.18MB]
pkg pkgs  10000: Peak[806.72MB] Size[806.72MB] RSS[581.25MB]
pkg pkgs  11000: Peak[849.99MB] Size[849.99MB] RSS[624.50MB]
pkg pkgs  12000: Peak[898.99MB] Size[898.99MB] RSS[673.76MB]
pkg pkgs  13000: Peak[942.99MB] Size[942.99MB] RSS[717.11MB]
pkg pkgs  14000: Peak[989.99MB] Size[989.99MB] RSS[764.95MB]
pkg pkgs  15000: Peak[1.02GB] Size[1.02GB] RSS[820.29MB]
pkg pkgs  16000: Peak[1.08GB] Size[1.08GB] RSS[875.52MB]
pkg pkgs  17000: Peak[1.13GB] Size[1.13GB] RSS[931.57MB]
pkg pkgs  18000: Peak[1.19GB] Size[1.19GB] RSS[997.45MB]
pkg pkgs  19000: Peak[1.24GB] Size[1.24GB] RSS[1.02GB]
pkg pkgs  20000: Peak[1.29GB] Size[1.29GB] RSS[1.07GB]
pkg pkgs  21000: Peak[1.36GB] Size[1.36GB] RSS[1.14GB]
pkg pkgs  22000: Peak[1.42GB] Size[1.42GB] RSS[1.20GB]
pkg pkgs  23000: Peak[1.47GB] Size[1.47GB] RSS[1.25GB]
pkg pkgs  24000: Peak[1.54GB] Size[1.54GB] RSS[1.32GB]
pkg pkgs  25000: Peak[1.61GB] Size[1.61GB] RSS[1.39GB]
pkg pkgs  26000: Peak[1.71GB] Size[1.71GB] RSS[1.49GB]
pkg pkgs  27000: Peak[1.74GB] Size[1.74GB] RSS[1.52GB]
pkg pkgs  28000: Peak[1.79GB] Size[1.79GB] RSS[1.57GB]
pkg pkgs  29000: Peak[1.83GB] Size[1.83GB] RSS[1.61GB]
pkg pkgs  30000: Peak[1.86GB] Size[1.86GB] RSS[1.64GB]
pkg pkgs  31000: Peak[1.92GB] Size[1.92GB] RSS[1.70GB]
pkg pkgs  32000: Peak[1.96GB] Size[1.96GB] RSS[1.74GB]
pkg pkgs  33000: Peak[2.02GB] Size[2.02GB] RSS[1.80GB]
pkg pkgs  34000: Peak[2.06GB] Size[2.06GB] RSS[1.84GB]
pkg pkgs  35000: Peak[2.10GB] Size[2.10GB] RSS[1.88GB]
pkg pkgs  36000: Peak[2.15GB] Size[2.15GB] RSS[1.93GB]
pkg pkgs  37000: Peak[2.20GB] Size[2.20GB] RSS[1.98GB]
pkg pkgs  38000: Peak[2.24GB] Size[2.24GB] RSS[2.02GB]
pkgsack END: Peak[2.26GB] Size[2.26GB] RSS[2.04GB]
rpmsack BEG: Peak[2.26GB] Size[2.26GB] RSS[2.04GB]
rpm pkgs   1000: Peak[2.29GB] Size[2.29GB] RSS[2.07GB]
rpm pkgs   2000: Peak[2.35GB] Size[2.35GB] RSS[2.13GB]
rpmsack END: Peak[2.39GB] Size[2.39GB] RSS[2.17GB]
Total pkgs: 40865
Total files: 6505289
Total prco: 728762
Total ChangeLog: 0
END: Peak[2.39GB] Size[2.39GB] RSS[2.17GB]

> Panu,
> Why does the machine swapping and top showing the process as using 2.5g of ram
> lead you to believe it was already in use pre-transaction?

Because it's just as likely/unlikely as rpm consuming 2.5GB during package installation in transaction consisting of ~20 packages. *Something* is loading up a huge amount of data and forgetting to release it. If yum can load up > 2GB worth data before transaction, it can also forget to free it by the way of something holding onto a reference to the data, refcounting bug in one of the involved bindings or python itself etc. 

Guessing and betting does zero good, we need to find out what uses up that memory, and if it's already there before the transaction starts we have eliminated some possibilities.

Ok... the thing IS inside rpmtsRun(), I can reproduce the memory blow-up with just kernel-devel being "updated" alone in the transaction. The funny thing is, this doesn't show up in rpm cli usage at all (and that's why I was so sceptical about it as I've been looking the memory behavior of rpm a lot lately).

And why does it not show up in rpm cli use is that the memory explodes on the second transaction run only. Yum does a test transaction first, and this behaves as expected. The actual transaction then goes ballistic in ... surprise surprise, rpmdbFindFpList():

Running Transaction Test
rpmtsRun start                : 100688 kB
rpmtsRun sanity checking      : 101000 kB
rpmtsRun fingerprint start    : 102264 kB
rpmtsRun fingerprint stop     : 102268 kB
rpmtsRun TRANS_START          : 102268 kB
rpmtsRun TRANS_PROGRESS       : 102268 kB
rpmtsRun start findfplist     : 102268 kB
rpmdbFindFP start             : 102272 kB
rpmdbFindFP exit              : 152812 kB
rpmtsRun stop findfplist      : 152812 kB
rpmtsRun build shared         : 152812 kB
rpmtsRun done shared          : 152812 kB
rpmtsRun update diskinfo      : 152812 kB
rpmtsRun TRANS_STOP           : 152812 kB
rpmtsRun freed memory         : 152812 kB
rpmtsRun process pkg          : 152812 kB
rpmtsRun end                  : 152836 kB
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
rpmtsRun start                : 151292 kB
rpmtsRun sanity checking      : 151568 kB
rpmtsRun fingerprint start    : 152824 kB
rpmtsRun fingerprint stop     : 152828 kB
rpmtsRun TRANS_START          : 152828 kB
rpmtsRun TRANS_PROGRESS       : 152828 kB
rpmtsRun start findfplist     : 152828 kB
rpmdbFindFP start             : 152828 kB
rpmdbFindFP exit              : 774252 kB
rpmtsRun stop findfplist      : 774252 kB
rpmtsRun build shared         : 774252 kB
rpmtsRun done shared          : 774252 kB
rpmtsRun update diskinfo      : 774252 kB
rpmtsRun TRANS_STOP           : 774252 kB
rpmtsRun freed memory         : 774252 kB
rpmtsRun process pkg          : 774252 kB
rpmtsRun end                  : 774260 kB

If I remove the test transaction for similar use pattern with rpm cli, it completes with ~150MB RSS which is pretty in the expected range when overhead from yum is taken out. Now just to figure out what happens...

Interestingly, the mystery memory eater is not something allocating more and more memory like there's no tomorrow, but memory fragmentation resulting from thousands of realloc()'s to oddball sizes in rpmdbGrowIterator(). That also explains why the second run is more prone to blowing up than the first, and why it shows far more clearly under python than in rpm cli (as python is doing a whole lot of dynamic allocation of its own).

By reallocating in larger, even sized chunks, RSS comes back to a reasonable range. Just need to play around a bit to find a good reallocation scheme.

*** Bug 441262 has been marked as a duplicate of this bug. ***

This should be fixed in rawhide now. Backporting to 4.4.2.x is entirely possible too.

rpm-4.6.0-0.rc3.1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/rpm-4.6.0-0.rc3.1.fc10

rpm-4.6.0-0.rc3.1.fc10 has been pushed to the Fedora 10 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update rpm'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F10/FEDORA-2008-11332

rpm-4.6.0-0.rc3.1.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.

It would be interesting to know whether the resolution of this bug reduces the install requirements for fedora to 512MB from 764MB.

So on a system with 256 mb ram only a 256mb swap needs to be created instead of a 512mb swap as happens now. Perhaps memory usage during install might go down even further to 384mb during install so that a 128mb machine would only need 256 mb swap.

Just for informational purposes this bug might also be related to this one

https://bugs.launchpad.net/smart/+bug/244943

*** Bug 477564 has been marked as a duplicate of this bug. ***