Created attachment 115317 [details]
A testcase

Here is the list of rpms I was trying to update. I did

# rpm -Uvh `cat /tmp/update.list`

The memory usage of rpm went from 10MB to 33MB, then jump to 600MB, then 1.8G:

  PID USER	PR  NI	VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13524 root	18   0 1168g 1.7g 1.7g R 51.9 89.3   0:15.31 rpm

I had to kill it.

This is possibly an attempt to calculate md5 on a sparse file marked with %config.
The fix for that cannot be handled through normal batch upgrades, as the already installed
rpm has the problem. The fix is also isolated to the /var/log/lastlog by strcmp on path

Hmmm, I don't see the setup package within the list of packages. Perhaps there's another sparse file 
marked with %config within the set. If so, that will need treatment like /var/log/lastlog.

Created attachment 115321 [details]
A testcase

Here is a shorter list. I will keep searching.

Thanks. If the mechanism I think, look for a sparse file marked %config. The code path includes all files
on the file system from old packages as well as the install set that triggers.

strace shows an attempt to mmap() the sparse file, grep'ping out all the mmap calls
might confirm if attempting a digest on a sparse file is actually the cause. Perhaps
easier than binary search on the installed files.

Yes, it is /var/log/lastlog:

-r--------  1 root root 1254130450140 Jun 11 09:28 /var/log/lastlog

Upgrading rpm to version that avoids md5 digest on sparse /var/log/lastlog
marked with %config is first step.

Upgrading to a setup package that removes the %config marker is the 2nd step.

The complexities come from attempting to do both these steps at once, as
the old version of rpmlib (wich does not avoid /var/log/lastlog digest) and the old
version of setup (which has the %config marker on /var/log/lastlog) are both
involved in a singles step upgrade.

Divide and conquer ...

I couldn't find the new updated rpm for FC3. The only one I found for FC3 is
rpm-4.3.2-21.

No idea where the patch that fixes /var/log/lastlog has wandered. Verify with
    strings /usr/lib/librpm.a | grep lastlog
If the library contains the string "/var/log/lastlog", then you have the fix.

There is no /usr/lib/librpm.a in rpm-4.3.2-21. BTW, rpm-4.3.2-21 is the only
rpm for FC3. There is no update since the initial FC3 release.

Then look at /usr/lib/librpm*.so

And if on an elf64 system, /usr/lib64 instead.

I got

[hjl@gnu-9 kernel-FC3]$  strings /usr/lib64/librpm.so | grep lastlog
[hjl@gnu-9 kernel-FC3]$

Then that version of rpm does not have the fix.

Try rpm-4.4.1-21 packages from FC4, that almost certainly has the fix (can't check, sorry).

Renaming /var/log/lastlog while installing a version of setup that does
not include the %config marker.

The command "rpm -qc setup" or "rpm -qcp setup*.rpm" will spew
which files have %config markers.

An update has been released for this issue - please update rpm* in FC3.

*** This bug has been marked as a duplicate of 155730 ***