Bug 128335 - rpm doesn't handle sparse files efficiently
rpm doesn't handle sparse files efficiently
Product: Fedora
Classification: Fedora
Component: rpm (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: Paul Nasrat
Mike McLean
: FutureFeature
Depends On:
Blocks: FC6Blocker
  Show dependency treegraph
Reported: 2004-07-21 16:37 EDT by Jakub Jelinek
Modified: 2009-11-16 10:16 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2006-04-04 07:16:57 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Jakub Jelinek 2004-07-21 16:37:40 EDT
Some ELF programs/shared libraries are sparse.
E.g. on x86-64 if -Wl,-z,relro is used, most of the shared libraries
have around 1MB of zeros in the middle.
$ rpm -qf --qf '%{name}-%{version}-%{release}.%{arch}\n' /usr/lib64/gconv/IBM904.so
$ cp -a --sparse=always /usr/lib64/gconv{,.new}
$ du -sk /usr/lib64/gconv{,.new}
197972  /usr/lib64/gconv
6368    /usr/lib64/gconv.new

Either rpm could do what e.g. cpio --sparse does when unpacking,
or it could special case just ELF files which have sufficiently big
gap between PT_LOAD segments.
Comment 1 Jeff Johnson 2004-07-26 08:54:31 EDT
rpm decompresses into a mmap'd buffer, so sparse files
are filled in by zlib I believe.

Possibly blocks of zero's can be eliminated after installing.

Special handling for PT_LOAD and elf within cpio unpacking
is almost certainly not the right approach.
Comment 2 Ulrich Drepper 2004-07-26 11:07:19 EDT
> Special handling for PT_LOAD and elf within cpio unpacking
> is almost certainly not the right approach.

If you don't want to do this, it is required that you record somehow
somewere in the .rpm file which parts of the original files have been
sparsely allocated.  The problem is that you in general cannot just
force every file to be sparse.  There are noticeable difference: the
file layout will be non-sequential in some cases (-> performance), or
no disk space can be allocated for a file once data is written to the
sparse area.

On the o ther hand, these issues will never pop up with executables. 
So it makes sense to treat them special if you don't want to do it
100% right.
Comment 3 Jeff Johnson 2004-07-28 16:59:17 EDT
OK, I know how to implement sparse file handling in rpm for PT_LOAD
segment gaps.

If this change is actually necessary, rather than desirable,
then I suggest you expedite through other channels than bugzilla.
Comment 4 Jeff Johnson 2004-12-08 00:13:39 EST
NEEDINFO, I know what to do in rpm, but perhaps installing
all executables sparsely needs to be carefully thought through
before attempting an implementation.

I've never seen any unix user who was not surprised by, say,
    cp -R /usr /var/tmp
when there are sparse files that are involved,
one's disk space mysteriously vanishes. And yes I know
there are options to handle sparse files correctly, I
just don't believe those options are widely known or used.
Comment 5 Ulrich Drepper 2005-01-04 15:58:37 EST
I cannot see any problem with installing all binaries sparsely.  You
can recognize ELF binaries easily and then treat them appropriately.
Comment 6 Jeff Johnson 2005-01-07 20:10:08 EST
Sure there's no problem installing elf files sparsely in rpm.

The problem is the change when users do "cp -R" or any other
command that does not copy files sparsely. Disk space can/will
be used when sparse blocks are filled in, that is invariably
surprising to end-users.

Again, not my call. You want sparse file handling in rpm, it's
easier to implement than to discuss the merits of sparse files.

Off to distribution, please attach to some tracking bug so that
I have a clue when and where sparse file rpm installs are desired.
Comment 7 Paul Nasrat 2005-02-08 14:33:02 EST
Currently not being considered for U5 unless informed otherwise.

Note You need to log in before you can comment on or make changes to this bug.