Bug 3956 - tar is very slow when /etc/passwd is very big
Summary: tar is very slow when /etc/passwd is very big
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: tar
Version: 6.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: David Lawrence
QA Contact:
URL: http://www.engin.umich.edu/caen/syste...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 1999-07-09 04:11 UTC by wingc
Modified: 2008-05-01 15:37 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 1999-07-13 22:00:06 UTC


Attachments (Terms of Use)

Description wingc 1999-07-09 04:11:42 UTC
When /etc/passwd is huge (e.g. 100,000 users or more)*, tar
extractions can be excruciatingly slow. This is because,
although tar caches the last successful uid->username lookup
or username->uid lookup, it does not cache nonexistent user
names.

Thus, if /etc/passwd is huge and you have a tar file
containing many files owned by a user that is not on your
system, it will take forever to extract. (it will scan
through the entire /etc/passwd file for each file it
extracts)

The following patch makes tar extractions go about 100 times
faster in these cases, by caching non-existent user names,
group names, UIDs, and GIDs:

http://www.engin.umich.edu/caen/systems/Linux/code/patches/tar-1.12-faster.patch

(you should probably revert the "tar-1.12-namecache.patch"
in your Red Hat 6.0 tar RPM before applying this patch)


I have sent this patch to tar-bugs@gnu.ai.mit.edu, but if it
does not become part of the standard tar distribution, it
would be a good idea if it or something similar was included
in the next release of Red Hat Linux.

Thank you very much,

Chris Wing
wingc@engin.umich.edu



* - Yes, I know that Linux only supports 65,536 users
standard. I fixed that, too:

http://www.engin.umich.edu/caen/systems/Linux/highuids

Comment 1 Bill Nottingham 1999-07-09 16:45:59 UTC
Out of curiousity, does using nscd help at all?

Comment 2 wingc 1999-07-13 05:05:59 UTC
I haven't tried using nscd. Does nscd do any good if you aren't using
NIS?

In any case, I heard back from the maintainers of 'tar'-- the latest
version of GNU tar (1.13) contains a fix similar to mine.
Perhaps you should upgrade tar to v1.13 in the next version of Red Hat
Linux.

Comment 3 Bill Nottingham 1999-07-13 22:00:59 UTC
nscd might help some on huge passwd files; using db-based
passwd/group files would help more.

tar-1.13 will be in the next rawhide release.


Note You need to log in before you can comment on or make changes to this bug.