Bug 3956 - tar is very slow when /etc/passwd is very big
tar is very slow when /etc/passwd is very big
Status: CLOSED NEXTRELEASE
Product: Red Hat Linux
Classification: Retired
Component: tar (Show other bugs)
6.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: David Lawrence
http://www.engin.umich.edu/caen/syste...
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 1999-07-09 00:11 EDT by wingc
Modified: 2008-05-01 11:37 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 1999-07-13 18:00:06 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description wingc 1999-07-09 00:11:42 EDT
When /etc/passwd is huge (e.g. 100,000 users or more)*, tar
extractions can be excruciatingly slow. This is because,
although tar caches the last successful uid->username lookup
or username->uid lookup, it does not cache nonexistent user
names.

Thus, if /etc/passwd is huge and you have a tar file
containing many files owned by a user that is not on your
system, it will take forever to extract. (it will scan
through the entire /etc/passwd file for each file it
extracts)

The following patch makes tar extractions go about 100 times
faster in these cases, by caching non-existent user names,
group names, UIDs, and GIDs:

http://www.engin.umich.edu/caen/systems/Linux/code/patches/tar-1.12-faster.patch

(you should probably revert the "tar-1.12-namecache.patch"
in your Red Hat 6.0 tar RPM before applying this patch)


I have sent this patch to tar-bugs@gnu.ai.mit.edu, but if it
does not become part of the standard tar distribution, it
would be a good idea if it or something similar was included
in the next release of Red Hat Linux.

Thank you very much,

Chris Wing
wingc@engin.umich.edu



* - Yes, I know that Linux only supports 65,536 users
standard. I fixed that, too:

http://www.engin.umich.edu/caen/systems/Linux/highuids
Comment 1 Bill Nottingham 1999-07-09 12:45:59 EDT
Out of curiousity, does using nscd help at all?
Comment 2 wingc 1999-07-13 01:05:59 EDT
I haven't tried using nscd. Does nscd do any good if you aren't using
NIS?

In any case, I heard back from the maintainers of 'tar'-- the latest
version of GNU tar (1.13) contains a fix similar to mine.
Perhaps you should upgrade tar to v1.13 in the next version of Red Hat
Linux.
Comment 3 Bill Nottingham 1999-07-13 18:00:59 EDT
nscd might help some on huge passwd files; using db-based
passwd/group files would help more.

tar-1.13 will be in the next rawhide release.

Note You need to log in before you can comment on or make changes to this bug.