Bug 24122 - tar sucks real memory and never free it !
tar sucks real memory and never free it !
Status: CLOSED NOTABUG
Product: Red Hat Linux
Classification: Retired
Component: tar (Show other bugs)
7.0
i686 Linux
high Severity high
: ---
: ---
Assigned To: Bernhard Rosenkraenzer
David Lawrence
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2001-01-16 12:35 EST by Bassem Jamaleddine
Modified: 2005-10-31 17:00 EST (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2001-02-13 13:38:32 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Updated package (307.75 KB, application/octet-stream)
2001-01-17 18:01 EST, Bernhard Rosenkraenzer
no flags Details

  None (edit)
Description Bassem Jamaleddine 2001-01-16 12:35:59 EST
When I used tar to move files around as in:
tar cvf - -C /source . | tar xvf -
and kept an eye on the memory consumed (using top),
I realized that the memory usage kept on escalating
and up to some 600M (more than half Giga) because
the source directory contained some 420M. And when
tar was done, the memory was never freed. The machine
has 4x256M=1G of memory. The file system is mounted on
scsi. I repeated the experience on a machine with different
mother board and with 256MB+256MB+128MB+128MB, and
IDE disks used, I had the same problem.
Other memory intensive applications, such as IBM WebSphere, do not suffer
of freeing the memory when the application terminate.
Comment 1 Bernhard Rosenkraenzer 2001-01-17 15:09:49 EST
I can't reproduce this...
Does it still happen for you with the current package from rawhide?
Comment 2 Bassem Jamaleddine 2001-01-17 17:53:10 EST
I don't know the package from rawhide. 'tar' is a basic and essential
unix archiver; to reproduce the problem, you need and real big directort\y
that you are trying yto move, say 400MB, then try to do:
tar cvf - -C /sourcedirectory  .  | tar xvf -
while you are in the destination directory. I assume you know what that
command does ! 
It is really a problem, since I reproduce it today again on a third server.
Comment 3 Bernhard Rosenkraenzer 2001-01-17 18:01:27 EST
Created attachment 7753 [details]
Updated package
Comment 4 Bernhard Rosenkraenzer 2001-01-17 18:08:53 EST
Please try the package from rawhide (I've attached it).
I've used it to move several GBs of files and this did not happen.

Comment 5 Bassem Jamaleddine 2001-01-18 23:25:13 EST
the "tar cvf - -C /opt . | tar xvf - " still not freeing the memory
even after the install of the rpm -Fvh tar-1.13.18-1.i386.rpm,
which superceeded the previous tar install.
memory escalated to mor ethan half gigabyte and never get to be freed.
Also notice that I issued a "CONTROL-C" in the middle of the process
when the memry get too highlt consumed:
 11:38am  up 1 day, 12:25,  2 users,  load average: 0.09, 0.26, 0.14
119 processes: 118 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  3.0% user,  6.7% system,  0.0% nice, 90.1% idle
Mem:  1036092K av,  657744K used,  378348K free,  284864K shrd,  299900K buff
Swap: 2097136K av,       0K used, 2097136K free  

Comment 6 Bernhard Rosenkraenzer 2001-01-19 09:33:54 EST
Are you sure you aren't counting buffered and cache memory as used? This is 
supposed to stay in memory (it stores used files and directory structures in memory 
so you don't have to access the disk every time; this type of memory is always freed 
when more free memory is needed, but not before)?
Comment 7 Preston Brown 2001-01-19 11:39:20 EST
Please post the results of the following test:

1. run free before doing the big tar operation
2. run the big tar operation, and near the end of the operation, do a ps
auxw|grep tar, and show us that line so we can see how much memory tar is
actually using
3. run free after the big tar operation.

Only with this complete amount of information can we really determine if this is
a real problem.
Comment 8 Bassem Jamaleddine 2001-01-19 11:50:59 EST
IF THAT IS THE CASE, THEN LOOK AT THESE FACTS:

*** AFTER THE FIRST TAR: tar cvf - -C /opt . | tar xvf -
11:44pm  up 2 days, 31 min,  2 users,  load average: 0.14, 0.14, 0.11
 120 processes: 119 sleeping, 1 running, 0 zombie, 0 stopped
 CPU states:  0.4% user,  1.7% system,  0.0% nice, 24.0% idle
 Mem:  1036092K av,  662680K used,  373412K free,  285808K shrd,  301160K buff
 Swap: 2097136K av,       0K used, 2097136K free                  257636K cached
 
*** AFTER RUNNING AN INTENSIVE JAVA APP, WEBSPHERE:
  11:48pm  up 2 days, 35 min,  2 users,  load average: 1.20, 0.65, 0.30
  203 processes: 202 sleeping, 1 running, 0 zombie, 0 stopped
  CPU states:  0.4% user,  1.7% system,  0.0% nice, 24.0% idle
  Mem:  1036092K av,  805464K used,  230628K free,  446048K shrd,  303488K buff
  Swap: 2097136K av,       0K used, 2097136K free                  277768K
cached

*** AFTER ANOTHER TAR:tar cvf - -C /tmp  . | tar xvf -
USING PAGING SPACE, THE SYSTEM WAS TERRIBLY SLOW EVEN FOR A vi SESSION !!!
 11:53pm  up 2 days, 40 min,  2 users,  load average: 1.24, 1.33, 0.69
 203 processes: 202 sleeping, 1 running, 0 zombie, 0 stopped
 CPU states:  0.4% user,  1.8% system,  0.0% nice, 24.1% idle
 Mem:  1036092K av, 1033020K used,    3072K free,  389580K shrd,  515628K buff
 Swap: 2097136K av,    7544K used, 2089592K free                  296540K cached
 
*** AFTER ANOTHER TAR: tar cvf - -C /opt . | tar xvf -
11:55pm  up 2 days, 42 min,  2 users,  load average: 1.59, 1.38, 0.76
205 processes: 204 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  6.0% user, 21.1% system,  0.0% nice, 72.7% idle
Mem:  1036092K av, 1034556K used,    1536K free,  391252K shrd,  427632K buff
Swap: 2097136K av,    7544K used, 2089592K free                  387564K cached

FINALLY, NOW THE SYSTEM IS PAGING... I BETTER SHUTDOWN AND RESTART.

(I'll do another test with free/ps auxw later on)
Comment 9 Bassem Jamaleddine 2001-01-19 21:15:08 EST
BEFORE STARTING THE:  tar cvf - -C /opt . | tar xvf -
ot@node1 /root]# free
     total       used       free     shared    buffers     cached
     Mem:       1036092     143512     892580     265112       4008      61572
     -/+ buffers/cache:      77932     958160
     Swap:      2097136          0    2097136
 
 
PS, DURING THE:  tar cvf - -C /opt . | tar xvf -
[root@node1 /root]# ps auxw | grep tar
root      1184  0.0  0.0  2080  972 tty1     S    08:18   0:00 sh
/usr/X11R6/bin/startx
root      1437  5.8  0.1  2104 1100 pts/5    S    09:20   0:04 tar cvf - -C /opt
.
root      1438  7.8  0.0  1880  880 pts/5    S    09:20   0:06 tar xvf -
root      1464  0.0  0.0  1600  624 pts/9    S    09:21   0:00 grep tar
 
PS, BEFORE A SECOND BEFORE THE END OF:  tar cvf - -C /opt . | tar xvf -
[root@node1 /root]# ps auxw | grep tar
root      1184  0.0  0.0  2080  972 tty1     S    08:18   0:00 sh
/usr/X11R6/bin/startx
root      1437  6.1  0.1  2104 1100 pts/5    D    09:20   0:05 tar cvf - -C /opt
.
root      1438  7.7  0.0  1880  880 pts/5    S    09:20   0:07 tar xvf -
root      1466  0.0  0.0  1600  624 pts/9    S    09:21   0:00 grep tar
 
 
FREE, AFTER THE END OF THE:  tar cvf - -C /opt . | tar xvf -
[root@node1 /root]# free
     total       used       free     shared    buffers     cached
     Mem:       1036092     607272     428820     275264     284064     228276
     -/+ buffers/cache:      94932     941160
     Swap:      2097136          0    2097136
 
 
TOP, AFTER THE TAR:  tar cvf - -C /opt . | tar xvf -
  9:25am  up  1:29,  1 user,  load average: 0.30, 0.46, 0.23
  113 processes: 111 sleeping, 2 running, 0 zombie, 0 stopped
  CPU states:  2.1% user,  7.0% system,  0.0% nice, 90.8% idle
  Mem:  1036092K av,  607232K used,  428860K free,  274900K shrd,  284064K buff
  Swap: 2097136K av,       0K used, 2097136K free                  228276K
cached
~
Comment 10 Bernhard Rosenkraenzer 2001-01-22 08:08:05 EST
Free indicates there are 17000 kB less available after the tar run (the "+/-
buffers/cache" line is the only one that matters here)... Did you start any
other processes (not necessarily ones you started manually, this could be a cron
or at job, or a server process if the machine is serving anything) that might
use up 17 MB?
In any case, 17 MB is nowhere near the 600 MB you were talking about. You
probably misinterpreted the "free" or "top" output.
Comment 11 Bernhard Rosenkraenzer 2001-02-06 13:06:24 EST
Assuming there is no problem due to lack of further input.
Comment 12 Bassem Jamaleddine 2001-02-13 13:36:47 EST
I really cannot see how do you that 17M only being in use in this:
FREE, AFTER THE END OF THE:  tar cvf - -C /opt . | tar xvf -
 [root@node1 /root]# free
                           total       used       free     shared    buffers    
cached
      Mem:       1036092     607272     428820     275264     284064     228276
      -/+ buffers/cache:      94932     941160
      Swap:      2097136          0    2097136
This is exacly from my last reply, showing free after the tar command
terminated.
This is a problem !
Comment 13 Bill Nottingham 2001-02-14 12:41:54 EST
If you'll look at the ps results posted, tar is only using 2MB of memory.

What you're seeing is just an increase in the buffers used by the kernel;
as it accesses data, it caches it.

Note You need to log in before you can comment on or make changes to this bug.