Bug 147933 - Mem leak in 2.6.10-1.760_FC3
Mem leak in 2.6.10-1.760_FC3
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
3
i386 Linux
medium Severity high
: ---
: ---
Assigned To: Dave Jones
Brian Brock
:
Depends On:
Blocks: 147461
  Show dependency treegraph
 
Reported: 2005-02-13 09:06 EST by Mike Bird
Modified: 2015-01-04 17:16 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-05-22 02:11:46 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/proc/slabinfo from server up 39 hours (12.33 KB, text/plain)
2005-02-14 11:16 EST, Mike Bird
no flags Details
/proc/slabinfo, same server, uptime now 65 hours (12.33 KB, text/plain)
2005-02-15 13:25 EST, Mike Bird
no flags Details

  None (edit)
Description Mike Bird 2005-02-13 09:06:58 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041111 Firefox/1.0

Description of problem:
Our mail server ground to a standstill after 8 days uptime.  Nearly all of the
1GB RAM had leaked resulting in massive swapping.  Idle pegged at 0% and wait
running around 90%.  Have not noticed this problem on news server or workstations so leak there is either absent or much less.  All do significant disk I/O.  Difference may be heavy process creation on mail server.

Version-Release number of selected component (if applicable):
2.6.10-1.760_FC3

How reproducible:
Always

Steps to Reproduce:
1. Run a mail server under load for several days.
2. Monitor "free" output, -/+ buffers/cache.
3. Watch it grow.
  

Actual Results:  At end of 8 days we had 900MB used with all services stopped, exclusive of buffers and cache.

Expected Results:  Should have been under 200MB used with all services stopped, exclusive of buffers and cache.

Additional info:

Seems to have started with 2.6.10-1.760_FC3.  Might have started with 741 which we installed two weeks earlier but most likely 760.
Comment 1 Mike Bird 2005-02-13 22:06:50 EST
Two servers running sub-DS1 loads seem to be leaking both "bio" and "biovec-1"
at a rate of just under one million of each type of object per day - approx
110MB per day.

As yet, unable to develop a simple test case.  Have tried process creation, file
creation and deletion, user quotas.
Comment 2 Mike Bird 2005-02-13 22:31:27 EST
Leak seems to be related to setting up or tearing down TCP connections.  Problem
seems to be unrelated to volume of TCP data passed.

Assuming you have sshd configured to allow localhost to login to itself without
user interaction, this will demonstrate the leak:

  while true; do ssh localhost echo -n .; done

Monitor with slabtop.
Comment 3 Dave Jones 2005-02-13 22:37:09 EST
davem, does this ring a bell ?  ISTR some talk of networking related leaks post
2.6.10, so if something went into 2.6.11rc, it's possible we need that
backported until we rebase to 2.6.11.
Comment 4 David Miller 2005-02-14 11:02:21 EST
Can we get a /proc/slabinfo snapshot?  That will help me figure
out what is leaking exactly.

There were some fixes post-2.6.11rc, but they were related to ipv6
which I'm not sure these folks are using.

Hmmm...
Comment 5 Mike Bird 2005-02-14 11:16:26 EST
Created attachment 111058 [details]
/proc/slabinfo from server up 39 hours

Attachment shows 1.7M leaked of each of biovec-1 and bio resulting from normal
use of a server for 39 hours.  This server was not used for the various tests
described in this log.

Running following on system `A' results in leakage only on system `B'. 
Suggests `accept(2)' or something close to it is leaking.

  while true; do ssh B echo -n; done
Comment 6 David Miller 2005-02-14 11:27:15 EST
Leaked BIO objects resulting from networking leak is painfully
mysterious to me.

I thought you were saying that if you run the "ssh" loop, and
that's the entire test, you get leaks.

This may not be my department after all. :-)
Comment 7 Mike Bird 2005-02-15 13:25:27 EST
Created attachment 111098 [details]
/proc/slabinfo, same server, uptime now 65 hours

For comparison purposes, /proc/slabinfo with uptime now 65 hours.  Another 1M
each of leaked "bio" and "biovec-1" objects.  Remember this is one of several
servers hanging off a DS1.  A mail or news server handling DS3 loads with this
O/S is going to leak RAM at around 2-4GB per day - i.e. this O/S is unusable
for any but the smallest ISP's.

We'll have to reboot again in a day or two.  If you need more info before we
reboot, let me know soon.

BTW, bug is still assigned to davem who has disavowed knowledge of kernel
networking.  How do we get this reassigned?
Comment 8 David Miller 2005-02-15 15:13:11 EST
I have not disavowed knowledge of kernel networking, I am in fact
to person for those issues.

I have disavowed that this is even a networking problem.  What is
leaking is filesystem I/O buffers, not networking resources.  Therefore
I do not think it is a networking bug at all.

There have been a couple of BIO leaks in the 2.6.x kernel fixed over
the past month or two, so depending upon your configuration (using device
mapper, LVM, RAID, or similar) you could be hitting one of those problems.
Therefore someone more well versed in this area should take hold of this
bug.
Comment 9 Dave Jones 2005-02-15 15:15:15 EST
I'll take it back for now and have a trawl through the post 2.6.10 changes to
see if I can spot anything obvious.  Thanks for taking a look davem, feel free
to drop off the Cc:
Comment 10 Dave Jones 2005-02-15 15:17:20 EST
Mike, can you tell me some more info about the IO subsystem in use ? IDE ? SCSI
? MD ? Raid ?
Comment 11 Mike Bird 2005-02-15 15:58:48 EST
Apologies to davem for misunderstanding his comment.

The servers are each a mix of EXT3/LVM/RAID1/IDE and EXT3/RAID1/IDE.  Most of
the regular files (/var, /tmp, /usr etc) are on EXT3/RAID1/IDE without LVM.

Looks like problem doesn't occur on EXT3/IDE systems without RAID1.  Will
verify, investigate some more, and get back to you in an hour or two.
Comment 12 Mike Bird 2005-02-15 17:56:29 EST
Some kind of syslog-RAID1 leak, or just a coincidence?

Simply copying/deleting large file trees doesn't seem to cause leaks.
Leak occurs on EXT3/RAID1/IDE filesystems with syslog running.
Leak does not occur with syslog off (i.e. minilogd running).
We use LVM on some of these systems but not /tmp, /usr, /var.
Leak does not occur on systems A or B (see below) which are EXT3/IDE.
Leak doesn't occur on system R (see below) which is EXT3/RAID5/SCSI.
I don't have RAID1/SCSI or RAID5/IDE for comparison.

Stats on some systems (EXT3/RAID1/IDE except where noted otherwise):

System   Uptime  Est Daily  Est Syslogs    Active    Significant
                  Syslogs    Since Boot   bio objs   Services (3,6)
  A (5)  49 hrs      12k         25k           0k       none
  B (5)  49 hrs       1k          2k           0k    Workstation
  D      56 hrs     568k       1325k        2071k    QMail/Apache
  H      49 hrs      15k         31k         497k      Samba
  L      48 hrs     146k        292k         649k    Squid/QMail
  N (1)  68 hrs     429k       1215k        2937k     INN/QMail
  N (2)  68 hrs     696k       1972k        2937k     INN/QMail
  R (4)  57 hrs      58k        138k         132k   Samba/Squid
  S      56 hrs       5k         12k         160k      Apache
  U      68 hrs       6k         17k         140k    Workstation

Notes: (1) Counting only syslogged logs on EXT3/RAID1/IDE filesystems,..
       (2) Counting all syslogged logs.
       (3) Apache/Samba/Squid logs not counted as they're direct logs.
       (4) EXT3/RAID5/SCSI
       (5) EXT3/IDE
       (6) news.{crit,err,notice} counted but not INND direct logs.
Comment 13 Dave Jones 2005-02-15 18:36:24 EST
I just built a test kernel fixing up a bio leak in md. Can you try the -767
kernels at http://people.redhat.com/davej/kernels/Fedora/FC3/ and see if thats
any better ?
Comment 14 David Miller 2005-02-15 18:37:53 EST
One thing syslog does (that perhaps minilogd does not) is call
fsync() on the log every time it writes a new message there.

This might help narrow down the true cause.
Comment 15 Mike Bird 2005-02-15 19:16:21 EST
767 fixes problem in test case.  Will roll out to news server tonight and let
you know real world results.  Have to shake it down a bit before rolling out to
other servers.

Thanks.
Comment 16 Mike Bird 2005-02-15 20:39:16 EST
Just checked 766 which appeared on mirrors last night.  Problem exists in 766.

767 still looking good.  Early days.
Comment 17 Mike Bird 2005-02-16 16:13:36 EST
Confirming 767 fixes bug with no undesirable side-effects on the eight systems
we've tried it on thus far.  We'll be rolling it out to our other systems
tonight.  Thanks guys.
Comment 18 Dave Jones 2005-02-17 12:25:44 EST
thanks for testing, this will go out in the next official errata, though theres
a bunch of other things I'd like to get fixed before I push something else out.
Comment 19 Paul Jakma 2005-03-07 22:05:26 EST
Machine here hit this bug too. 770 has been running for a day now and problem
appears alleviated (bio slabs actually shrunk in half from an earlier peak
today). It took about a week to really become painfuly obvious previously
though, however looks good so far.
 
Comment 20 Jay Turner 2005-04-16 07:25:09 EDT
Moving this to PROD_READY based on the positive testing feedback.

Note You need to log in before you can comment on or make changes to this bug.