Bug 66818 - ldt allocation failed
ldt allocation failed
Status: CLOSED CURRENTRELEASE
Product: Red Hat Linux
Classification: Retired
Component: kernel (Show other bugs)
7.1
i686 Linux
high Severity high
: ---
: ---
Assigned To: Arjan van de Ven
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2002-06-17 07:30 EDT by william ewing
Modified: 2005-10-31 17:00 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2003-04-04 20:13:58 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description william ewing 2002-06-17 07:30:53 EDT
From Bugzilla Helper:
User-Agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 95)

Description of problem:
Shortly after ldt allocation failed messages appear on the screen the server 
freezes (although I can still ping it) and can only be unlocked with a re-boot.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.Allow over 1200 users to access HP OpenMail
2.
3.
	

Actual Results:  ldt allocation failed messages appear on the screen

Expected Results:  More usere should be able to access their E-Mial accounts

Additional info:

The server is a dual processor and I have tried the following combinations of 
memory to no avail
1. 4 GB of RAM & 1.8 GB of Swap
2. 4 GB of RAM & 4 x 1.8 GB Swap devices
3. 6 GB of RAM & 4 x 1.8 GB Swap devices
4. 1 GB of RAM & 4 x 1.8 GB Swap Devices
5. 2 GB of RAM & 1.8 GB Swap Device
Comment 1 Arjan van de Ven 2002-06-17 07:48:11 EDT
1) what kernel version are you using
2) do you have any idea how many threads there are from openmail when things
   start to stop working?
Comment 2 william ewing 2002-06-17 08:16:01 EDT
1. I have tried Kernel 2.4.2, 2.4.9-31gz and 2.4.9-34 in SMP and enterprise 
mode.
2. No, how can I tell what threads are being used ?
Comment 3 william ewing 2002-06-18 05:21:32 EDT
Could this be a GLIBC restriction on the number of Threads ?
Would I need to generate a libpthread.so ?
As the system is crashing everyday with over 1200 users on any help or guidance 
would be apprepricated
Comment 4 Arjan van de Ven 2002-06-18 06:43:10 EDT
ps -waux | wc -l

will give an approximation of the number of threads.
I doubt it's a glibc limitation (adding our glibc maintainter to CC to confirm)
but each thread will use in the order of 16Kb of unswappable kernel memory +
whatever else the threads do. The maximum amount of unswappable kernel memory is
limited to some 860Mb. 

You could try the kernel from the Advanced Server product, there is code in that
kernel to keep extra "unswappable kernel memory" available for situations like
yours.
Comment 5 Jakub Jelinek 2002-06-18 07:14:09 EDT
glibc has limitation on the amount of threads in one process (1024), but that
can be changed by recompiling glibc. How many threads is HP OpenMail using?
The reason why ldt allocation fails is more likely that there are no more 64KB
contiguous chunks of physical memory which are needed for LDT.
But it would much more likely trigger if HP OpenMail created 1200 separate
processes, each linked with -lpthread, thus each of those 1200 processes would
require allocation of 64K physically contiguous memory.
Comment 6 william ewing 2002-06-18 07:26:22 EDT
>But it would much more likely trigger if HP OpenMail created 1200 separate
>processes, each linked with -lpthread, thus each of those 1200 processes would
>require allocation of 64K physically contiguous memory.
 How can I tell if this is the case and if so what do I need to do to resolve 
the problem ?
Comment 7 Arjan van de Ven 2002-06-18 07:44:22 EDT
If Jakub's hypothesis is correct then HP needs to fix their program....

64Kb extra (+ the 16 normal Kb's per process) means the unswappable memory gets
exhausted fast. The Advanced Server kernel (2.4.9-e.3) will extend the limit a
bit, but not to inifinte. 

The correct way (and tons faster) to do this would be to not make 1200 threads
or processes but just a few that each handle several connections at once.
Comment 8 william ewing 2002-06-18 08:17:52 EDT
Is there any way that I can prove that this is the problem ?
ie Show the number of threads and how much unswappable kernel memory they use.

Am in right in thinking that there is 860MB of unswapable kernel memory.
How much does the system use as 860MB/1200 users = 700+ KB per user !

As my active user population is 1600, would the advanced server allow me to get 
these addition 300 to 400 users on ?
Is there any way to decrease the sixe of ldt ?
Can I run the Advanced server Kernel on Redhat 7.x ?
if so where would I get it from and would it need recompiled ?

Comment 9 Arjan van de Ven 2002-06-18 08:38:47 EDT
> As my active user population is 1600, would the advanced server allow me to get 
> these addition 300 to 400 users on ?

300 to 400 sounds possible, depending a bit on how the memory is used exactly.

>

ftp://ftp.redhat.com/pub/redhat/linux/enterprise/2.1AS/en/os/i386/SRPMS/

has the src.rpm of the kernel; easy to rebuild with rpm --rebuild --target=i686
kernel-2.4.9-e.3.src.rpm

(you'll need one or two other packages as well, same directory)
Comment 10 Arjan van de Ven 2002-06-18 09:07:14 EDT
http://people.redhat.com/arjanv/advanced_server_kernel
has it too
Comment 11 william ewing 2002-06-18 14:01:35 EDT
It would help me if you could answer the questions below

1. >Is there any way that I can prove that this is the problem ?
    >ie Show the number of threads and how much unswappable kernel memory they 
use.


2. >Am in right in thinking that there is 860MB of unswapable kernel memory.
>How much does the system use as 860MB/1200 users = 700+ KB per user !
Can you explain why my maths has gone wrong above 

3. >Is there any way to decrease the size of ldt ?

4. > Can I run the Advanced server Kernel on Redhat 7.x ?
Comment 12 Arjan van de Ven 2002-06-18 14:20:23 EDT
1) cat /proc/meminfo combined with cat /proc/slabinfo will give me enough info
so that I can say a "most probably" or "no"

2) The 860Mb is used for other things too, so it's not THAT simple. Also things
like open files, administration data for allocated memory etc etc scale with the
number of processes and come from this pool

3) Not that I know of currently

4) It works. If it works it might be a plan to go to AS completely though..
Comment 13 william ewing 2002-06-18 17:34:08 EDT
Taking into account that I have a dual processor server with upto 4 x 1.8 GB 
Swap devices and 1 to 6 GB of RAM what do you think is tyhe best combiantion to 
resolve my problem ? 
Am I likely to hit any VM or Big Memory problems ?
Is there an Advanced Kernel for 2.4.18 ?

Many Thanks for the Info so far. 
As I am a novice when I do a 
rpm -ivh --test kernel....
I get 
  initscripts < 6.41 conflicts with kernel-enterprise-2.4.9-e.3
  dev < 3.2-9 conflicts with kernel-enterprise-2.4.9-e.3
I take it I get these files from the redhat site ?
Comment 14 william ewing 2002-06-19 07:33:14 EDT
MEMINFO and SLABinfo details as requested 

With 1159 users on 
cat /proc/meminfo is
       total:    used:    free:  shared: buffers:  cached:
Mem:  2108424192 2099486720  8937472  4304896 34455552 802594816
Swap: 1846616064 596975616 1249640448
MemTotal:      2059008 kB
MemFree:          8728 kB
MemShared:        4204 kB
Buffers:         33648 kB
Cached:         249364 kB
SwapCached:     534420 kB
Active:         195712 kB
Inact_dirty:    624160 kB
Inact_clean:      1764 kB
Inact_target:   524284 kB
HighTotal:     1179632 kB
HighFree:         3116 kB
LowTotal:       879376 kB
LowFree:          5612 kB
SwapTotal:     1803336 kB
SwapFree:      1220352 kB       

cat /proc/slabinfo   is
slabinfo - version: 1.1 (SMP)
kmem_cache            80     80    244    5    5    1 :  252  126
ip_fib_hash           10    226     32    2    2    1 :  252  126
ip_conntrack           0      0    384    0    0    1 :  124   62
clip_arp_cache         0      0    128    0    0    1 :  252  126
ip_mrt_cache           0      0     96    0    0    1 :  252  126
tcp_tw_bucket          4     30    128    1    1    1 :  252  126
tcp_bind_bucket      147    226     32    2    2    1 :  252  126
tcp_open_request       0     40     96    0    1    1 :  252  126
inet_peer_cache        4     59     64    1    1    1 :  252  126
ip_dst_cache        2105   2200    192  110  110    1 :  252  126
arp_cache            471    630    128   21   21    1 :  252  126
blkdev_requests    44352  44360     96 1109 1109    1 :  252  126
dnotify cache          0      0     20    0    0    1 :  252  126
file lock cache      168    168     92    4    4    1 :  252  126
fasync cache           0      0     16    0    0    1 :  252  126
uid_cache           1214   1356     32   12   12    1 :  252  126
skbuff_head_cache   1340   1560    160   65   65    1 :  252  126
sock                1263   1284   1312  428  428    1 :   60   30
sigqueue              58     58    132    2    2    1 :  252  126
kiobuf                 0      0   8768    0    0    4 :    0    0
cdev_cache            64    177     64    3    3    1 :  252  126      
bdev_cache            11    177     64    3    3    1 :  252  126
mnt_cache             18    118     64    2    2    1 :  252  126
inode_cache        33728  42570    448 4730 4730    1 :  124   62
dentry_cache       27765  36600    128 1220 1220    1 :  252  126
dquot                  0      0    128    0    0    1 :  252  126
filp              106082 106120     96 2653 2653    1 :  252  126
names_cache            7      7   4096    7    7    1 :   60   30
buffer_head        82051 109880     96 2747 2747    1 :  252  126
mm_struct           1416   1416    160   59   59    1 :  252  126
vm_area_struct    113512 118413     64 2007 2007    1 :  252  126
fs_cache            1416   1416     64   24   24    1 :  252  126
files_cache         1346   1359    416  151  151    1 :  124   62
signal_act          1268   1332   1312  444  444    1 :   60   30
pae_pgd             1335   1469     32   13   13    1 :  252  126
size-131072(DMA)       0      0 131072    0    0   32 :    0    0
size-131072            0      0 131072    0    0   32 :    0    0
size-65536(DMA)        0      0  65536    0    0   16 :    0    0
size-65536             2      2  65536    2    2   16 :    0    0
size-32768(DMA)        0      0  32768    0    0    8 :    0    0
size-32768             4      4  32768    4    4    8 :    0    0
size-16384(DMA)        0      0  16384    0    0    4 :    0    0
size-16384             8      8  16384    8    8    4 :    0    0      
size-8192(DMA)         0      0   8192    0    0    2 :    0    0
size-8192              4      4   8192    4    4    2 :    0    0
size-4096(DMA)         0      0   4096    0    0    1 :   60   30
size-4096             57     58   4096   57   58    1 :   60   30
size-2048(DMA)         4      4   2048    2    2    1 :   60   30
size-2048           1120   1138   2048  565  569    1 :   60   30
size-1024(DMA)         1      4   1024    1    1    1 :  124   62
size-1024           1350   1352   1024  338  338    1 :  124   62
size-512(DMA)          0      0    512    0    0    1 :  124   62
size-512             712    712    512   89   89    1 :  124   62
size-256(DMA)          0      0    256    0    0    1 :  252  126
size-256             300    300    256   20   20    1 :  252  126
size-128(DMA)         68    120    128    4    4    1 :  252  126
size-128             840    870    128   29   29    1 :  252  126
size-64(DMA)           0      0     64    0    0    1 :  252  126
size-64             2419   2419     64   41   41    1 :  252  126
size-32(DMA)           0      0     32    0    0    1 :  252  126
size-32             5505   6215     32   55   55    1 :  252  126 


Comment 15 william ewing 2002-06-19 09:02:12 EDT
Latest Info.
  I have a test server (dual processor, 4 GB of RAm, 6 GB of SWAP) and RedHat 
7.3 installed. I have installed the Redhat 2.4.18-4 Big Memory Kernel and the 
Advanced Server Kernel. Greg Royle from HP OpenMail supplied me waith a Memory 
Grabbing and process using C program. (Each process allocates itself more 
memory every some nany seconds and a new process is generated every second).
Using a batch file I requested the program to add 100KB every 60 seconds to 
each process and then run it on both Kernels.
For the advanced Server Kernel , using Top, I used 1335 process.
For the Big memory Kernel , Using Top, I used 1342 process.
In both cases allow the program continued to run!, no more processes were 
generated and in each case I still had memory left over.
If required I can supply the C Program.
Can you supply me with any more information/suggestions ?
Comment 16 Arjan van de Ven 2002-06-19 15:32:08 EDT
programs can grab swapable memory just as well... but your system has over
100000 files open during the problem time, for example, and each open file takes
unswapable memory. The vm_area_struct number is high too, but we fixed some
things in the AS kernel for that.
Comment 17 william ewing 2002-06-19 17:10:35 EDT
Sorry not sure how you would like me to proceed ?
Do you want me to install the Advanced Server Kernel on the Live server and 
test?
If so is there any other values that I need to change in either the 
kernel ,  /proc or anywhere else (glibc ...) ?
Does the Advanced Server Kernel support large Memory ?

Or do you think that this application is unsuitable for Redhat ?

Comment 18 Michael K. Johnson 2002-06-19 18:16:07 EDT
The test program doesn't look like it has anything to do with the
problem.  I think that the only way to be sure that the Advanced
Server kernel fixes this particular problem for you without deploying
live would be to set up a test environment that duplicates your live
environment and do something like use fetchmail running on other
machines to try to duplicate the problem, then switch kernels and test
again.

The problem you are seeing shows up when there is contention for low
memory.  We did several things in Advanced Server to decrease demand
for low memory.  Exactly how much help it can be in this case is not
something that we can determine precisely here, but it should be an
improvement.

One note: if you wish to deploy this in a supported environment, the
Advanced Server kernel is only officially supported by Red Hat as
part of Red Hat Linux Advanced Server.  We've made it available for
testing, but aren't recommending a deployment of that kernel that is
not on the Red Hat Linux Advanced Server platform.
Comment 19 william ewing 2002-06-20 10:05:13 EDT
If this works I have no problem buying the Advanced server package.
What is the latest version of Kernel in the advanced server range as I believe 
that 2.4.18 and above has far better Big Memory support ?

Would there be any gain in using Redhat 7.3 with the 2.4.18-4 kernel?


Comment 20 Arjan van de Ven 2002-06-20 10:22:00 EDT
The AS kernel is called 2.4.9-e.3 but it has a LOT of changes from later kernels
included.
Comment 21 william ewing 2002-06-20 10:40:31 EDT
Is the following revelant 
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=43742
 as I am running Redhat 7.1 and on a dual processor server ?
If so was there a fix apart from going to REDHAT 7.2

Comment 22 Arjan van de Ven 2002-06-20 10:56:34 EDT
Jakub?
Comment 23 Jakub Jelinek 2002-06-20 11:20:33 EDT
There was an errata for 7.1 (well, at least 2, one including security fixes).
The latest one is http://rhn.redhat.com/errata/RHBA-2002-056.html
But I don't think #43742 has to do anything with this bugreport.
Comment 24 william ewing 2002-06-21 08:09:05 EDT
With regard to this 
>  I have a test server (dual processor, 4 GB of RAm, 6 GB of SWAP) and RedHat 
>7.3 installed. I have installed the Redhat 2.4.18-4 Big Memory Kernel and the 
>Advanced Server Kernel. Greg Royle from HP OpenMail supplied me waith a Memory 
>Grabbing and process using C program. (Each process allocates itself more 
>memory every some nany seconds and a new process is generated every second).
>Using a batch file I requested the program to add 100KB every 60 seconds to 
>each process and then run it on both Kernels.
>For the advanced Server Kernel , using Top, I used 1335 process.
>For the Big memory Kernel , Using Top, I used 1342 process.
>In both cases allow the program continued to run!, no more processes were 
>generated and in each case I still had memory left over.
I have now recompiled the c program WITHOUT lpthread.
Rerunning the tests I get about 1577 processes before the program generates a 
segmentation fault. If I run it again I get 3076 processes before the next 
segmentation fault and so on .... 
As the application uses lpthreads my test would seem to suggest that I can only 
generate 1342 processes in total so 1342  minus (system and HP OpenMAil 
processes) = about 1200 users allowed Access !
Is this correct ?
Is there a limit on the number of forks or processes that a parent process can 
generate ?
C program can be supplied. 
Comment 25 william ewing 2002-06-26 15:15:25 EDT
As each user access the HP Openamil system they are allocated a UAL.REMOTE 
process. This program was compiled with -lpthread. When this program is not 
compiled with lpthread the number of users allowed access has increased to the 
number requiring access. THe previous C program I mentioned was able to 
duplicate this no matter what Kernel I used. Can you explain why ?
Comment 26 Ben LaHaise 2002-11-05 15:25:42 EST
This bug will be fixed when NPTL is included in a future release.
Comment 27 Arjan van de Ven 2002-11-05 15:29:33 EST
It seems there's now an openmail version that does NOT link to pthreads
Comment 28 Need Real Name 2003-02-05 15:38:48 EST
I have seen similar problems when running both kernel-summit-2.4.9-e.3 and
kernel-summit-2.4.9-e.10 on an IBM x440 with 4 processors and 4gb RAM.  We are 
running into the problem when running apache and apache forks out over 1200 
processes.  You can reproduce easily enough by writing a small script that 
forks off a bunch of sleeps.  Here's how I reproduce:

for ((i=1;i<1000;i++)); do ./fork.sh; done


fork.sh looks like this:

#!/bin/sh
sleep 3600 &
Comment 29 Need Real Name 2003-02-05 16:21:27 EST
I should note that to reproduce it you'll have to run that command a few 
times.  Normally I get about 1296 sleeps before I hit the wall.  This is also 
reproducible on the uniprocessor and smp 2.4.18-19 kernels on the ibm x335 
series.  I would expect this behavior on these smaller machines, but would sure 
like to be able to run more apache children on my x440 if possible.  Is this 
likely better in an upcoming version of the advanced server kernel?
Comment 30 Ryan Rubley 2003-04-04 20:08:42 EST
I get this same problem on RH7.3 with apache configured to allow 2048 processes
running - after about 1300 are created, they start failing with the ldt
allocation failed messages - On a test machine I upgraded to a stock 2.4.20
kernel and the problem went away, but I would rather use RH approved kernels. 
2.4.18-27.7.x has the problem with ldt allocation I can verify.
Comment 31 Ben LaHaise 2003-04-04 20:13:58 EST
This problem is fixed in the 2.4.20 kernel released as part of Red Hat Linux 9

Note You need to log in before you can comment on or make changes to this bug.