I have a related problem so I thought I would search bugzilla, and here we are
;)

What happens is that efence keeps freeed pages mapped (to protect them), if you
application however does a lott of alloc/freeing this results in it running out
of memory. So I dunno if this really is a bug.

I find it really hard to believe that the problem I'm experiencing is due to the
application running out of memory because of pages saved by ElectricFence. 
First of all, it happens shortly after the program starts up, when I don't
believe it's done enough to consume a significant amount of memory.  Second, I
used to be able to run the same application with EF_PROTECT_FREE for days at a
time.  Third, my system has 512mb of RAM and 265mb of swap space, hardly amounts
which are easy to fill up.  Fourth, I just duplicated the problem, and
immediately before the application crashed, I had 6mb of memory and 237mb of
swap space free.

Okay, I've been thinking about this some more, and the real problem is not that
teh app runs out of memory, but it runs out of virtual memory space, ehat
happens is that efence maps a protected page to each adres when it is freed, to
protect it, hence if the application allocs/frees a lott of stuff, all its
virtual memory space will; be filled with mappings to the protected page. so it
doesn't use a lott of memory (since it is all mapped to one page), butit has all
if its virtual memory space filled with mappings. Try stracing the application,
you'll see that the address hint for the last (failed) mmap passes the 3G memory
limit (this is the size of the user space segment under linux). This is what
causes the invalid parameter error, I've also tried it with a patched version
which wrapped around startAddr (see page.c) if (startAddr + size) > 0xC0000000
this however just changed the invalid parameter error to an out of memory error.

IOW, the protect free pages option just doesnot work with programs which do a
lott of allocs/frees, since it keeps al virtual memory space ever used mapped to
a protected page causiong some programs to run out of virtual memory.

assigned to jakub

Jakub do you need something to reproduce this, I think a program just doing
malloc free in an endless loop will, I've got real world examples too.

on 7.1 and rawhide versions of Red Hat Linux, I cannot reproduce this with a 
simple test case:

#include <stdio.h>
#include <string.h>
#include <malloc.h>
 
int main(int argc, char *argv[])
{
  char *s;
 
  s = (char *) malloc((sizeof(char)) * 80);
  strcpy(s, "Hello World!");
  printf("%s\n", s);
  free(s);
}

If you have a similarly simple case that you can show me that reproduces the 
problem, please include it here and reopen the bug.

As indicated by hans' comments, your simple test case will not reproduce this
problem because your simple test case does not exhaust the virtual address
space.  You need to allocate and free over and over in an endless loop to
duplicate this problem.

The only way to fix it is to each ElectricFence to reuse old pages when the
virtual addres space is exhauseted, in first-in-first-out order.  I.e.,
EF_PROTECT_FREE can't protect pages *forever*, because then eventually the
virual address space will be exhausted.  When that happens, ElectricFence needs
to start reusing old pages.

This means that it needs to keep a queue of pages that it has protected or know
how to search through the address space for them.

At this stage of ElectricFences evolution, I would definitely consider that 
more of an enhancement request.