Description of problem:
Setting user limits in /etc/security/limits.con are ignored by kernel

Version-Release number of selected component (if applicable):
[ben@benvmpcm12 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.3 (Tikanga)

[ben@benvmpcm12 ~]$ cat /proc/version
Linux version 2.6.18-128.el5 (mockbuild.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Dec 17 11:41:38 EST 2008
How reproducible:
Easy to reproduce

Steps to Reproduce:
1.Set 'memlock' and/or 'rss' limits in limits.conf for user <user>
2.ssh to that host as <user>
3.run an application to exceede memory limit set in step 1.
  
Actual results:
Application runs

Expected results:
Application should fail.

Additional info:
Please see this thread which points to kernel bug:

http://lists.debian.org/debian-user/2001/03/msg04306.html

This is simple C program I used to test this behavior:
======================================================
#include<stdio.h>
#include<unistd.h>
#include<stdlib.h>
main(){

    /* Allocate space for an array with ten elements of type int. */
    char *ptr;
    ptr =  malloc(500000000);
    if (ptr == NULL) {
    /* Memory could not be allocated, the program should handle the error here as appropriate. */
    printf("Memory could not be allocated\n");
    } else {
        /* Allocation succeeded.  Do something.  */
        printf("Memory allocation successful\n");
        sleep(20);
        free(ptr);  /* We are done with the int objects, and free the associated pointer. */
        ptr = NULL; /* The pointer must not be used again, unless re-assigned to using malloc again. */
    }
}
====================================================================