Bug 112764 - RHEL3 kernel not preventing or recovering from fork bomb when ulimit used
Summary: RHEL3 kernel not preventing or recovering from fork bomb when ulimit used
Alias: None
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel   
(Show other bugs)
Version: 3.0
Hardware: i686
OS: Linux
Target Milestone: ---
Assignee: Dave Anderson
QA Contact: Brian Brock
Depends On:
TreeView+ depends on / blocked
Reported: 2003-12-30 21:51 UTC by Jason Tackaberry
Modified: 2007-11-30 22:06 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2004-05-12 01:08:09 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2004:188 normal SHIPPED_LIVE Important: Updated kernel packages available for Red Hat Enterprise Linux 3 Update 2 2004-05-11 04:00:00 UTC

Description Jason Tackaberry 2003-12-30 21:51:57 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6b)
Gecko/20031206 Firebird/0.7+

Description of problem:
Using the stock RHEL3 kernel, I'm not able to prevent a fork bomb from
severely affecting the system.  In the best case, I will get out of
memory errors ("fork: Cannot allocate memory"), and in the worst case,
OOM killer will start slaughtering necessary services.

I have tried both ulimit (-u 50 -v 512000) and setting these limits
via /etc/security/limits.conf:

        @students       hard    nproc           10
        @students       hard    cpu             1
        @students       hard    as              10240

A simple while(1)fork(); causes all other processes to hang, until OOM
killer starts terminating processes.  This usually isn't enough for
the system to recover -- I can ctrl-c in the terminal the fork bomb is
running in and sometimes the system will become responsive again,
although kswapd starts using 2-3% CPU indefinitely even though there
is 0 swapfile usage (15 minutes was the longest I let it run before
rebooting).  Other times the system doesn't ever seem to fully
recover, and spawning new processes is very slow.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Do ulimit -u 50
2. Compile void main(){ while(1) fork(); }
3. Run ./a.out
4. Try running something in another window as root -- ps, say.

Actual Results:  With either "ulimit -u 50" or setting a hard nproc
limit via limits.conf, doing ps (for example) in another terminal
either hangs indefinitely or yields "fork: Cannot allocate memory"
errors.  The OOM killer quickly steps in.  Leaving top open in another
terminal, I can see that the number of a.out processes is in fact
contained to 50 (or so) and memory usage isn't abnormal. 
Nevertheless, the system is unresponsive.

Expected Results:  I compiled and booted linus kernel 2.4.22 and it
behaves properly: there are no problems executing processes as other
users and the system is perfectly responsive.  As with RHEL3 kernel,
the number of processes for the user is capped at 50, except that
unlike RHEL3 kernel, the system remains stable.

Additional info:

The system is a PowerEdge 4600 with 2G RAM, PERC 3Di in a RAID5
configuration.  It's a dual Xeon 2.6G with hyperthreading enabled. 
Here is output from free(1) prior to running the fork bomb:

             total       used       free     shared    buffers     cached
Mem:       2061552     151448    1910104          0      10376      68660
-/+ buffers/cache:      72412    1989140
Swap:      2096472          0    2096472

If I put a short sleep after the fork, usleep(10) say, the system
behaves correctly and contains the fork bomb.

Comment 1 Jason Tackaberry 2003-12-30 21:54:47 UTC
RHEL version should be 3 -- haven't tested on 2.1

Comment 2 Dave Anderson 2004-01-12 15:09:44 UTC
Thank you for this report.

The bug has been identified and will be fixed/included in the
RHEL3 U2 release timeframe.

The bug does not exist in the AS2.1 kernel or upstream.

Comment 3 John Flanagan 2004-05-12 01:08:09 UTC
An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.


Comment 4 Anand Jahagirdar 2007-05-17 03:31:41 UTC
I found one surprizing thing on my machine having Fedora Core 6 as 
distribution and Linux Kernel 2.6.18. I have set ulimit -u as 8000. I have 
written one simple Fork Bombing Program while(1) fork();. this fork bombing 
attack killed the box and machine needed a reboot. will any body please tell 
me why this Fork bombing attack killed the box even after setting ulimit as 

Note You need to log in before you can comment on or make changes to this bug.