Bug 449381 - System hangs when using /proc/sys/vm/drop_caches under heavy load on large system.
System hangs when using /proc/sys/vm/drop_caches under heavy load on large sy...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.7
All Linux
high Severity high
: rc
: ---
Assigned To: Larry Woodman
Martin Jenner
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-06-02 11:04 EDT by Larry Woodman
Modified: 2009-02-12 13:56 EST (History)
2 users (show)

See Also:
Fixed In Version: RHSA-2008-0665
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-07-24 15:30:01 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch that fixes this problem. (2.04 KB, patch)
2008-06-02 11:04 EDT, Larry Woodman
no flags Details | Diff

  None (edit)
Description Larry Woodman 2008-06-02 11:04:35 EDT
Description of problem:

Eliminate hang when using /proc/sys/vm/drop_caches under heavy load on large system.

Version-Release number of selected component (if applicable):

REHL4-U6

How reproducible:

Frequest but requires large system(~64GB) and multiple CPUs(~8) running
several(more then CPU count).

Steps to Reproduce:
1. Start several file system exercisers that create and/or read large enoug
files to exhaust memory in the pagecache.
2. "echo 3 > /proc/sys/vm /drop_caches" until system hangs
3. Capture AltSysrq-W ad verify all CPUs are stuck on the inode_lock.
  
Actual results:

System Hang.

Expected results:

Pagecache memory is freed without system hanging.

Additional info:

Back in RHEL4-U6 we backported the /proc/sys/vm/drop_caches
functionality from upstream to RHEL4.  Recently I encountered hang in
this code while creating 256GB files on a 64GB 4-core system and
dropping the pagecache at the same time.  The cause of the hang is
invalidate_list() calls invalidate_inode_pages() which calls
invalidate_mapping_pages() with the inode_lock held.  Since
invalidate_mapping_pages() calls cond_resched(), every CPU can try to
acquire the inode_lock if the time quantum of the process writing
to /proc/sys/vm/drop_caches expires.  So far I have only been able to
reproduce this problem when writing multiple huge files on every CPU and
"echo 3 > /proc/sys/vm/drop_caches" from a shell, but it can happen
randomly.

The attached patch fixes this problem by creating and calling a new
function invalidate_all_mapping_pages() which does not reschedule.  I
could not backport the upstream solution to RHEL4 because
invalidate_mapping_pages() is exported and the fix would break the kABI
but the fix is basically the same logic that is upstream.

The original BZ is 205722.
Comment 1 Larry Woodman 2008-06-02 11:04:35 EDT
Created attachment 307376 [details]
Patch that fixes this problem.
Comment 2 Linda Wang 2008-06-09 18:02:34 EDT
How big of the impact to the customer base w/o the patch?
Comment 5 Vivek Goyal 2008-06-12 14:32:13 EDT
Committed in 73.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/
Comment 8 errata-xmlrpc 2008-07-24 15:30:01 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2008-0665.html

Note You need to log in before you can comment on or make changes to this bug.