Bug 98333 - RFE: The Kernel hangs up when using HyperThreading
Summary: RFE: The Kernel hangs up when using HyperThreading
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Enterprise Linux 2.1
Classification: Red Hat
Component: kernel
Version: 2.1
Hardware: i686
OS: Linux
high
medium
Target Milestone: ---
Assignee: Larry Woodman
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-07-01 11:15 UTC by Yoshihide Sonoda
Modified: 2007-11-30 22:06 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2005-09-30 18:30:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Yoshihide Sonoda 2003-07-01 11:15:55 UTC
Description of problem:

The Kernel hangs up when using Hyper-Threading.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Server 2.1AS
 kernel 2.4.9-e.3 ~ e.25

How reproducible:

Steps to Reproduce:
1. Enable HyperThreading
2. Heavy disk access and using memory.

Actual results:
The system hangs up.
However, when several hours pass, it may begin to move.

Additional info:

The cause of this problem is in exclusion control of the memory manager and
the file system. These have very long spin locks.
When Hyper-Threading is used, two logical CPUs can be used on one physical CPU.
When the process holding a very long spin lock and the process which waits for
the lock operate by a logical CPU on the same physical CPU simultaneously,
in order that the process which performs waiting for a lock may use all the
execution units of the CPU, it will become impossible then, for a logical CPU
of another side to almost advance processing.
Since the execution of a process which holds a lock according to the process
which waits for a lock is barred, a deadlock occurs and the system will hang
up.

We investigated inside the kernel what has happened using 
LKST(http://lkst.sourceforge.net/), when the system hung-up. 

The test environment is as follows.
CPU: Pentium 4 Xeon 2.0GHz * 2 (with Hyper-Threading)
Memory: 2GB
OS: Red Hat Enterprise Server 2.1AS
Kernel: 2.4.9-e.16smp

When it hangs-up, the process which each CPU was executing is as follows.
CPU0: kswapd
CPU1: hadkwd
CPU2: hadaemon
CPU3: uagentd
(The hadkwd, hadaemon are 3rd party's cluster software.
The uagentd is 3rd party's backup software.)

The stack trace of each process is as follows.

kswapd:
  do_try_to_free_pages()
    shrink_icache_memory()
      prune_icache()
       ->(spinlock: inode_lock)
       invalidate_inode_pages()
       ->(spinlock: pagemap_lru_lock)

hadkwd:
  path_walk()
    real_lookup()
      (Ext3fs related function called)
        -> (spinlock: inode_lock)

hadaemon:
  generic_file_write()
   -> (spinlock: inode_lock)

uagentd:
  do_generic_file_read()
    do_generic_readahead()
      drop_behind()
        -> (spinlock: pagemap_lru_lock)

All processes are waited for kswapd.
Since Hyper-Threading is used, the spin lock executed by one logical
CPU bars execution of another logical CPU. Thereby, since a lock cannot
be canceled, the system stops.

Workaround:
 Disable Hyper-Threading

Fundamental measure:
 Improve the memory manager and file system. (without spin lock)

Comment 3 Rik van Riel 2003-12-25 15:36:12 UTC
IIRC the spinlock waiting code in later 2.4 kernels got an "I'm idle,
yield to the other logical CPU" opcode in it. That is, one of the
various NOP style opcodes gets interpreted in a magical way by HT CPUs.

I could imagine that extra opcode not yet being there in AS2.1. The
bug is certainly subtle enough that we would forget about backporting
that from later kernels...

Comment 4 Bernd Schmidt 2004-01-09 15:24:01 UTC
There's a rep nop in our spin_lock function, but later 2.4 kernels
have added another one in irq_enter in hardirq.h.  Do you think this
could be the problem?

Comment 5 Larry Woodman 2005-09-30 18:30:33 UTC
We cant change the scheduler in AS2.1 at theis point in the lifecycle of the
AS2.1 kernel.  This problem was fixed in RHEL3, please upgrade to that kernel
version.

Larry Woodman



Note You need to log in before you can comment on or make changes to this bug.