Red Hat Bugzilla – Bug 98333
RFE: The Kernel hangs up when using HyperThreading
Last modified: 2007-11-30 17:06:53 EST
Description of problem:
The Kernel hangs up when using Hyper-Threading.
Version-Release number of selected component (if applicable):
Red Hat Enterprise Server 2.1AS
kernel 2.4.9-e.3 ~ e.25
Steps to Reproduce:
1. Enable HyperThreading
2. Heavy disk access and using memory.
The system hangs up.
However, when several hours pass, it may begin to move.
The cause of this problem is in exclusion control of the memory manager and
the file system. These have very long spin locks.
When Hyper-Threading is used, two logical CPUs can be used on one physical CPU.
When the process holding a very long spin lock and the process which waits for
the lock operate by a logical CPU on the same physical CPU simultaneously,
in order that the process which performs waiting for a lock may use all the
execution units of the CPU, it will become impossible then, for a logical CPU
of another side to almost advance processing.
Since the execution of a process which holds a lock according to the process
which waits for a lock is barred, a deadlock occurs and the system will hang
We investigated inside the kernel what has happened using
LKST(http://lkst.sourceforge.net/), when the system hung-up.
The test environment is as follows.
CPU: Pentium 4 Xeon 2.0GHz * 2 (with Hyper-Threading)
OS: Red Hat Enterprise Server 2.1AS
When it hangs-up, the process which each CPU was executing is as follows.
(The hadkwd, hadaemon are 3rd party's cluster software.
The uagentd is 3rd party's backup software.)
The stack trace of each process is as follows.
(Ext3fs related function called)
-> (spinlock: inode_lock)
-> (spinlock: inode_lock)
-> (spinlock: pagemap_lru_lock)
All processes are waited for kswapd.
Since Hyper-Threading is used, the spin lock executed by one logical
CPU bars execution of another logical CPU. Thereby, since a lock cannot
be canceled, the system stops.
Improve the memory manager and file system. (without spin lock)
IIRC the spinlock waiting code in later 2.4 kernels got an "I'm idle,
yield to the other logical CPU" opcode in it. That is, one of the
various NOP style opcodes gets interpreted in a magical way by HT CPUs.
I could imagine that extra opcode not yet being there in AS2.1. The
bug is certainly subtle enough that we would forget about backporting
that from later kernels...
There's a rep nop in our spin_lock function, but later 2.4 kernels
have added another one in irq_enter in hardirq.h. Do you think this
could be the problem?
We cant change the scheduler in AS2.1 at theis point in the lifecycle of the
AS2.1 kernel. This problem was fixed in RHEL3, please upgrade to that kernel