Description of problem: The Kernel hangs up when using Hyper-Threading. Version-Release number of selected component (if applicable): Red Hat Enterprise Server 2.1AS kernel 2.4.9-e.3 ~ e.25 How reproducible: Steps to Reproduce: 1. Enable HyperThreading 2. Heavy disk access and using memory. Actual results: The system hangs up. However, when several hours pass, it may begin to move. Additional info: The cause of this problem is in exclusion control of the memory manager and the file system. These have very long spin locks. When Hyper-Threading is used, two logical CPUs can be used on one physical CPU. When the process holding a very long spin lock and the process which waits for the lock operate by a logical CPU on the same physical CPU simultaneously, in order that the process which performs waiting for a lock may use all the execution units of the CPU, it will become impossible then, for a logical CPU of another side to almost advance processing. Since the execution of a process which holds a lock according to the process which waits for a lock is barred, a deadlock occurs and the system will hang up. We investigated inside the kernel what has happened using LKST(http://lkst.sourceforge.net/), when the system hung-up. The test environment is as follows. CPU: Pentium 4 Xeon 2.0GHz * 2 (with Hyper-Threading) Memory: 2GB OS: Red Hat Enterprise Server 2.1AS Kernel: 2.4.9-e.16smp When it hangs-up, the process which each CPU was executing is as follows. CPU0: kswapd CPU1: hadkwd CPU2: hadaemon CPU3: uagentd (The hadkwd, hadaemon are 3rd party's cluster software. The uagentd is 3rd party's backup software.) The stack trace of each process is as follows. kswapd: do_try_to_free_pages() shrink_icache_memory() prune_icache() ->(spinlock: inode_lock) invalidate_inode_pages() ->(spinlock: pagemap_lru_lock) hadkwd: path_walk() real_lookup() (Ext3fs related function called) -> (spinlock: inode_lock) hadaemon: generic_file_write() -> (spinlock: inode_lock) uagentd: do_generic_file_read() do_generic_readahead() drop_behind() -> (spinlock: pagemap_lru_lock) All processes are waited for kswapd. Since Hyper-Threading is used, the spin lock executed by one logical CPU bars execution of another logical CPU. Thereby, since a lock cannot be canceled, the system stops. Workaround: Disable Hyper-Threading Fundamental measure: Improve the memory manager and file system. (without spin lock)
IIRC the spinlock waiting code in later 2.4 kernels got an "I'm idle, yield to the other logical CPU" opcode in it. That is, one of the various NOP style opcodes gets interpreted in a magical way by HT CPUs. I could imagine that extra opcode not yet being there in AS2.1. The bug is certainly subtle enough that we would forget about backporting that from later kernels...
There's a rep nop in our spin_lock function, but later 2.4 kernels have added another one in irq_enter in hardirq.h. Do you think this could be the problem?
We cant change the scheduler in AS2.1 at theis point in the lifecycle of the AS2.1 kernel. This problem was fixed in RHEL3, please upgrade to that kernel version. Larry Woodman