Bug 450953 - el4u6 xenU guest kernel lockup due to mm_unpinned_lock and runqueue spinlock deadlock
el4u6 xenU guest kernel lockup due to mm_unpinned_lock and runqueue spinlock ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel-xen (Show other bugs)
4.6
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Chris Lalancette
Martin Jenner
:
Depends On:
Blocks: 461304 470304
  Show dependency treegraph
 
Reported: 2008-06-11 17:13 EDT by Herbert van den Bergh
Modified: 2010-10-22 21:51 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-05-18 15:15:53 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
fix for mm_unpinned_lock / runq deadlock (2.29 KB, patch)
2008-06-11 17:13 EDT, Herbert van den Bergh
no flags Details | Diff
Backport of upstream xen-unstable c/s 10343 and 10995 (4.29 KB, patch)
2008-11-22 04:31 EST, Chris Lalancette
no flags Details | Diff

  None (edit)
Description Herbert van den Bergh 2008-06-11 17:13:09 EDT
Description of problem:

After running an arbitrary workload involving network traffic for some time (1-2
days), a xen guest running the 2.6.9-67 x86_64 xenU kernel locks up with both
vcpu's spinning at 100%.  

Version-Release number of selected component (if applicable):
kernel-2.6.9-67

How reproducible:
Reproduces after running a test workload (involving network traffic) for 1-2
days, but not consistently.

Steps to Reproduce:
1. Run a test workload involving network traffic
2. monitor cpu usage of guest
3. wait... until guest cpu usage goes to 100%
  
Actual results:
Guest kernel spins on both vcpu's

Expected results:
Guest kernel doesn't spin

Additional info:
The problem is due to a race between the scheduler and network interrupts.  On
one vcpu, the scheduler takes the runqueue spinlock of the other vcpu to
schedule a process, and attempts to lock mm_unpinned_lock.  On the other vcpu,
another process is holding mm_unpinned_lock (because it is starting or exiting),
and is interrupted by a network interrupt.  The network interrupt handler
attempts to wake up the same process that the first vcpu is trying to schedule,
and will try to get the runqueue spinlock that the first vcpu is already holding.

I was not able to obtain a full kernel stack from the interrupt, but do have
kernel stacks of the tasks on the vcpu's, if needed.
Comment 1 Herbert van den Bergh 2008-06-11 17:13:09 EDT
Created attachment 309000 [details]
fix for mm_unpinned_lock / runq deadlock
Comment 2 Rik van Riel 2008-07-30 11:17:56 EDT
The patch looks good to me.
Comment 3 RHEL Product and Program Management 2008-09-03 09:17:16 EDT
Updating PM score.
Comment 4 RHEL Product and Program Management 2008-09-17 14:52:15 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 10 Chris Lalancette 2008-11-22 04:29:09 EST
OK.  I posted this patch to upstream Xen, and they pointed out that not only does 2.6.18 probably not have this issue, but also that the patch isn't entirely sufficient, since you might still deadlock with the mm->page_table_lock.  Upstream 2.6.18 does this a different way (which the RHEL-5 kernel already has); I've backported the changes, and built a test kernel available at http://people.redhat.com/clalance/bz450953/.  Can the reporter please download the test kernel, try it out, and report back test results?

Thank you,
Chris Lalancette
Comment 11 Chris Lalancette 2008-11-22 04:31:04 EST
Created attachment 324399 [details]
Backport of upstream xen-unstable c/s 10343 and 10995

This is a backport of xen-unstable c/s 10343 and 10995, that I believe will fix the issue.  Instead of changing the locking, this actually defers the mm_pin time until activate_mm, which means that we should avoid the deadlock in the first place.
Comment 12 Rik van Riel 2008-11-22 10:32:29 EST
Chris, the upstream patches are indeed better.  I'll go NACK Larry's patches :)
Comment 14 Vivek Goyal 2008-11-27 14:38:23 EST
Committed in 78.20.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/
Comment 17 Chris Ward 2009-05-05 09:56:31 EDT
Any updates here? Has this issue been resolved in the RHEL 4.8 Beta? later kernel?
Comment 21 errata-xmlrpc 2009-05-18 15:15:53 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1024.html

Note You need to log in before you can comment on or make changes to this bug.