Bug 429103 - Allocations on resume path can cause deadlock due to attempting to swap
Allocations on resume path can cause deadlock due to attempting to swap
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel-xen (Show other bugs)
4.6
All Linux
low Severity low
: rc
: ---
Assigned To: Don Dutile
Martin Jenner
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-01-17 05:58 EST by Ian Campbell
Modified: 2008-07-24 15:24 EDT (History)
1 user (show)

See Also:
Fixed In Version: RHSA-2008-0665
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-07-24 15:24:48 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
xen-unstable.hg 12562:c242b6d6a64a backported to 2.6.9-67.0.1.EL (8.76 KB, patch)
2008-01-17 06:03 EST, Ian Campbell
no flags Details | Diff
linux-2.6.18-xen.hg 377:e8b49cfbdac0 backported to 2.6.9-67.0.1.EL (4.21 KB, patch)
2008-01-17 06:04 EST, Ian Campbell
no flags Details | Diff
xen-unstable.hg 13222:98dadb3df5ca backported to 2.6.9-67.0.1.EL (892 bytes, patch)
2008-01-22 04:11 EST, Ian Campbell
no flags Details | Diff
Patch for fix to 4.7 if applied before 5-patch set to 437423 (5.81 KB, text/x-patch)
2008-03-20 14:46 EDT, Don Dutile
no flags Details
Patch for fix to 4.7 if applied *after* bz437423's 5-patch post (5.76 KB, text/plain)
2008-03-20 14:47 EDT, Don Dutile
no flags Details
linux-2.6.18-xen.hg 497:fdb998e79aba backported to 2.6.9-67.0.7.EL (applied with no changes) (1.03 KB, patch)
2008-03-28 07:21 EDT, Ian Campbell
no flags Details | Diff
linux-2.6.18-xen.hg 498:0637d22ed554 backported to 2.6.9-67.0.7.EL (simple s/mutex_unlock/up/ required) (2.30 KB, patch)
2008-03-28 07:23 EDT, Ian Campbell
no flags Details | Diff
Updated patch including comments 13 &14 (7.80 KB, application/octet-stream)
2008-05-23 16:01 EDT, Don Dutile
no flags Details

  None (edit)
Description Ian Campbell 2008-01-17 05:58:25 EST
+++ This bug was initially created as a clone of Bug #429102 +++

Allocations made on the resume path, particularly in the blkfront reattach path,
can cause swap activity which cannot be performed because we are in the middle
of reattaching the swap disk. The solution is to use __GFP_HIGH on such
allocations which will use the emergency pool if necessary.

This was fixed upstream by linux-2.6.18-xen.hg 377:e8b49cfbdac0
http://hg.uk.xensource.com/linux-2.6.18-xen.hg?cs=e8b49cfbdac0

This issue effects RHEL5u1 (2.6.18-53.1.4.el) and RHEL4u6 (2.6.9-67.0.1.EL). I
will clone this issue into a RHEL4 issue as well.

To reproduce run a guest workload with large memory consumption (such as a
userspace memtest type application). The issue is seen after a few iterations of
save and restore, typically somrthing less than a dozen.

-- Additional comment from ijc@hellion.org.uk on 2008-01-17 05:57 EST --
Created an attachment (id=291977)
linux-2.6.18-xen.hg 377:e8b49cfbdac0 backported to 2.6.18-53.1.4.el
Comment 1 Ian Campbell 2008-01-17 06:01:33 EST
Backporting linux-2.6.18-xen.hg 377:e8b49cfbdac0 was simplified by also
backporting xen-unstable xen-unstable.hg 12562:c242b6d6a64a
http://xenbits.xensource.com/xen-unstable.hg?rev/c242b6d6a64a

Also, hg.uk.xensource.com is an internal address, external address is
http://xenbits.xensource.com/linux-2.6.18-xen.hg?cs=e8b49cfbdac0.
Comment 2 Ian Campbell 2008-01-17 06:03:31 EST
Created attachment 291978 [details]
xen-unstable.hg 12562:c242b6d6a64a backported to 2.6.9-67.0.1.EL
Comment 3 Ian Campbell 2008-01-17 06:04:11 EST
Created attachment 291979 [details]
linux-2.6.18-xen.hg 377:e8b49cfbdac0 backported to 2.6.9-67.0.1.EL
Comment 4 Don Dutile 2008-01-21 15:21:15 EST
Are the changes to kasprint() necessary for a bug fix?
In other words, can't the changes to the kmalloc()'s  & get_zeroed_page() from
GFP_KERNEL to GFP_KERNEL|__GFP_HIGH be a sufficient fix ?
Comment 5 Ian Campbell 2008-01-21 15:41:09 EST
The changes to kasprintf() are so we can change some but not all of the callers
to use |__GFP_HIGH. Alternatively you could change the allocation inside the
existing kasprintf() function to be |__GFP_HIGH, but since kasprintf() is called
on paths other than the one which is of interest here you'd be changing
behaviour for them too -- which is probably harmless but doesn't match what the
xen.org kernel does.

kasprintf only exists in this kernel to support the Xen stuff (only user) and
the patches change it to match upstream behaviour, I think it's pretty low risk
myself.
Comment 6 Chris Wright 2008-01-21 18:58:31 EST
The changes to lib/vsprintf.c are touching core code and introducing a bug.
 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0a6047eef1c465c38aacfbdab193161b3f0cd144

This will be triggered if the buffer is too short for the payload (and caused
oopsen for various sysfs files that truncated based on PAGE_SIZE, for example).
Comment 7 Ian Campbell 2008-01-22 04:11:58 EST
Created attachment 292483 [details]
xen-unstable.hg 13222:98dadb3df5ca backported to 2.6.9-67.0.1.EL

Sorry, I somehow missed xen-unstable.hg 13222:98dadb3df5ca
(http://xenbits.xensource.com/xen-unstable.hg?rev/98dadb3df5ca) which is the
same fix. Hardly worth calling it "backported" but here it is.
Comment 8 Don Dutile 2008-03-20 14:46:05 EDT
Created attachment 298729 [details]
Patch for fix to 4.7 if applied before 5-patch set to 437423

see next comment for patch if needed after the 5-patch post for 437423
Comment 9 Don Dutile 2008-03-20 14:47:15 EDT
Created attachment 298730 [details]
Patch for fix to 4.7 if applied *after* bz437423's 5-patch post

Patch if applied after 5-patch posting for bz437423
Comment 10 Bill Burns 2008-03-27 14:11:22 EDT
Set flags for Don.
Comment 12 Ian Campbell 2008-03-28 07:19:26 EDT
I'm afraid I've found a couple more instances of this issue:
http://xenbits.xensource.com/staging/linux-2.6.18-xen.hg?rev/fdb998e79aba
http://xenbits.xensource.com/staging/linux-2.6.18-xen.hg?rev/0637d22ed554

If "Status: POST" means you would like a new ticket I can do that. I'll attach
the backported patches in a moment.

The thread related one is a bit subtle: the xenbus_watch thread blocks with
xenbus_mutex held in kthread_create waiting for the completion to say the thread
has been spawned successfully. The thread is stuck waiting on IO due to an
attempt to swap while allocating memory in copy_process. This causes the suspend
process to block waiting for the xenbus_mutex and therefore the swap device
never gets attached.

These traces are from a linux-2.6.18-xen kernel but the code paths are the same.

suspend       D C02DF2F5     0 14792      1         14772 14790 (L-TLB)
       c20b5ea8 00000246 00000002 c02df2f5 00000008 c11ba000 00000000 c038aa00 
       c038aa00 00000000 c3e57660 00000000 00000000 00000009 c3e57550 89c61ba6 
       00023227 000004b3 c3e57660 c1101960 00000002 89b240b0 89c616f3 00023227 
Call Trace:
 [<c02df2f5>] __mutex_lock_slowpath+0xc5/0x2f0
 [<c02df528>] mutex_lock+0x8/0x10
 [<c025ef0f>] unregister_xenbus_watch+0x12f/0x1a0
 [<c025f82b>] free_otherend_watch+0x1b/0x40
 [<c025f869>] talk_to_otherend+0x19/0x40
 [<c02608aa>] resume_dev+0x2a/0xd0
 [<c0252d54>] bus_for_each_dev+0x54/0x80
 [<c02609e4>] xenbus_resume+0x44/0x50
 [<c025aa3a>] __xen_suspend+0x9a/0x110
 [<c025a1a8>] xen_suspend+0x68/0xd0
 [<c0102b55>] kernel_thread_helper+0x5/0x10

blocked waiting to lock xenwatch_mutex in unregister_xenbus_watch:
	/* Flush any currently-executing callback, unless we are it. :-) */
	if (current->pid != xenwatch_pid) {
		mutex_lock(&xenwatch_mutex);
		mutex_unlock(&xenwatch_mutex);
	}

the current holder is the xenwatch thread:

xenwatch      D C02DE102     0     9      7            10       (L-TLB)
       c11bbee8 00000246 00000002 c02de102 89c366df 00023227 c53e7200 c038aa00 
       c038aa00 00023227 89c54017 00023227 00000000 0000000a c11b6a70 89c5416d 
       00023227 00000f8d c11b6b80 c1101960 0000008f 00000000 89c531e0 00023227 
Call Trace:
 [<c02de102>] wait_for_completion+0x82/0xf0
 [<c0136c0c>] kthread_create+0x7c/0xd0
 [<c025f33b>] xenwatch_thread+0x10b/0x140
 [<c0136b86>] kthread+0x106/0x110
 [<c0102b55>] kernel_thread_helper+0x5/0x10

and the thread itself:

kthread       D C02DE736     0     7      1     9     758     6 (L-TLB)
       c11a9a60 00000246 00000002 c02de736 00000000 c11a9a08 00000003 c038aa00 
       c038aa00 c11a9ff8 c11bdf80 00000003 00000000 00000009 c1165550 89c616f3 
       00023227 0000d586 c1165660 c1101960 c01058b1 00000003 89c5416d 00023227 
Call Trace:
 [<c02de736>] io_schedule+0x26/0x30
 [<c02226aa>] get_request_wait+0xca/0x110
 [<c0223717>] __make_request+0x87/0x3b0
 [<c022141a>] generic_make_request+0xea/0x1b0
 [<c0223c8b>] submit_bio+0x6b/0x120
 [<c015f9ba>] swap_writepage+0x9a/0xc0
 [<c014f67a>] shrink_zone+0xefa/0x1080
 [<c014ff4a>] try_to_free_pages+0xca/0x1f0
 [<c014ad78>] __alloc_pages+0x178/0x2f0
 [<c01671fa>] cache_alloc_refill+0x2ea/0x590
 [<c0166eff>] kmem_cache_alloc+0x9f/0xb0
 [<c011e8c7>] copy_process+0x97/0x1240
 [<c011fd5b>] do_fork+0x6b/0x1c0
 [<c0102fdb>] kernel_thread+0x8b/0xa0
 [<c0136a27>] keventd_create_kthread+0x27/0x80
 [<c0132be5>] run_workqueue+0x75/0xf0
 [<c0133918>] worker_thread+0x138/0x160
 [<c0136b86>] kthread+0x106/0x110
 [<c0102b55>] kernel_thread_helper+0x5/0x10
Comment 13 Ian Campbell 2008-03-28 07:21:58 EDT
Created attachment 299451 [details]
linux-2.6.18-xen.hg 497:fdb998e79aba backported to 2.6.9-67.0.7.EL (applied with no changes)
Comment 14 Ian Campbell 2008-03-28 07:23:09 EDT
Created attachment 299452 [details]
linux-2.6.18-xen.hg 498:0637d22ed554 backported to 2.6.9-67.0.7.EL (simple s/mutex_unlock/up/ required)
Comment 17 Don Dutile 2008-05-23 16:01:26 EDT
Created attachment 306545 [details]
Updated patch including comments 13 &14
Comment 18 Vivek Goyal 2008-05-29 16:50:44 EDT
Committed in 71.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/
Comment 20 Ian Campbell 2008-06-10 06:33:57 EDT
The fixes given above worked for me in practice however according to upstream
the correct fix is to use GFP_NOIO:
http://marc.info/?l=linux-kernel&m=121222807617695&w=2

This has been applied to the Xen kernel at:
http://xenbits.xensource.com/staging/linux-2.6.18-xen.hg?rev/5db911a71eac
Comment 22 errata-xmlrpc 2008-07-24 15:24:48 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2008-0665.html

Note You need to log in before you can comment on or make changes to this bug.