Bug 219563 - network-attaching and detaching too fast from xenbus will crash domU
network-attaching and detaching too fast from xenbus will crash domU
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel-xen (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: Glauber Costa
Depends On:
  Show dependency treegraph
Reported: 2006-12-13 17:59 EST by Glauber Costa
Modified: 2007-11-30 17:07 EST (History)
3 users (show)

See Also:
Fixed In Version: RHBA-2007-0959
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-11-07 14:17:05 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Upstream proposal (1011 bytes, patch)
2006-12-14 11:50 EST, Glauber Costa
no flags Details | Diff
upstream commit (1.32 KB, patch)
2006-12-14 14:38 EST, Glauber Costa
no flags Details | Diff

  None (edit)
Description Glauber Costa 2006-12-13 17:59:43 EST
Description of problem:

network-attaching and detaching too fast will lead to domU crashes. The
underlying network device hits a BUG.

How reproducible:

Steps to Reproduce:
1. in dom0, 
   for i in $(seq 1000); 
     xm network-attach <domid>;  
     xm network-detach <domid> $i;
2. wait some iterations
3. watch domU crash
Actual results:

----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at net/core/dev.c:3341
invalid opcode: 0000 [1] SMP
last sysfs file: /class/net/eth2/address
Modules linked in: xennet ipv6 dm_multipath parport_pc lp parport pcspkr
dm_snapshot dm_zero dm_mirror dm_mod xenblk ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 9, comm: xenwatch Not tainted 2.6.18-1.2857.8.2.fc6_glommerxen #1
RIP: e030:[<ffffffff803f6234>]  [<ffffffff803f6234>] unregister_netdevice+0x6e/0x215
RSP: e02b:ffff8800013f1de0  EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffff880001a28000 RCX: ffff8800013f1c00
RDX: 0000000000000000 RSI: 0000000000000056 RDI: ffffffff80521de0
RBP: ffffffff80395148 R08: 000000000000000a R09: 0000000000000005
R10: ffffffff8046cdfc R11: ffffffff88174aa7 R12: ffff880000035c80
R13: 0000000000000000 R14: ffff880000035c70 R15: ffff880001a28000
FS:  00002aaaaaabddb0(0000) GS:ffffffff80593000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process xenwatch (pid: 9, threadinfo ffff8800013f0000, task ffff8800013ec7d0)
Stack:  ffff880000035c70  ffff880001a28000  ffffffff80395148  ffffffff803f63ec
 ffff88000212c540  ffffffff881752d7  ffff880002081400  ffff880001a28580
 0000000000000000  ffffffff8020b44c
Call Trace:
 [<ffffffff80395148>] xenwatch_thread+0x0/0x13e
 [<ffffffff803f63ec>] unregister_netdev+0x11/0x17
 [<ffffffff881752d7>] :xennet:backend_changed+0x830/0x865
 [<ffffffff8020b44c>] kfree+0x15/0xbc
 [<ffffffff80393867>] xenbus_read_driver_state+0x26/0x36
 [<ffffffff80395148>] xenwatch_thread+0x0/0x13e
 [<ffffffff80295d9f>] keventd_create_kthread+0x0/0x66
 [<ffffffff803945a1>] xenwatch_handle_callback+0x15/0x48
 [<ffffffff8039526d>] xenwatch_thread+0x125/0x13e
 [<ffffffff80295f53>] autoremove_wake_function+0x0/0x2e
 [<ffffffff80295d9f>] keventd_create_kthread+0x0/0x66
 [<ffffffff8023321d>] kthread+0xfe/0x132
 [<ffffffff8025dec8>] child_rip+0xa/0x12
 [<ffffffff80295d9f>] keventd_create_kthread+0x0/0x66
 [<ffffffff8025fbf3>] thread_return+0x0/0xfb
 [<ffffffff8023311f>] kthread+0x0/0x132
 [<ffffffff8025debe>] child_rip+0x0/0x12

Code: 0f 0b 68 c5 0c 49 80 c2 0d 0d f6 83 98 00 00 00 01 74 08 48
RIP  [<ffffffff803f6234>] unregister_netdevice+0x6e/0x215
 RSP <ffff8800013f1de0>

Expected results:

attach, detach, attach, detach...
Comment 1 Glauber Costa 2006-12-13 18:02:33 EST
After a lof of research, the reason for that is xenbus state changes being
delivered twice to frontend. He sees XenbusStateClosing twice, disconnect itself
twice, etc. In the second time, it's internal state is not valid anymore, and
the BUG() is hit. I have not yet determined the reason behind the double delivery.
Comment 2 Jay Turner 2006-12-13 21:47:02 EST
This seems a little corner-case . . . in addition, without some leads to the
solution, I recommend this be deferred to 5.1.
Comment 3 Glauber Costa 2006-12-14 11:50:29 EST
Created attachment 143657 [details]
Upstream proposal 

Both Keir an Ewan confirms that although undesirable, it is perfectly legal for
messages to be delivered twice. So, it becomes simpler than the path I was
taking (trying to figure out why the message was being delivered twice and
delivering it only once)
Comment 4 Glauber Costa 2006-12-14 14:38:01 EST
Created attachment 143685 [details]
upstream commit 

This is what was commited upstream.
Comment 5 RHEL Product and Program Management 2007-03-14 22:43:42 EDT
This request was evaluated by Red Hat Kernel Team for inclusion in a Red
Hat Enterprise Linux maintenance release, and has moved to bugzilla 
status POST.
Comment 6 Don Zickus 2007-04-23 17:58:45 EDT
in 2.6.18-16.el5
Comment 9 errata-xmlrpc 2007-11-07 14:17:05 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.