Description of problem: 1 multicall(s) failed: cpu 0 call 1/1: op=14 arg=[40002000] result=-22 ------------[ cut here ]------------ kernel BUG at arch/x86/xen/multicalls.c:103! invalid opcode: 0000 [#1] SMP Modules linked in: nfs lockd nfs_acl autofs4 sunrpc ipv6 dm_mirror dm_mod pcspkr xen_netfront xen_blkfront ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd Pid: 25479, comm: javac Not tainted (2.6.25.2-4.fc10.i686.xen #1) EIP: 0061:[<c0404043>] EFLAGS: 00010202 CPU: 0 EIP is at xen_mc_flush+0x163/0x16f EAX: 00000000 EBX: c125b054 ECX: 00000000 EDX: c125b054 ESI: c125b074 EDI: 00000000 EBP: d204cefc ESP: d204cee4 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 Process javac (pid: 25479, ti=d204c000 task=c1570000 task.ti=d204c000) Stack: c125b054 00000001 00000200 00000000 00030065 d1db7018 d204cf04 c0403e03 d204cf74 c04783dd 00000000 10fd50e0 40002fff 40002fff 80000000 00000025 40003000 d204cfa0 c4944e00 00000070 00000001 40003000 00000000 cd934000 Call Trace: [<c0403e03>] ? xen_leave_lazy+0x12/0x14 [<c04783dd>] ? mprotect_fixup+0x329/0x45d [<c047867b>] ? sys_mprotect+0x16a/0x1c7 [<c0408baa>] ? syscall_call+0x7/0xb ======================= Code: e8 8b 84 fa 04 0a 00 00 ff 94 fa 00 0a 00 00 47 8b 5d e8 3b bb 08 0b 00 00 72 e3 c7 83 08 0b 00 00 00 00 00 00 83 7d ec 00 74 04 <0f> 0b eb fe 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 89 d7 56 89 c6 EIP: [<c0404043>] xen_mc_flush+0x163/0x16f SS:ESP 0069:d204cee4 ---[ end trace a25165b18ad25d4a ]--- note: javac[25479] exited with preempt_count 1 BUG: sleeping function called from invalid context at kernel/rwsem.c:21 in_atomic():1, irqs_disabled():0 Pid: 25479, comm: javac Tainted: G D 2.6.25.2-4.fc10.i686.xen #1 [<c04238cb>] __might_sleep+0xae/0xb3 [<c0634ba9>] down_read+0x15/0x29 [<c0445967>] futex_wake+0x2e/0xc7 [<c0555a28>] ? scrup+0x87/0xb5 [<c0446950>] do_futex+0x91/0xa15 [<c063798d>] ? atomic_notifier_call_chain+0xf/0x11 [<c0555dcd>] ? vt_console_print+0x277/0x286 [<c0555b56>] ? vt_console_print+0x0/0x286 [<c042b925>] ? __call_console_drivers+0x56/0x63 [<c0635979>] ? _spin_unlock_irqrestore+0x10/0x14 [<c042bd5d>] ? release_console_sem+0x195/0x19d [<c04473a5>] sys_futex+0xd1/0xe7 [<c0429b68>] mm_release+0x61/0x6b [<c042d033>] exit_mm+0x15/0xdf [<c042e712>] do_exit+0x1ee/0x554 [<c040a19d>] die+0x15c/0x164 [<c06362ae>] do_trap+0x8a/0xa3 [<c040a564>] ? do_invalid_op+0x0/0x76 [<c040a5d0>] do_invalid_op+0x6c/0x76 [<c0404043>] ? xen_mc_flush+0x163/0x16f [<c042bd5d>] ? release_console_sem+0x195/0x19d [<c042c244>] ? vprintk+0x301/0x330 [<c042c244>] ? vprintk+0x301/0x330 [<c0635c0a>] error_code+0x72/0x78 [<c0404043>] ? xen_mc_flush+0x163/0x16f [<c0403e03>] xen_leave_lazy+0x12/0x14 [<c04783dd>] mprotect_fixup+0x329/0x45d [<c047867b>] sys_mprotect+0x16a/0x1c7 [<c0408baa>] syscall_call+0x7/0xb ======================= BUG: scheduling while atomic: javac/25479/0x10000001 Pid: 25479, comm: javac Tainted: G D 2.6.25.2-4.fc10.i686.xen #1 [<c0428a05>] __schedule_bug+0x4b/0x50 [<c0633b68>] schedule+0x97/0x6db [<c0428def>] __cond_resched+0x25/0x3b [<c0634299>] _cond_resched+0x24/0x2f [<c0634bae>] down_read+0x1a/0x29 [<c0445967>] futex_wake+0x2e/0xc7 [<c0555a28>] ? scrup+0x87/0xb5 [<c0446950>] do_futex+0x91/0xa15 [<c063798d>] ? atomic_notifier_call_chain+0xf/0x11 [<c0555dcd>] ? vt_console_print+0x277/0x286 [<c0555b56>] ? vt_console_print+0x0/0x286 [<c042b925>] ? __call_console_drivers+0x56/0x63 [<c0635979>] ? _spin_unlock_irqrestore+0x10/0x14 [<c042bd5d>] ? release_console_sem+0x195/0x19d [<c04473a5>] sys_futex+0xd1/0xe7 [<c0429b68>] mm_release+0x61/0x6b [<c042d033>] exit_mm+0x15/0xdf [<c042e712>] do_exit+0x1ee/0x554 [<c040a19d>] die+0x15c/0x164 [<c06362ae>] do_trap+0x8a/0xa3 [<c040a564>] ? do_invalid_op+0x0/0x76 [<c040a5d0>] do_invalid_op+0x6c/0x76 [<c0404043>] ? xen_mc_flush+0x163/0x16f [<c042bd5d>] ? release_console_sem+0x195/0x19d [<c042c244>] ? vprintk+0x301/0x330 [<c042c244>] ? vprintk+0x301/0x330 [<c0635c0a>] error_code+0x72/0x78 [<c0404043>] ? xen_mc_flush+0x163/0x16f [<c0403e03>] xen_leave_lazy+0x12/0x14 [<c04783dd>] mprotect_fixup+0x329/0x45d [<c047867b>] sys_mprotect+0x16a/0x1c7 [<c0408baa>] syscall_call+0x7/0xb ======================= BUG: scheduling while atomic: javac/25479/0x00000001 Pid: 25479, comm: javac Tainted: G D 2.6.25.2-4.fc10.i686.xen #1 [<c0428a05>] __schedule_bug+0x4b/0x50 [<c0633b68>] schedule+0x97/0x6db [<c063417a>] ? schedule+0x6a9/0x6db [<c0635504>] rwsem_down_failed_common+0x13e/0x15a [<c0635560>] rwsem_down_read_failed+0x1d/0x25 [<c06355e3>] call_rwsem_down_read_failed+0x7/0xc [<c0634bba>] ? down_read+0x26/0x29 [<c0445967>] futex_wake+0x2e/0xc7 [<c0555a28>] ? scrup+0x87/0xb5 [<c0446950>] do_futex+0x91/0xa15 [<c063798d>] ? atomic_notifier_call_chain+0xf/0x11 [<c0555dcd>] ? vt_console_print+0x277/0x286 [<c0555b56>] ? vt_console_print+0x0/0x286 [<c042b925>] ? __call_console_drivers+0x56/0x63 [<c0635979>] ? _spin_unlock_irqrestore+0x10/0x14 [<c042bd5d>] ? release_console_sem+0x195/0x19d [<c04473a5>] sys_futex+0xd1/0xe7 [<c0429b68>] mm_release+0x61/0x6b [<c042d033>] exit_mm+0x15/0xdf [<c042e712>] do_exit+0x1ee/0x554 [<c040a19d>] die+0x15c/0x164 [<c06362ae>] do_trap+0x8a/0xa3 [<c040a564>] ? do_invalid_op+0x0/0x76 [<c040a5d0>] do_invalid_op+0x6c/0x76 [<c0404043>] ? xen_mc_flush+0x163/0x16f [<c042bd5d>] ? release_console_sem+0x195/0x19d [<c042c244>] ? vprintk+0x301/0x330 [<c042c244>] ? vprintk+0x301/0x330 [<c0635c0a>] error_code+0x72/0x78 [<c0404043>] ? xen_mc_flush+0x163/0x16f [<c0403e03>] xen_leave_lazy+0x12/0x14 [<c04783dd>] mprotect_fixup+0x329/0x45d [<c047867b>] sys_mprotect+0x16a/0x1c7 [<c0408baa>] syscall_call+0x7/0xb ======================= Version-Release number of selected component (if applicable): 2.6.25.2-4.fc10.i686.xen Using javac from java-1.6.0-openjdk.
Do you have the output of 'xm dmesg' on the host, when the problem happened?
Well, the mm.c messages may be tied to it. I didn't see them before. See lots of the traps.c messages from before as well. (XEN) mm.c:625:d2 Non-privileged (2) attempt to map I/O space 00000030 (XEN) traps.c:1747:d4 Domain attempted WRMSR 000000000000017b from 00000000:0000001f to ffffffff:ffffffff. (XEN) traps.c:1747:d4 Domain attempted WRMSR 0000000000000400 from 00000000:00000000 to ffffffff:ffffffff. (XEN) traps.c:1747:d4 Domain attempted WRMSR 0000000000000404 from 00000000:ffffffff to ffffffff:ffffffff. (XEN) traps.c:1747:d4 Domain attempted WRMSR 0000000000000408 from 00000000:000fffff to ffffffff:ffffffff. (XEN) traps.c:1747:d4 Domain attempted WRMSR 000000000000040c from 00000000:00000007 to ffffffff:ffffffff. (XEN) traps.c:1747:d4 Domain attempted WRMSR 0000000000000410 from 00000000:00000000 to ffffffff:ffffffff. (XEN) mm.c:625:d4 Non-privileged (4) attempt to map I/O space 00000000
I'm now seeing similar on boot with 2.6.26-0.1.rc6.git2.fc10.i686.xen Checking if this processor honours the WP bit even in supervisor mode...Ok. CPA: page pool initialized 16 of 16 pages preallocated 1 multicall(s) failed: cpu 0 call 1/1: op=1 arg=[c160e854] result=-22 ------------[ cut here ]------------ kernel BUG at arch/x86/xen/multicalls.c:103! invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: Pid: 0, comm: swapper Tainted: G W (2.6.26-0.1.rc6.git2.fc10.i686.xen #1) EIP: e019:[<c040310e>] EFLAGS: 00010002 CPU: 0 EIP is at xen_mc_flush+0x18a/0x196 EAX: 00000001 EBX: c160e054 ECX: c042ded2 EDX: c074e3fc ESI: c160e074 EDI: 00000000 EBP: c078bf6c ESP: c078bf54 DS: e021 ES: e021 FS: 00d8 GS: 0000 SS: e021 Process swapper (pid: 0, ti=c078b000 task=c074e3fc task.ti=c078b000) Stack: c160e054 00000001 00000000 c160e854 c077c000 c160e054 c078bf88 c04038e0 2aa5c001 00000000 c077c000 c077d000 00000000 c078bfa0 c041c59a 2aa5c001 f4ffe000 c077d000 c087d000 c078bfc4 c07a200b 000004da 00000248 000012cc Call Trace: [<c04038e0>] ? xen_set_pud+0xbb/0xf3 [<c041c59a>] ? zap_low_mappings+0x2f/0x45 [<c07a200b>] ? mem_init+0x337/0x33f [<c078f844>] ? start_kernel+0x25d/0x316 [<c0795210>] ? xen_start_kernel+0x499/0x4a1 ======================= Code: e8 8b 84 fa 04 0a 00 00 ff 94 fa 00 0a 00 00 47 8b 5d e8 3b bb 08 0b 00 00 72 e3 c7 83 08 0b 00 00 00 00 00 00 83 7d ec 00 74 04 <0f> 0b eb fe 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 89 d7 56 89 c6 EIP: [<c040310e>] xen_mc_flush+0x18a/0x196 SS:ESP e021:c078bf54 ---[ end trace 4eaa2a86a8e2da22 ]--- Kernel panic - not syncing: Attempted to kill the idle task! [root@hammer console]# ls -ltr total 82868 -rw-r----- 1 root root 17296 2008-05-08 13:24 guest-xenearth.log -rw-r----- 1 root root 82922712 2008-06-16 12:06 guest-xenfdev64.log -rw-r----- 1 root root 1809648 2008-06-17 09:32 guest-xenfdev32.log [root@hammer console]# tail guest-xenfdev32.log [<c04038e0>] ? xen_set_pud+0xbb/0xf3 [<c041c59a>] ? zap_low_mappings+0x2f/0x45 [<c07a200b>] ? mem_init+0x337/0x33f [<c078f844>] ? start_kernel+0x25d/0x316 [<c0795210>] ? xen_start_kernel+0x499/0x4a1 ======================= Code: e8 8b 84 fa 04 0a 00 00 ff 94 fa 00 0a 00 00 47 8b 5d e8 3b bb 08 0b 00 00 72 e3 c7 83 08 0b 00 00 00 00 00 00 83 7d ec 00 74 04 <0f> 0b eb fe 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 89 d7 56 89 c6 EIP: [<c040310e>] xen_mc_flush+0x18a/0x196 SS:ESP e021:c078bf54 ---[ end trace 4eaa2a86a8e2da22 ]--- Kernel panic - not syncing: Attempted to kill the idle task! xm dmesg: (XEN) mm.c:694:d11 Bad L3 flags 6
Orion: do you still see this with kernel-PAE-2.6.27-0.180.rc0.git11.fc10 ?
Actually, looking at this bug, it's exactly the same as the one I've been debugging (on and off) in BZ 459067. I'm going to close it as a dup. Chris Lalancette *** This bug has been marked as a duplicate of bug 459067 ***