Bug 571241

Summary: [2.6.33.2] Linux 2.6.33 kernel crashes in free_init_pages() as Xen PV domU
Product: [Fedora] Fedora Reporter: Pasi Karkkainen <pasik>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 13CC: anton, dougsland, drjones, gansalmon, itamar, jforbes, jonathan, kernel-maint, kevin, mike, xen-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-04-09 13:50:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pasi Karkkainen 2010-03-07 19:52:01 UTC 
Description of problem:
Fedora 13 alpha 2.6.33 kernel (2.6.33-0.52.rc8.git6.fc13.x86_64) crashes in free_init_pages() when booted as Xen PV guest. There's a patch available to fix this bug.

Version-Release number of selected component (if applicable):
2.6.33-0.52.rc8.git6.fc13.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. Boot F13 kernel as Xen PV domU
2. kernel crashes during the boot process
  
Actual results:
Kernel crashes.

Expected results:
Kernel works normally.

Additional info:

Patch available here:
http://lists.xensource.com/archives/html/xen-devel/2010-02/msg01154.html

That patch has also been sent to upstream kernel.org for stable 2.6.32 and 2.6.33 kernels.

Log of the crashing boot:

(early) Initializing cgroup subsys cpuset
(early) Initializing cgroup subsys cpu
(early) Linux version 2.6.33-0.52.rc8.git6.fc13.x86_64 (mockbuild.fedoraproject.org) (gcc version 4.4.3 20100211 (Red Hat 4.4.3-6) (GCC) ) #1 SMP Tue Feb 23 04:52:05 UTC 2010
(early) Command line: earlyprintk=xen console=hvc0
(early) ACPI in unprivileged domain disabled
(early) BIOS-provided physical RAM map:
(early)  Xen: 0000000000000000 - 00000000000a0000 (usable)
(early)  Xen: 00000000000a0000 - 0000000000100000 (reserved)
(early)  Xen: 0000000000100000 - 0000000020000000 (usable)
(early) bootconsole [xenboot0] enabled
(early) NX (Execute Disable) protection: active
(early) DMI not present or invalid.
(early) No AGP bridge found
(early) last_pfn = 0x20000 max_arch_pfn = 0x400000000
(early) init_memory_mapping: 0000000000000000-0000000020000000
(early) RAMDISK: 029da000 - 06102000
(early) No NUMA configuration found
(early) Faking a node at 0000000000000000-0000000020000000
(early) Bootmem setup node 0 0000000000000000-0000000020000000
(early)   NODE_DATA [0000000000007000 - 000000000001ffff]
(early)   bootmap [0000000000020000 -  0000000000023fff] pages 4
(early) (8 early reservations) ==> bootmem [0000000000 - 0020000000]
(early)   #0 [0000000000 - 0000001000]   BIOS data page(early)  ==> [0000000000 - 0000001000]
(early)   #1 [0006205000 - 000623a000]   XEN PAGETABLES(early)  ==> [0006205000 - 000623a000]
(early)   #2 [0001000000 - 00029b9138]    TEXT DATA BSS(early)  ==> [0001000000 - 00029b9138]
(early)   #3 [00029da000 - 0006102000]          RAMDISK(early)  ==> [00029da000 - 0006102000]
(early)   #4 [0006102000 - 0006205000]   XEN START INFO(early)  ==> [0006102000 - 0006205000]
(early)   #5 [0000001000 - 0000003000]       TRAMPOLINE(early)  ==> [0000001000 - 0000003000]
(early)   #6 [0000003000 - 0000007000]      ACPI WAKEUP(early)  ==> [0000003000 - 0000007000]
(early)   #7 [0000100000 - 00001ca000]          PGTABLE(early)  ==> [0000100000 - 00001ca000]
(early) Zone PFN ranges:
(early)   DMA      0x00000000 -> 0x00001000
(early)   DMA32    0x00001000 -> 0x00100000
(early)   Normal   0x00100000 -> 0x00100000
(early) Movable zone start PFN for each node
(early) early_node_map[2] active PFN ranges
(early)     0: 0x00000000 -> 0x000000a0
(early)     0: 0x00000100 -> 0x00020000
(early) SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
(early) SMP: Allowing 1 CPUs, 0 hotplug CPUs
(early) No local APIC present
(early) APIC: disable apic facility
(early) APIC: switched to apic NOOP
(early) PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
(early) Allocating PCI resources starting at 20000000 (gap: 20000000:e0000000)
(early) Booting paravirtualized kernel on Xen
(early) Xen version: 4.0.0-rc5 (preserve-AD)
(early) setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:1 nr_node_ids:1
(early) PERCPU: Embedded 478 pages/cpu @ffff8800062e0000 s1927384 r8192 d22312 u1957888
(early) pcpu-alloc: s1927384 r8192 d22312 u1957888 alloc=478*4096(early) 
(early) pcpu-alloc: (early) [0] (early) 0 (early) 
(early) Xen: using vcpu_info placement
(early) Built 1 zonelists in Node order, mobility grouping on.  Total pages: 128975
(early) Policy zone: DMA32
(early) Kernel command line: earlyprintk=xen console=hvc0
(early) PID hash table entries: 2048 (order: 2, 16384 bytes)
(early) Checking aperture...
(early) No AGP bridge found
(early) Memory: 429100k/524288k available (4608k kernel code, 384k absent, 94804k reserved, 7334k data, 2752k init)
(early) SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
(early) Hierarchical RCU implementation.
(early) NR_IRQS:4352 nr_irqs:256
(early) Console: colour dummy device 80x25
(early) Initializing cgroup subsys cpuset
(early) Initializing cgroup subsys cpu
(early) Linux version 2.6.33-0.52.rc8.git6.fc13.x86_64 (mockbuild.fedoraproject.org) (gcc version 4.4.3 20100211 (Red Hat 4.4.3-6) (GCC) ) #1 SMP Tue Feb 23 04:52:05 UTC 2010
(early) Command line: earlyprintk=xen console=hvc0
(early) ACPI in unprivileged domain disabled
(early) BIOS-provided physical RAM map:
(early)  Xen: 0000000000000000 - 00000000000a0000 (usable)
(early)  Xen: 00000000000a0000 - 0000000000100000 (reserved)
(early)  Xen: 0000000000100000 - 0000000020000000 (usable)
(early) bootconsole [xenboot0] enabled
(early) NX (Execute Disable) protection: active
(early) DMI not present or invalid.
(early) No AGP bridge found
(early) last_pfn = 0x20000 max_arch_pfn = 0x400000000
(early) init_memory_mapping: 0000000000000000-0000000020000000
(early) RAMDISK: 029da000 - 06102000
(early) No NUMA configuration found
(early) Faking a node at 0000000000000000-0000000020000000
(early) Bootmem setup node 0 0000000000000000-0000000020000000
(early)   NODE_DATA [0000000000007000 - 000000000001ffff]
(early)   bootmap [0000000000020000 -  0000000000023fff] pages 4
(early) (8 early reservations) ==> bootmem [0000000000 - 0020000000]
(early)   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
(early)   #1 [0006205000 - 000623a000]   XEN PAGETABLES ==> [0006205000 - 000623a000]
(early)   #2 [0001000000 - 00029b9138]    TEXT DATA BSS ==> [0001000000 - 00029b9138]
(early)   #3 [00029da000 - 0006102000]          RAMDISK ==> [00029da000 - 0006102000]
(early)   #4 [0006102000 - 0006205000]   XEN START INFO ==> [0006102000 - 0006205000]
(early)   #5 [0000001000 - 0000003000]       TRAMPOLINE ==> [0000001000 - 0000003000]
(early)   #6 [0000003000 - 0000007000]      ACPI WAKEUP ==> [0000003000 - 0000007000]
(early)   #7 [0000100000 - 00001ca000]          PGTABLE ==> [0000100000 - 00001ca000]
(early) Zone PFN ranges:
(early)   DMA      0x00000000 -> 0x00001000
(early)   DMA32    0x00001000 -> 0x00100000
(early)   Normal   0x00100000 -> 0x00100000
(early) Movable zone start PFN for each node
(early) early_node_map[2] active PFN ranges
(early)     0: 0x00000000 -> 0x000000a0
(early)     0: 0x00000100 -> 0x00020000
(early) SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
(early) SMP: Allowing 1 CPUs, 0 hotplug CPUs
(early) No local APIC present
(early) APIC: disable apic facility
(early) APIC: switched to apic NOOP
(early) PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
(early) Allocating PCI resources starting at 20000000 (gap: 20000000:e0000000)
(early) Booting paravirtualized kernel on Xen
(early) Xen version: 4.0.0-rc5 (preserve-AD)
(early) setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:1 nr_node_ids:1
(early) PERCPU: Embedded 478 pages/cpu @ffff8800062e0000 s1927384 r8192 d22312 u1957888
(early) pcpu-alloc: s1927384 r8192 d22312 u1957888 alloc=478*4096
(early) pcpu-alloc: [0] 0 
(early) Xen: using vcpu_info placement
(early) Built 1 zonelists in Node order, mobility grouping on.  Total pages: 128975
(early) Policy zone: DMA32
(early) Kernel command line: earlyprintk=xen console=hvc0
(early) PID hash table entries: 2048 (order: 2, 16384 bytes)
(early) Checking aperture...
(early) No AGP bridge found
(early) Memory: 429100k/524288k available (4608k kernel code, 384k absent, 94804k reserved, 7334k data, 2752k init)
(early) SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
(early) Hierarchical RCU implementation.
(early) NR_IRQS:4352 nr_irqs:256
(early) Console: colour dummy device 80x25
(early) console [tty0] enabled
console [hvc0] enabled, bootconsole disabled
(early) console [hvc0] enabled, bootconsole disabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:  8
... MAX_LOCK_DEPTH:          48
... MAX_LOCKDEP_KEYS:        8191
... CLASSHASH_SIZE:          4096
... MAX_LOCKDEP_ENTRIES:     16384
... MAX_LOCKDEP_CHAINS:      32768
... CHAINHASH_SIZE:          16384
 memory used by lock dependency info: 6367 kB
 per task-struct memory footprint: 2688 bytes
allocated 5242880 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
Detected 2826.320 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5652.64 BogoMIPS (lpj=2826320)
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Unsupported number of siblings 4
Performance Events: unsupported p6 CPU model 23 no PMU driver, software events only.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 30k freed
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 21581 entries in 85 pages
Brought up 1 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
Time: 165:165:165  Date: 165/165/65
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
kstop/0 used greatest stack depth: 6248 bytes left
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 8, 1179648 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
UDP hash table entries: 256 (order: 3, 40960 bytes)
UDP-Lite hash table entries: 256 (order: 3, 40960 bytes)
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 56480k freed
DMA-API: preallocated 32768 debug entries
DMA-API: debugging enabled by kernel config
platform rtc_cmos: registered platform RTC device (no PNP device found)
Intel PCLMULQDQ-NI instructions are not detected.
audit: initializing netlink socket (disabled)
type=2000 audit(1268026831.770:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 948
cryptomgr_test used greatest stack depth: 5416 bytes left
alg: No test for stdrng (krng)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.16.0-ioctl (2009-11-05) initialised: dm-devel
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
nf_conntrack version 0.5.0 (3793 buckets, 15172 max)
CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
No TPM chip found, activating TPM-bypass!
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/console/0
  Magic number: 1:252:3141
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 2752k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 1516k freed
BUG: unable to handle kernel paging request at ffff880001485000
IP: [<ffffffff8102ec62>] free_init_pages+0xb2/0xdb
PGD 1a44067 PUD 1a48067 PMD 6212067 PTE 10000001485025
Oops: 0003 [#1] SMP 
last sysfs file: 
CPU 0 
Pid: 1, comm: swapper Not tainted 2.6.33-0.52.rc8.git6.fc13.x86_64 #1 /
RIP: e030:[<ffffffff8102ec62>]  [<ffffffff8102ec62>] free_init_pages+0xb2/0xdb
RSP: e02b:ffff88001fdcbe60  EFLAGS: 00010286
RAX: 00000000cccccccc RBX: ffff880001600000 RCX: 0000000000000400
RDX: ffff880001485000 RSI: 0000000000000000 RDI: ffff880001485000
RBP: ffff88001fdcbe90 R08: 0000000000002839 R09: ffffffff8107ba5c
R10: ffffffff81a569f8 R11: 0000000000000000 R12: ffff880001485000
R13: 0000000000000400 R14: ffffea0000000000 R15: 00000000cccccccc
FS:  0000000000000000(0000) GS:ffff8800062e0000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffff880001485000 CR3: 0000000001a43000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo ffff88001fdca000, task ffff88001fdd0000)
Stack:
 0000000000000000 ffff880000000000 6db6db6db6db6db7 ffffffff81a00000
<0> 0000000000a00000 0000000000000000 ffff88001fdcbec0 ffffffff8102efe3
<0> ffffffff81e4c6e8 ffffffff81ba3050 0000000000000200 0000000000000200
Call Trace:
 [<ffffffff8102efe3>] mark_rodata_ro+0xea/0x151
 [<ffffffff810021ea>] init_post+0x30/0x116
 [<ffffffff81d827b2>] kernel_init+0x260/0x26f
 [<ffffffff8100aae4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81479010>] ? restore_args+0x0/0x30
 [<ffffffff8100aae0>] ? kernel_thread_helper+0x0/0x10
Code: c5 49 00 00 48 c1 e8 0c 4c 89 e2 4c 89 e9 48 6b c0 38 48 81 e2 00 f0 ff ff 31 f6 48 89 d7 4c 01 f0 c7 40 08 01 00 00 00 44 89 f8 <f3> ab 4c 89 e7 49 81 c4 00 10 00 00 e8 ef 13 0b 00 48 ff 05 46 
RIP  [<ffffffff8102ec62>] free_init_pages+0xb2/0xdb
 RSP <ffff88001fdcbe60>
CR2: ffff880001485000
---[ end trace 62b0169f63ee0a59 ]---
swapper used greatest stack depth: 4536 bytes left
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: swapper Tainted: G      D    2.6.33-0.52.rc8.git6.fc13.x86_64 #1
Call Trace:
 [<ffffffff814759c9>] panic+0x7a/0x142
 [<ffffffff8105404b>] ? do_exit+0x3a8/0x7a5
 [<ffffffff81053d1e>] do_exit+0x7b/0x7a5
 [<ffffffff81478d22>] ? _raw_spin_unlock_irqrestore+0x40/0x56
 [<ffffffff81479f4b>] oops_end+0xbf/0xc7
 [<ffffffff8102fc35>] no_context+0x1fc/0x20b
 [<ffffffff8107ba5c>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff8102fdd6>] __bad_area_nosemaphore+0x192/0x1b5
 [<ffffffff81004435>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8102fe0c>] bad_area_nosemaphore+0x13/0x15
 [<ffffffff8147ba80>] do_page_fault+0x1c1/0x331
 [<ffffffff814792b5>] page_fault+0x25/0x30
 [<ffffffff8107ba5c>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff8102ec62>] ? free_init_pages+0xb2/0xdb
 [<ffffffff8102efe3>] mark_rodata_ro+0xea/0x151
 [<ffffffff810021ea>] init_post+0x30/0x116
 [<ffffffff81d827b2>] kernel_init+0x260/0x26f
 [<ffffffff8100aae4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81479010>] ? restore_args+0x0/0x30
 [<ffffffff8100aae0>] ? kernel_thread_helper+0x0/0x10
Comment 1 Andrew Jones 2010-03-10 18:50:53 UTC 
*** Bug 567002 has been marked as a duplicate of this bug. ***
Comment 2 Chuck Ebbert 2010-03-14 06:58:23 UTC 
X-Git-Url: http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Flinux-2.6.git;a=commitdiff_plain;h=281ff33b7c1b1ba2a5f9b03425e5f692a94913fa
Comment 3 Pasi Karkkainen 2010-03-27 17:41:33 UTC 
Hopefully this patch can be added to next Fedora 13 kernel build so we can test F13 Xen PV guests aswell..
Comment 4 Pasi Karkkainen 2010-03-29 07:47:52 UTC 
This crash still happens with 2.6.33.1-19.fc13.x86_64
Comment 5 Pasi Karkkainen 2010-04-01 08:10:35 UTC 
This patch is included in the 2.6.33.2 stable review series on lkml.
Comment 6 Chuck Ebbert 2010-04-06 11:24:21 UTC 
2.6.33.2 is in koji now
Comment 7 Pasi Karkkainen 2010-04-06 11:58:08 UTC 
Nice. Will 2.6.33.2 be in the f13 beta release?
Comment 8 Kevin Fenzi 2010-04-08 03:01:21 UTC 
*** Bug 568415 has been marked as a duplicate of this bug. ***
Comment 9 Kevin Fenzi 2010-04-08 03:02:20 UTC 
2.6.33.2 boots here as a Xen domU. 

It falls down under heavy use, but thats another bug. ;)