RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 605566 - RHEL6 PV domU kernel panic on RHEL5 xen host
Summary: RHEL6 PV domU kernel panic on RHEL5 xen host
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.0
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Andrew Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 582286
TreeView+ depends on / blocked
 
Reported: 2010-06-18 09:44 UTC by Alexander Todorov
Modified: 2013-01-09 22:45 UTC (History)
10 users (show)

Fixed In Version: kernel-2.6.32-37.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-11-10 20:55:36 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
disable transparent hugepages when running on Xen (1.91 KB, patch)
2010-06-18 17:59 UTC, Rik van Riel 		no flags Details | Diff
View All

Description Alexander Todorov 2010-06-18 09:44:25 UTC 
Description of problem:
On RHEL5 Xen host I started a PV domU with RHEL6 (0617.0 tree). I've chosen Linux/RHEL6 as os type/version. Right after boot the guest panics. See full console log below.

Version-Release number of selected component (if applicable):
kernel 2.6.32-36.el6.x86_64

How reproducible:
1/1 - I wasn't able to reproduce twice even with the same config

Steps to Reproduce:
1. Setup Xen host on ibm-crichton-02 with RHEL5.5 GA
2. Start a PV domU with latest RHEL6 snapshot
3. Select Linux/RHEL6 as OS type/variant
  
Actual results:
Kernel panic in the guest.

Expected results:
Guest boots.

Additional info:

# xm console t1
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-36.el6.x86_64 (mockbuild.redhat.com) (gcc version 4.4.4 20100611 (Red Hat 4.4.4-8) (GCC) ) #1 SMP Wed Jun 16 15:48:48 EDT 2010
Command line:  method=http://qafiler.bos.redhat.com/redhat/rel-eng/RHEL6.0-20100617.0/6/Server/x86_64/os vnc
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000020000000 (usable)
DMI not present or invalid.
last_pfn = 0x20000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000020000000
RAMDISK: 01cd8000 - 05315000
No NUMA configuration found
Faking a node at 0000000000000000-0000000020000000
Bootmem setup node 0 0000000000000000-0000000020000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [000000000003c000 -  000000000003ffff] pages 4
(7 early reservations) ==> bootmem [0000000000 - 0020000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0005418000 - 0005447000]   XEN PAGETABLES ==> [0005418000 - 0005447000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 0001cb7098]    TEXT DATA BSS ==> [0001000000 - 0001cb7098]
  #4 [0001cd8000 - 0005315000]          RAMDISK ==> [0001cd8000 - 0005315000]
  #5 [0005315000 - 0005418000]   XEN START INFO ==> [0005315000 - 0005418000]
  #6 [0000100000 - 00001d0000]          PGTABLE ==> [0000100000 - 00001d0000]
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00020000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 1 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at 20000000 (gap: 20000000:e0000000)
Booting paravirtualized kernel on Xen
Xen version: 3.1.2-194.el5 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff88000547d000 s94744 r8192 d24040 u126976
pcpu-alloc: s94744 r8192 d24040 u126976 alloc=31*4096
pcpu-alloc: [0] 0 
Xen: using vcpu_info placement
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 128972
Policy zone: DMA32
Kernel command line:  method=http://qafiler.bos.redhat.com/redhat/rel-eng/RHEL6.0-20100617.0/6/Server/x86_64/os vnc
PID hash table entries: 2048 (order: 2, 16384 bytes)
Checking aperture...
No AGP bridge found
Memory: 445488k/524288k available (4998k kernel code, 388k absent, 78412k reserved, 3964k data, 1220k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:256
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 5242880 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
Detected 1866.731 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 3733.46 BogoMIPS (lpj=1866731)
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Unsupported number of siblings 4
Performance Events: unsupported p6 CPU model 15 no PMU driver, software events only.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 33k freed
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20453 entries in 81 pages
Brought up 1 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 55540k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1276853656.085:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 978
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key F1E714242B6EEA55
- User ID: Red Hat, Inc. (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Console: switching to colour frame buffer device 100x37
console [tty0] enabled
xen-platform-pci: failed Xen IOPORT backend handshake: unrecognised magic value
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.0
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
input: Xen Virtual Keyboard as /devices/virtual/input/input1
input: Xen Virtual Pointer as /devices/virtual/input/input2
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
nf_conntrack version 0.5.0 (3914 buckets, 15656 max)
CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
No TPM chip found, activating TPM-bypass!
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vif/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1220k freed
Write protecting the kernel read-only data: 7276k
BUG: unable to handle kernel paging request at ffff88000446b040
IP: [<ffffffff81162966>] khugepaged+0xa86/0x1180
PGD 1002067 PUD 1006067 PMD 543d067 PTE 1000000446b065
Oops: 0003 [#1] SMP 
last sysfs file: /sys/module/gf128mul/initstate
CPU 0 
Modules linked in: xts(U) lrw(U) gf128mul(U) sha256_generic(U) cbc(U) dm_crypt(U) dm_round_robin(U) dm_multipath(U) dm_snapshot(U) dm_mirror(U) dm_region_hash(U) dm_log(U) dm_zero(U) dm_mod(U) linear(U) raid10(U) raid456(U) async_raid6_recov(U) async_pq(U) raid6_pq(U) async_xor(U) xor(U) async_memcpy(U) async_tx(U) raid1(U) raid0(U) ext2(U) mbcache(U) xen_netfront(U) xen_blkfront(U) ipv6(U) iscsi_ibft(U) pcspkr(U) iscsi_tcp(U) libiscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) squashfs(U) cramfs(U)

Modules linked in: xts(U) lrw(U) gf128mul(U) sha256_generic(U) cbc(U) dm_crypt(U) dm_round_robin(U) dm_multipath(U) dm_snapshot(U) dm_mirror(U) dm_region_hash(U) dm_log(U) dm_zero(U) dm_mod(U) linear(U) raid10(U) raid456(U) async_raid6_recov(U) async_pq(U) raid6_pq(U) async_xor(U) xor(U) async_memcpy(U) async_tx(U) raid1(U) raid0(U) ext2(U) mbcache(U) xen_netfront(U) xen_blkfront(U) ipv6(U) iscsi_ibft(U) pcspkr(U) iscsi_tcp(U) libiscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) squashfs(U) cramfs(U)
Pid: 27, comm: khugepaged Not tainted 2.6.32-36.el6.x86_64 #1 
RIP: e030:[<ffffffff81162966>]  [<ffffffff81162966>] khugepaged+0xa86/0x1180
RSP: e02b:ffff88001afebd90  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff88000412a900 RCX: ffff88000446b040
RDX: ffff88000412a900 RSI: ffff88000446b040 RDI: ffff880004707340
RBP: ffff88001afebee0 R08: ffffea0000380000 R09: ffff880004707870
R10: 0000000000000009 R11: ffff880000000040 R12: 0000000000000003
R13: ffff880004707340 R14: ffff8800041cf3f8 R15: 0000000000000000
FS:  00007fa0e679f700(0000) GS:ffff88000547d000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffff88000446b040 CR3: 0000000004554000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process khugepaged (pid: 27, threadinfo ffff88001afea000, task ffff88001afe4a70)
Stack:
 ffff88001afebdb0 ffffffff8100b7be ffff88001afe4a70 ffff88001afebe80
<0> 0000000001200000 00000000011ff000 ffff880000000040 0000000000000000
<0> ffff880000000000 ffff88001f1b8a70 ffff88001afea000 ffffffff81058d72
Call Trace:
 [<ffffffff8100b7be>] ? xen_end_context_switch+0x1e/0x30
 [<ffffffff81058d72>] ? finish_task_switch+0x52/0xd0
 [<ffffffff81090a50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff814da4ec>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff81161ee0>] ? khugepaged+0x0/0x1180
 [<ffffffff810906e6>] kthread+0x96/0xa0
 [<ffffffff810141ca>] child_rip+0xa/0x20
 [<ffffffff81013391>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81013b1d>] ? retint_restore_args+0x5/0x6
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
Code: bf 48 05 00 00 00 74 1a 48 8b 95 48 ff ff ff 48 8b b5 78 ff ff ff e8 0a d9 fe ff 48 8b 4d 80 48 8b 39 48 8b b5 68 ff ff ff 31 c0 <48> 87 06 48 89 85 50 ff ff ff 48 8b b5 78 ff ff ff 48 8b 95 68 
RIP  [<ffffffff81162966>] khugepaged+0xa86/0x1180
 RSP <ffff88001afebd90>
CR2: ffff88000446b040
---[ end trace 9647ca8849afcd47 ]---
Kernel panic - not syncing: Fatal exception
Pid: 27, comm: khugepaged Tainted: G      D    2.6.32-36.el6.x86_64 #1
Call Trace:
 [<ffffffff814d751d>] panic+0x78/0x137
 [<ffffffff814db654>] oops_end+0xe4/0x100
 [<ffffffff8104545b>] no_context+0xfb/0x260
 [<ffffffff8100c44d>] ? xen_write_cr0+0x4d/0xa0
 [<ffffffff810456e5>] __bad_area_nosemaphore+0x125/0x1e0
 [<ffffffff8100ca19>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff810457b3>] bad_area_nosemaphore+0x13/0x20
 [<ffffffff814dd138>] do_page_fault+0x2a8/0x3a0
 [<ffffffff814da9a5>] page_fault+0x25/0x30
 [<ffffffff81162966>] ? khugepaged+0xa86/0x1180
 [<ffffffff8100b7be>] ? xen_end_context_switch+0x1e/0x30
 [<ffffffff81058d72>] ? finish_task_switch+0x52/0xd0
 [<ffffffff81090a50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff814da4ec>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff81161ee0>] ? khugepaged+0x0/0x1180
 [<ffffffff810906e6>] kthread+0x96/0xa0
 [<ffffffff810141ca>] child_rip+0xa/0x20
 [<ffffffff81013391>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81013b1d>] ? retint_restore_args+0x5/0x6
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
Comment 1 Alexander Todorov 2010-06-18 09:48:22 UTC 
OK, so right after I filed the bug and said that I couldn't reproduce twice I continued with the install and this time got a different traceback while anaconda was scanning for disks.

------------[ cut here ]------------
Kernel BUG at ffffffff81164a80 [verbose debug info unavailable]
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/vbd-51712/block/xvda/xvda2/uevent
CPU 0 
Modules linked in: ext4(U) jbd2(U) fcoe(U) libfcoe(U) libfc(U) scsi_transport_fc(U) scsi_tgt(U) xts(U) lrw(U) gf128mul(U) sha256_generic(U) cbc(U) dm_crypt(U) dm_round_robin(U) dm_multipath(U) dm_snapshot(U) dm_mirror(U) dm_region_hash(U) dm_log(U) dm_zero(U) dm_mod(U) linear(U) raid10(U) raid456(U) async_raid6_recov(U) async_pq(U) raid6_pq(U) async_xor(U) xor(U) async_memcpy(U) async_tx(U) raid1(U) raid0(U) ext2(U) mbcache(U) xen_netfront(U) xen_blkfront(U) ipv6(U) iscsi_ibft(U) pcspkr(U) iscsi_tcp(U) libiscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) squashfs(U) cramfs(U) [last unloaded: scsi_wait_scan]

Modules linked in: ext4(U) jbd2(U) fcoe(U) libfcoe(U) libfc(U) scsi_transport_fc(U) scsi_tgt(U) xts(U) lrw(U) gf128mul(U) sha256_generic(U) cbc(U) dm_crypt(U) dm_round_robin(U) dm_multipath(U) dm_snapshot(U) dm_mirror(U) dm_region_hash(U) dm_log(U) dm_zero(U) dm_mod(U) linear(U) raid10(U) raid456(U) async_raid6_recov(U) async_pq(U) raid6_pq(U) async_xor(U) xor(U) async_memcpy(U) async_tx(U) raid1(U) raid0(U) ext2(U) mbcache(U) xen_netfront(U) xen_blkfront(U) ipv6(U) iscsi_ibft(U) pcspkr(U) iscsi_tcp(U) libiscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) squashfs(U) cramfs(U) [last unloaded: scsi_wait_scan]
Pid: 580, comm: lvm Not tainted 2.6.32-36.el6.x86_64 #1 
RIP: e030:[<ffffffff81164a80>]  [<ffffffff81164a80>] do_huge_pmd_anonymous_page+0x220/0x360
RSP: e02b:ffff880000455e58  EFLAGS: 00010293
RAX: ffff880005490680 RBX: 000000003144b0e7 RCX: 000000003144b0e7
RDX: ffff880000471038 RSI: 0000000000e00000 RDI: ffff8800006d2680
RBP: ffff880000455ea8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800007685c0
R13: ffff880000471038 R14: 0000000000e00000 R15: ffff8800006d2680
FS:  00007fe1faf257a0(0000) GS:ffff88000547d000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000e00000 CR3: 0000000012721000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process lvm (pid: 580, threadinfo ffff880000454000, task ffff8800005c9560)
Stack:
 ffffea000000f708 ffff8800006d2708 ffffea0000188000 ffffea0000000d90
<0> 0000000000000000 0000000000e00000 ffff8800007685c0 ffff8800006d2680
<0> ffff880000471038 ffff8800004eb000 ffff880000455ef8 ffffffff81133d05
Call Trace:
 [<ffffffff81133d05>] handle_mm_fault+0x245/0x2b0
 [<ffffffff814dcfb3>] do_page_fault+0x123/0x3a0
 [<ffffffff814da9a5>] page_fault+0x25/0x30
Code: 44 24 28 02 74 07 48 89 c3 48 83 cb 42 48 8b 7d c0 4c 89 f2 4c 89 e6 80 cb 80 e8 7c 78 fd ff 4c 89 ff 4c 89 f6 4c 89 ea 48 89 d9 <0f> 0b 0f 1f 44 00 00 49 8b b7 50 05 00 00 48 85 f6 0f 84 89 00 
RIP  [<ffffffff81164a80>] do_huge_pmd_anonymous_page+0x220/0x360
 RSP <ffff880000455e58>
---[ end trace 770e12bf94c53f97 ]---
Kernel panic - not syncing: Fatal exception
Pid: 580, comm: lvm Tainted: G      D    2.6.32-36.el6.x86_64 #1
Call Trace:
 [<ffffffff814d751d>] panic+0x78/0x137
 [<ffffffff814db654>] oops_end+0xe4/0x100
 [<ffffffff8101731b>] die+0x5b/0x90
 [<ffffffff814daf04>] do_trap+0xc4/0x160
 [<ffffffff81014ec5>] do_invalid_op+0x95/0xb0
 [<ffffffff81164a80>] ? do_huge_pmd_anonymous_page+0x220/0x360
 [<ffffffff810142fb>] ? xen_hypervisor_callback+0x1b/0x20
 [<ffffffff81013f5b>] invalid_op+0x1b/0x20
 [<ffffffff81164a80>] ? do_huge_pmd_anonymous_page+0x220/0x360
 [<ffffffff81164a74>] ? do_huge_pmd_anonymous_page+0x214/0x360
 [<ffffffff81133d05>] handle_mm_fault+0x245/0x2b0
 [<ffffffff814dcfb3>] do_page_fault+0x123/0x3a0
 [<ffffffff814da9a5>] page_fault+0x25/0x30
Comment 3 Andrew Jones 2010-06-18 10:07:03 UTC 
This is almost assuredly due to

commit 0617c50a1eb58625c011231c4892d1c9814d4839
Author: Aristeu Rozanski <arozansk>
Date:   Wed Jun 16 14:40:56 2010 -0400

    [mm] Reenable transparent hugepages
    
    Bugzilla: 602436
    
    This reverts commit 11907b67957b58e9d4763fa9021b48cd8e9a7251.


So it looks like we'll have to find a way to keep it unenabled for xen. I need to do some testing on a machine that uses huge pages.
Comment 4 Rik van Riel 2010-06-18 17:59:48 UTC 
Created attachment 425205 [details]
disable transparent hugepages when running on Xen
Comment 5 Dor Laor 2010-06-20 13:14:29 UTC 
Can you please test with guest kernel cmdline option of "transparent_hugepage=never" ?

It's from http://cleo.tlv.redhat.com/qumrawiki/KVM/TransparentHugepages
Comment 6 Alexander Todorov 2010-06-21 07:08:42 UTC 
Hi,
I've tested with transparent_hugepage=never. The install completed. Upon booting the installed system I got another panic:

PCI: Fatal: No config space access function found
------------[ cut here ]------------
Kernel BUG at ffffffff81164a80 [verbose debug info unavailable]
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/vbd-51712/block/xvda/xvda2/dev
CPU 0 
Modules linked in: xen_blkfront(U) dm_mod(U) [last unloaded: scsi_wait_scan]

Modules linked in: xen_blkfront(U) dm_mod(U) [last unloaded: scsi_wait_scan]
Pid: 234, comm: lvm Not tainted 2.6.32-36.el6.x86_64 #1 
RIP: e030:[<ffffffff81164a80>]  [<ffffffff81164a80>] do_huge_pmd_anonymous_page+0x220/0x360
RSP: e02b:ffff880003b05e58  EFLAGS: 00010297
RAX: ffff880004197680 RBX: 00000000212880e7 RCX: 00000000212880e7
RDX: ffff880003f2f028 RSI: 0000000000a00000 RDI: ffff880003c64700
RBP: ffff880003b05ea8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000206 R12: ffff880003ca8180
R13: ffff880003f2f028 R14: 0000000000a00000 R15: ffff880003c64700
FS:  00007fc5c50947a0(0000) GS:ffff880004184000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000a00100 CR3: 0000000003eaf000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process lvm (pid: 234, threadinfo ffff880003b04000, task ffff880003f39560)
Stack:
 ffffea00000cf990 ffff880003c64788 ffffea00000e7000 ffffea00000d9150
<0> 0000000000800000 0000000000a00100 ffff880003ca8180 ffff880003c64700
<0> ffff880003f2f028 ffff880003be5000 ffff880003b05ef8 ffffffff81133d05
Call Trace:
 [<ffffffff81133d05>] handle_mm_fault+0x245/0x2b0
 [<ffffffff814dcfb3>] do_page_fault+0x123/0x3a0
 [<ffffffff814da9a5>] page_fault+0x25/0x30
Code: 44 24 28 02 74 07 48 89 c3 48 83 cb 42 48 8b 7d c0 4c 89 f2 4c 89 e6 80 cb 80 e8 7c 78 fd ff 4c 89 ff 4c 89 f6 4c 89 ea 48 89 d9 <0f> 0b 0f 1f 44 00 00 49 8b b7 50 05 00 00 48 85 f6 0f 84 89 00 
RIP  [<ffffffff81164a80>] do_huge_pmd_anonymous_page+0x220/0x360
 RSP <ffff880003b05e58>
---[ end trace a017a41a7bfa1b9b ]---
Kernel panic - not syncing: Fatal exception
Pid: 234, comm: lvm Tainted: G      D    2.6.32-36.el6.x86_64 #1
Call Trace:
 [<ffffffff814d751d>] panic+0x78/0x137
 [<ffffffff814db654>] oops_end+0xe4/0x100
 [<ffffffff8101731b>] die+0x5b/0x90
 [<ffffffff814daf04>] do_trap+0xc4/0x160
 [<ffffffff81014ec5>] do_invalid_op+0x95/0xb0
 [<ffffffff81164a80>] ? do_huge_pmd_anonymous_page+0x220/0x360
 [<ffffffff810142fb>] ? xen_hypervisor_callback+0x1b/0x20
 [<ffffffff81013f5b>] invalid_op+0x1b/0x20
 [<ffffffff81164a80>] ? do_huge_pmd_anonymous_page+0x220/0x360
 [<ffffffff81164a74>] ? do_huge_pmd_anonymous_page+0x214/0x360
 [<ffffffff81133d05>] handle_mm_fault+0x245/0x2b0
 [<ffffffff814dcfb3>] do_page_fault+0x123/0x3a0
 [<ffffffff814da9a5>] page_fault+0x25/0x30
Comment 7 Andrew Jones 2010-06-21 08:57:48 UTC 
(In reply to comment #6)
> Hi,
> I've tested with transparent_hugepage=never. The install completed. Upon
> booting the installed system I got another panic:
> 

Are you sure that the command line parameter persisted? i.e. anaconda added it to the grub entry?

When you attempt to boot do you see it on the "Kernel command line:" boot output?
Comment 8 Andrew Jones 2010-06-21 09:06:39 UTC 
I just installed a -36 kernel guest with '-x transparent_hugepage=never' on my virt-install command line, and 'bootloader --append transparent_hugepage=never' in my kickstart, so it's in my grub as well. Everything seems to work fine.

It looks to me like the command line option is a viable workaround.
Comment 10 Paolo Bonzini 2010-06-21 13:22:59 UTC 
It looks like the bug is _triggered_ by transparent hugepages, but if the description by Rik in the patch is correct:

	/*
	 * Hugepages do not work in Xen's memory model, because even
	 * contiguous "physical" pages may not be machine contiguous.
	 */

Then we should make sure that PV guests never use hugepages---otherwise MAP_HUGETLB and hugetlbfs are at risk of panicking the guest too.
Comment 11 Andrea Arcangeli 2010-06-21 14:15:07 UTC 
The other problem is, a PV guest using hugepages to be safe in addition to create physically contiguous hugepages (which xen may not be able to as might misses the linux VM algorithms like buddy allocator, memory compaction, order-aware shrinker, migration to relocate all movable objects, anti-frag to keep movable and unmovable entities separated and more...) also needs to create the pagetable paravirt new methods that I added. Without those methods, even if xen was capable of creating hugepages on demand, it wouldn't work safe. Those methods are like .pmd_update, it's not enough anymore to implement .pte_update as the pmd is now pointing to hugepages directly.

I'm uncertain of the hugetlbfs implications, but surely there is no regression in that area. Perhaps hugetlbfs in guest PV never worked, I don't know.
Comment 12 Andrew Jones 2010-06-21 14:57:50 UTC 
It's true that trying to map many hugepages will not work. The problem is finding what the guest sees for contiguous pages. If you try to boot with too high of a value on the kernel command line, hugepages=<num_hugepages>, then you'll crash on boot. If you use a small number it can boot (64 worked for me). If you try to bump it up too high after booting, then it just fails gracefully by allocating as much as it can

# cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 

64

# echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 

1849


The guest functions fine with hugepages allocated. So I think we're safe for hugepages in general, and just need to ensure we don't attempt to use THP. This patch is good as it stands.
Comment 13 Aristeu Rozanski 2010-06-21 21:19:46 UTC 
Patch(es) available on kernel-2.6.32-37.el6
Comment 16 Alexander Todorov 2010-06-22 20:21:10 UTC 
(In reply to comment #7)
> (In reply to comment #6)
> > Hi,
> > I've tested with transparent_hugepage=never. The install completed. Upon
> > booting the installed system I got another panic:
> > 
> 
> Are you sure that the command line parameter persisted? i.e. anaconda added it
> to the grub entry?
> 

I don't think anaconda will add boot cmd line arguments to grub.conf by default. Your later test in comment #8 shows this works if grub.conf has the proper parameter.
Comment 19 Aristeu Rozanski 2010-07-01 16:24:15 UTC 
Patch(es) available on kernel-2.6.32-42.el6
Comment 24 releng-rhel@redhat.com 2010-11-10 20:55:36 UTC 
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.
Note You need to log in before you can comment on or make changes to this bug.