Bug 709856 - Kernel trace on m2.4xlarge or m2.2xlarge instances in EC2
Kernel trace on m2.4xlarge or m2.2xlarge instances in EC2
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.1
x86_64 Linux
urgent Severity urgent
: rc
: ---
Assigned To: Andrew Jones
Red Hat Kernel QE team
: EC2, ZStream
Depends On:
Blocks: 653816 710609 711317 711322
  Show dependency treegraph
 
Reported: 2011-06-01 15:17 EDT by Jay Greguske
Modified: 2013-01-09 18:55 EST (History)
16 users (show)

See Also:
Fixed In Version: kernel-2.6.32-156.el6
Doc Type: Bug Fix
Doc Text:
Xen guests cannot make use of all CPU features, and in some cases they are even risky to be advertised. One such feature is CONSTANT_TSC. This feature prevents the TSC (Time Stamp Counter) from being marked as unstable, which allows the sched_clock_stable option to be enabled. Having the sched_clock_stable option enabled is problematic for Xen PV guests because the sched_clock() function has been overridden with the xen_sched_clock() function, which is not synchronized between virtual CPUs. This update provides a patch, which sets all x86_power features to 0 as a preventive measure against other potentially dangerous assumptions the kernel could make based on the features, fixing this issue.
Story Points: ---
Clone Of:
: 711317 711322 (view as bug list)
Environment:
Last Closed: 2011-12-06 08:08:46 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
uname -a for instance in apac-sing (117 bytes, application/octet-stream)
2011-06-02 07:10 EDT, Chris Morgan
no flags Details
/proc/cpuinfo from apac-sing (4.45 KB, application/octet-stream)
2011-06-02 07:12 EDT, Chris Morgan
no flags Details
cpu proc, uptime data for 10 instances across all regions (47.90 KB, text/plain)
2011-06-02 09:00 EDT, wes hayutin
no flags Details
eu-west x86_64 m2.2xlarge crash console (62.61 KB, text/plain)
2011-06-02 11:26 EDT, wes hayutin
no flags Details
ap-southeast-x86_64_rhel61_m2.2xlargeCRASH (62.60 KB, text/plain)
2011-06-02 11:36 EDT, wes hayutin
no flags Details
mask nonstop_tsc (1.44 KB, patch)
2011-06-03 02:12 EDT, Andrew Jones
no flags Details | Diff
zero all x86_power feature bits (818 bytes, patch)
2011-06-03 02:44 EDT, Andrew Jones
no flags Details | Diff
beaker tier1/2 kernel qe tests.. tests still running no complete (133.36 KB, application/x-gzip)
2011-06-04 08:56 EDT, wes hayutin
no flags Details
tier1 & tier2 kernel qe tests (903.49 KB, text/plain)
2011-06-04 14:35 EDT, wes hayutin
no flags Details
guest console log (11.19 KB, text/plain)
2011-06-08 10:13 EDT, Qixiang Wan
no flags Details

  None (edit)
Description Jay Greguske 2011-06-01 15:17:26 EDT
Description of problem:

When starting a RHEL 6.1 high-memory or high-cpu instance in EC2 the kernel throws a stack trace.

Version-Release number of selected component (if applicable):

2.6.32-131.0.15.el6.x86_64

Actual results:

usbhid: v2.6:USB HID core driver
BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1]
Modules linked in:
CPU 0:
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff810a40f0>]  [<ffffffff810a40f0>] generic_exec_single+0x90/0xc0
RSP: e02b:ffff880431767b50  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff88002807f140 RCX: 000000000000003e
RDX: 0000000000000002 RSI: 0000000000000002 RDI: ffffffff8160ba00
RBP: ffff880431767b90 R08: ffffffff8160ba00 R09: 0000000000000000
R10: fffffffffffffff8 R11: 0000000000000030 R12: ffff880431767bb0
R13: ffff88002807f150 R14: 0000000000000001 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff880028050000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Call Trace:
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a4352>] smp_call_function_single+0x152/0x160
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a454a>] smp_call_function_many+0x1ea/0x210
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a4592>] smp_call_function+0x22/0x30
[<ffffffff8106f334>] on_each_cpu+0x24/0x50
[<ffffffff8115c46f>] do_tune_cpucache+0x12f/0x630
[<ffffffff8115cb4b>] enable_cpucache+0x3b/0xf0
[<ffffffff814c5d3f>] setup_cpu_cache+0x22f/0x340
[<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8115ac92>] ? kmem_cache_alloc+0x182/0x190
[<ffffffff8115d7da>] kmem_cache_create+0x3fa/0x580
[<ffffffff81bfd1b7>] flow_cache_init+0x4a/0x1af
[<ffffffff81bfd16d>] ? flow_cache_init+0x0/0x1af
[<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
[<ffffffff81bbd884>] kernel_init+0x29d/0x2f9
[<ffffffff8100c1ca>] child_rip+0xa/0x20
[<ffffffff8100b393>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100bb1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 66s! [swapper:1]
Modules linked in:
CPU 0:
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff810a40ea>]  [<ffffffff810a40ea>] generic_exec_single+0x8a/0xc0
RSP: e02b:ffff880431767b50  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff88002807f140 RCX: 000000000000003e
RDX: 0000000000000002 RSI: 0000000000000002 RDI: ffffffff8160ba00
RBP: ffff880431767b90 R08: ffffffff8160ba00 R09: 0000000000000000
R10: fffffffffffffff8 R11: 0000000000000030 R12: ffff880431767bb0
R13: ffff88002807f150 R14: 0000000000000001 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff880028050000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Call Trace:
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a4352>] smp_call_function_single+0x152/0x160
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a454a>] smp_call_function_many+0x1ea/0x210
[<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a4592>] smp_call_function+0x22/0x30
[<ffffffff8106f334>] on_each_cpu+0x24/0x50
[<ffffffff8115c46f>] do_tune_cpucache+0x12f/0x630
[<ffffffff8115cb4b>] enable_cpucache+0x3b/0xf0
[<ffffffff814c5d3f>] setup_cpu_cache+0x22f/0x340
[<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8115ac92>] ? kmem_cache_alloc+0x182/0x190
[<ffffffff8115d7da>] kmem_cache_create+0x3fa/0x580
[<ffffffff81bfd1b7>] flow_cache_init+0x4a/0x1af
[<ffffffff81bfd16d>] ? flow_cache_init+0x0/0x1af
[<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
[<ffffffff81bbd884>] kernel_init+0x29d/0x2f9
[<ffffffff8100c1ca>] child_rip+0xa/0x20
[<ffffffff8100b393>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100bb1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c1c0>] ? child_rip+0x0/0x20

Additional Information:

High-Memory Double Extra Large Instance

34.2 GB of memory
13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each)
850 GB of instance storage
64-bit platform
I/O Performance: High
API name: m2.2xlarge

High-Memory Quadruple Extra Large Instance

68.4 GB of memory
26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: m2.4xlarge

Other instance types boot ok. More info on instance types is here:

http://aws.amazon.com/ec2/instance-types/
Comment 2 Jay Greguske 2011-06-01 15:40:59 EDT
Ben,

RH Kernel folks are asking all of the output besides the kernel trace.
Comment 3 Ben Howard 2011-06-01 16:26:27 EDT
i-bb12a1d5
2011-06-01T20:24:40+0000
Xen Minimal OS!
  start_info: 0x4dec000(VA)
    nr_pages: 0x88b800
  shared_inf: 0xbf58d000(MA)
     pt_base: 0x4def000(VA)
nr_pt_frames: 0x2b
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line:  root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x5ff6d(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80b00(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: 4e1d
    max_pfn: 88b800
Mapping memory range 0x5000000 - 0x88b800000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 9273000(9273000)-88b800000(88b800000)
MM: done
Demand map pfns at 88b801000-288b801000.
Heap resides at 288b802000-488b802000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x88b801000.
Initialising scheduler
Thread "Idle": pointer: 0x288b802010, stack: 0xa010000
Initialising xenbus
Thread "xenstore": pointer: 0x288b8027c0, stack: 0xa020000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x288b802f70, stack: 0xa030000
"main" "root=/dev/sda1" "ro" "4"
vbd 2048 is hd0
******************* BLKFRONT for device/vbd/2048 **********


backend at /local/domain/0/backend/vbd/62/2048
Failed to read /local/domain/0/backend/vbd/62/2048/feature-barrier.
Failed to read /local/domain/0/backend/vbd/62/2048/feature-flush-cache.
  Booting 'RHEL6.1-20110510.1-Server-x86_64-starter-ec2 (2.6.32-131.0.15.el6.x8
6_64)'

root (hd0)
 Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=LABEL=_/
initrd /boot/initramfs-2.6.32-131.0.15.el6.x86_64.img

close blk: backend at /local/domain/0/backend/vbd/62/2048
close blk: backend at /local/domain/0/backend/vbd/62/2128
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-131.0.15.el6.x86_64 (mockbuild@x86-007.build.bos.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Tue May 10 15:42:40 EDT 2011
Command line: ro root=LABEL=_/
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000800000000 (usable)
DMI not present or invalid.
last_pfn = 0x800000 max_arch_pfn = 0x400000000
last_pfn = 0x100000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000100000000
init_memory_mapping: 0000000100000000-0000000800000000
RAMDISK: 01f68000 - 046d8000
No NUMA configuration found
Faking a node at 0000000000000000-0000000800000000
Bootmem setup node 0 0000000000000000-0000000800000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [00000000008b7000 -  00000000009b6fff] pages 100
(8 early reservations) ==> bootmem [0000000000 - 0800000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0008b37000 - 0008b82000]   XEN PAGETABLES ==> [0008b37000 - 0008b82000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 0001f474e4]    TEXT DATA BSS ==> [0001000000 - 0001f474e4]
  #4 [0001f68000 - 00046d8000]          RAMDISK ==> [0001f68000 - 00046d8000]
  #5 [00046d8000 - 0008b37000]   XEN START INFO ==> [00046d8000 - 0008b37000]
  #6 [0000100000 - 00008b7000]          PGTABLE ==> [0000100000 - 00008b7000]
  #7 [0008b82000 - 000c39e000]          PGTABLE ==> [0008b82000 - 000c39e000]
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00800000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00800000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 4 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 800100000 (gap: 800100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 3.1.2-128.1.10.el5 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 30 pages/cpu @ffff880028050000 s92504 r8192 d22184 u122880
pcpu-alloc: s92504 r8192 d22184 u122880 alloc=30*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Xen: using vcpu_info placement
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8271845
Policy zone: Normal
Kernel command line: ro root=LABEL=_/
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 32835652k/33554432k available (5014k kernel code, 388k absent, 718392k reserved, 6906k data, 1232k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:304
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 335544320 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
Detected 2666.760 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.52 BogoMIPS (lpj=2666760)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Unsupported number of siblings 16
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20700 entries in 82 pages
Performance Events: PEBS fmt1+, Nehalem events,
no APIC, boot with the "lapic" boot parameter to force-enable it.
no hardware sampling interrupt available.
Broken PMU hardware detected, using software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
installing Xen timer for CPU 1
SMP alternatives: switching to SMP code
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 2
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 3
CPU: Unsupported number of siblings 16
Brought up 4 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 40384k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1306959870.823:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key C9F0DFBFCE81A817
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
invalid opcode: 0000 [#1] SMP
last sysfs file:
CPU 3
Modules linked in:

Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff812bb828>]  [<ffffffff812bb828>] intel_idle+0x98/0x170
RSP: e02b:ffff8807dc979e80  EFLAGS: 00010046
RAX: ffff8807dc978010 RBX: 0000000000000004 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8807dc979fd8 RDI: ffff8800280b5040
RBP: ffff8807dc979ef0 R08: 0000000000000000 R09: 0000000000000320
R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000010
R13: 12234113f555059a R14: 0000000000000002 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff8800280aa000(0000) knlGS:0000000000000000
CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process swapper (pid: 0, threadinfo ffff8807dc978000, task ffff8807dc974ac0)
Stack:
 0000000000000002 ffff8800280b0240 0000000000000001 ffff8800280b6640
<0> 0000000006acfc00 00000000000861c3 ffffffff81007b3f 00000003814e0cb6
<0> ffff8807dc979ef0 ffff8800280c72c0 ffff8800280c7390 0000000000000000
Call Trace:
 [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81007af9>] ? xen_irq_enable_direct_end+0x0/0x7
 [<ffffffff814cee35>] cpu_bringup_and_idle+0x13/0x15
Code: 8b 7d cc 85 c0 0f 85 b3 00 00 00 65 48 8b 34 25 08 cc 00 00 48 8b 86 38 e0 ff ff a8 08 75 25 31 d2 48 8d 86 38 e0 ff ff 48 89 d1 <0f> 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89
RIP  [<ffffffff812bb828>] intel_idle+0x98/0x170
 RSP <ffff8807dc979e80>
invalid opcode: 0000 [#2]
---[ end trace 3a5e7c9663b4e1f1 ]---
SMP
last sysfs file:
CPU 0
Modules linked in:

Modules linked in:
Pid: 0, comm: swapper Tainted: G      D    ----------------   2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff812bb828>]  [<ffffffff812bb828>] intel_idle+0x98/0x170
RSP: e02b:ffffffff81a01ea8  EFLAGS: 00010046
RAX: ffffffff81a00010 RBX: 0000000000000008 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: ffff88002805b040
RBP: ffffffff81a01f18 R08: 0000000000000000 R09: 00000000000000c8
R10: 000000003b9aca00 R11: 00000000fffb6c80 R12: 0000000000000020
R13: 12234113f5554508 R14: 0000000000000003 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880028050000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000289560f000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a2d020)
Stack:
 ffff88002805c640 ffff880028050240 000000001dbe00c0 ffff88002805c640
<0> 0000000000000001 0000000017044e80 ffffffff81007b3f 00000000814e0cb6
<0> ffffffff81a01f18 ffff88002806d2c0 ffff88002806d3f0 0000000000000000
Call Trace:
 [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff813eccb7>] cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009e96>] cpu_idle+0xb6/0x110
 [<ffffffff814c376a>] rest_init+0x7a/0x80
 [<ffffffff81bbdf28>] start_kernel+0x41d/0x429
 [<ffffffff81bbd33a>] x86_64_start_reservations+0x125/0x129
 [<ffffffff81bc111b>] xen_start_kernel+0x582/0x586
Code: 8b 7d cc 85 c0 0f 85 b3 00 00 00 65 48 8b 34 25 08 cc 00 00 48 8b 86 38 e0 ff ff a8 08 75 25 31 d2 48 8d 86 38 e0 ff ff 48 89 d1 <0f> 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89
RIP  [<ffffffff812bb828>] intel_idle+0x98/0x170
 RSP <ffffffff81a01ea8>
---[ end trace 3a5e7c9663b4e1f2 ]---
invalid opcode: 0000 [#3] SMP
Kernel panic - not syncing: Fatal exception
RAX: ffffffff81a00010 RBX: 0000000000000008 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: ffff88002805b040
RBP: ffffffff81a01f18 R08: 0000000000000000 R09: 00000000000000c8
R10: 000000003b9aca00 R11: 00000000fffb6c80 R12: 0000000000000020
R13: 12234113f5554508 R14: 0000000000000003 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880028050000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000289560f000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a2d020)
Stack:
 ffff88002805c640 ffff880028050240 000000001dbe00c0 ffff88002805c640
<0> 0000000000000001 0000000017044e80 ffffffff81007b3f 00000000814e0cb6
<0> ffffffff81a01f18 ffff88002806d2c0 ffff88002806d3f0 0000000000000000
Call Trace:
 [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff813eccb7>] cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009e96>] cpu_idle+0xb6/0x110
 [<ffffffff814c376a>] rest_init+0x7a/0x80
 [<ffffffff81bbdf28>] start_kernel+0x41d/0x429
 [<ffffffff81bbd33a>] x86_64_start_reservations+0x125/0x129
 [<ffffffff81bc111b>] xen_start_kernel+0x582/0x586
Code: 8b 7d cc 85 c0 0f 85 b3 00 00 00 65 48 8b 34 25 08 cc 00 00 48 8b 86 38 e0 ff ff a8 08 75 25 31 d2 48 8d 86 38 e0 ff ff 48 89 d1 <0f> 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89
RIP  [<ffffffff812bb828>] intel_idle+0x98/0x170
 RSP <ffffffff81a01ea8>
---[ end trace 3a5e7c9663b4e1f2 ]---
invalid opcode: 0000 [#3] SMP
Kernel panic - not syncing: Fatal exception

last sysfs file:
Pid: 0, comm: swapper Tainted: G      D    ----------------   2.6.32-131.0.15.el6.x86_64 #1
CPU 1 Call Trace:
Modules linked in:
 [<ffffffff814dac28>] ? panic+0x78/0x143

Modules linked in: [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1

 [<ffffffff814ddb2c>] ? _spin_unlock_irqrestore+0x1c/0x20
Pid: 0, comm: swapper Tainted: G      D    ----------------   2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff812bb828>]  [<ffffffff814dec74>] ? oops_end+0xe4/0x100
 [<ffffffff812bb828>] intel_idle+0x98/0x170
RSP: e02b:ffff8807dc973e80  EFLAGS: 00010046
 [<ffffffff8100f2fb>] ? die+0x5b/0x90
RAX: ffff8807dc972010 RBX: 0000000000000004 RCX: 0000000000000000
Comment 4 Ben Howard 2011-06-01 16:28:42 EDT
root (hd0)
 Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=LABEL=_/
initrd /boot/initramfs-2.6.32-131.0.15.el6.x86_64.img

close blk: backend at /local/domain/0/backend/vbd/115/2048
close blk: backend at /local/domain/0/backend/vbd/115/2128
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-131.0.15.el6.x86_64 (mockbuild@x86-007.build.bos.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Tue May 10 15:42:40 EDT 2011
Command line: ro root=LABEL=_/
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000800000000 (usable)
DMI not present or invalid.
last_pfn = 0x800000 max_arch_pfn = 0x400000000
last_pfn = 0x100000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000100000000
init_memory_mapping: 0000000100000000-0000000800000000
RAMDISK: 01f68000 - 046d8000
No NUMA configuration found
Faking a node at 0000000000000000-0000000800000000
Bootmem setup node 0 0000000000000000-0000000800000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [00000000008b7000 -  00000000009b6fff] pages 100
(8 early reservations) ==> bootmem [0000000000 - 0800000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0008b37000 - 0008b82000]   XEN PAGETABLES ==> [0008b37000 - 0008b82000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 0001f474e4]    TEXT DATA BSS ==> [0001000000 - 0001f474e4]
  #4 [0001f68000 - 00046d8000]          RAMDISK ==> [0001f68000 - 00046d8000]
  #5 [00046d8000 - 0008b37000]   XEN START INFO ==> [00046d8000 - 0008b37000]
  #6 [0000100000 - 00008b7000]          PGTABLE ==> [0000100000 - 00008b7000]
  #7 [0008b82000 - 000c39e000]          PGTABLE ==> [0008b82000 - 000c39e000]
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00800000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00800000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 4 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 800100000 (gap: 800100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 3.1.2-128.1.10.el5 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 30 pages/cpu @ffff880028050000 s92504 r8192 d22184 u122880
pcpu-alloc: s92504 r8192 d22184 u122880 alloc=30*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Xen: using vcpu_info placement
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8271845
Policy zone: Normal
Kernel command line: ro root=LABEL=_/
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 32835652k/33554432k available (5014k kernel code, 388k absent, 718392k reserved, 6906k data, 1232k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:304
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 335544320 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
Detected 2666.760 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.52 BogoMIPS (lpj=2666760)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Unsupported number of siblings 16
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20700 entries in 82 pages
Performance Events: PEBS fmt1+, Nehalem events,
no APIC, boot with the "lapic" boot parameter to force-enable it.
no hardware sampling interrupt available.
Broken PMU hardware detected, using software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
installing Xen timer for CPU 1
SMP alternatives: switching to SMP code
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 2
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 3
CPU: Unsupported number of siblings 16
Brought up 4 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 40384k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1306960050.551:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key C9F0DFBFCE81A817
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
invalid opcode: 0000 [#1] SMP
last sysfs file:
CPU 3
Modules linked in:

Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff812bb828>]  [<ffffffff812bb828>] intel_idle+0x98/0x170
RSP: e02b:ffff8807dc979e80  EFLAGS: 00010046
RAX: ffff8807dc978010 RBX: 0000000000000004 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8807dc979fd8 RDI: ffff8800280b5040
RBP: ffff8807dc979ef0 R08: 0000000000000000 R09: 0000000000000320
R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000010
R13: 1223413dcdc8106e R14: 0000000000000002 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff8800280aa000(0000) knlGS:0000000000000000
CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process swapper (pid: 0, threadinfo ffff8807dc978000, task ffff8807dc974ac0)
Stack:
 0000000000000002 ffff8800280b0240 0000000000000001 ffff8800280b6640
<0> 0000000006ea0500 00000000000a9a47 ffffffff81007b3f 00000003814e0cb6
<0> ffff8807dc979ef0 ffff8800280c72c0 ffff8800280c7390 0000000000000000
Call Trace:
 [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff813eccb7>] cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009e96>] cpu_idle+0xb6/0x110
 [<ffffffff81007af9>] ? xen_irq_enable_direct_end+0x0/0x7
 [<ffffffff814cee35>] cpu_bringup_and_idle+0x13/0x15
Code: 8b 7d cc 85 c0 0f 85 b3 00 00 00 65 48 8b 34 25 08 cc 00 00 48 8b 86 38 e0 ff ff a8 08 75 25 31 d2 48 8d 86 38 e0 ff ff 48 89 d1 <0f> 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89
RIP  [<ffffffff812bb828>] intel_idle+0x98/0x170
 RSP <ffff8807dc979e80>
invalid opcode: 0000 [#2]
---[ end trace 7163428e87a00d47 ]---
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D    ----------------   2.6.32-131.0.15.el6.x86_64 #1
Call Trace:
 [<ffffffff814dac28>] ? panic+0x78/0x143
 [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ddb2c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff814dec74>] ? oops_end+0xe4/0x100
 [<ffffffff8100f2fb>] ? die+0x5b/0x90
 [<ffffffff814de544>] ? do_trap+0xc4/0x160
 [<ffffffff8100ceb5>] ? do_invalid_op+0x95/0xb0
 [<ffffffff812bb828>] ? intel_idle+0x98/0x170
 [<ffffffff8109d1d7>] ? tick_broadcast_oneshot_control+0xc7/0x120
 [<ffffffff8109cd05>] ? tick_notify+0x325/0x410
 [<ffffffff8100bf5b>] ? invalid_op+0x1b/0x20
Comment 5 Ben Howard 2011-06-01 16:34:54 EDT
Command line: ro root=LABEL=_/
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000800000000 (usable)
DMI not present or invalid.
last_pfn = 0x800000 max_arch_pfn = 0x400000000
last_pfn = 0x100000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000100000000
init_memory_mapping: 0000000100000000-0000000800000000
RAMDISK: 01f68000 - 046d8000
No NUMA configuration found
Faking a node at 0000000000000000-0000000800000000
Bootmem setup node 0 0000000000000000-0000000800000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [00000000008b7000 -  00000000009b6fff] pages 100
(8 early reservations) ==> bootmem [0000000000 - 0800000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0008b37000 - 0008b82000]   XEN PAGETABLES ==> [0008b37000 - 0008b82000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 0001f474e4]    TEXT DATA BSS ==> [0001000000 - 0001f474e4]
  #4 [0001f68000 - 00046d8000]          RAMDISK ==> [0001f68000 - 00046d8000]
  #5 [00046d8000 - 0008b37000]   XEN START INFO ==> [00046d8000 - 0008b37000]
  #6 [0000100000 - 00008b7000]          PGTABLE ==> [0000100000 - 00008b7000]
  #7 [0008b82000 - 000c39e000]          PGTABLE ==> [0008b82000 - 000c39e000]
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00800000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00800000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 4 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 800100000 (gap: 800100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 3.1.2-128.1.10.el5 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 30 pages/cpu @ffff880028050000 s92504 r8192 d22184 u122880
pcpu-alloc: s92504 r8192 d22184 u122880 alloc=30*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Xen: using vcpu_info placement
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8271845
Policy zone: Normal
Kernel command line: ro root=LABEL=_/
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 32835652k/33554432k available (5014k kernel code, 388k absent, 718392k reserved, 6906k data, 1232k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:304
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 335544320 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
Detected 2666.760 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.52 BogoMIPS (lpj=2666760)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Unsupported number of siblings 16
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20700 entries in 82 pages
Performance Events: PEBS fmt1+, Nehalem events,
no APIC, boot with the "lapic" boot parameter to force-enable it.
no hardware sampling interrupt available.
Broken PMU hardware detected, using software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
installing Xen timer for CPU 1
SMP alternatives: switching to SMP code
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 2
CPU: Unsupported number of siblings 16
installing Xen timer for CPU 3
CPU: Unsupported number of siblings 16
Brought up 4 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 40384k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1306959872.310:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key C9F0DFBFCE81A817
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1]
Modules linked in:
CPU 0:
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff810a4512>]  [<ffffffff810a4512>] smp_call_function_many+0x1b2/0x210
RSP: e02b:ffff8807dc96bc10  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff880028061160 RCX: 000000000000003c
RDX: 0000000000000004 RSI: 0000000000000004 RDI: ffff8807ffe7dc00
RBP: ffff8807dc96bc50 R08: ffff8807ffe7dc00 R09: 0000000000000000
R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff81b9e080 R14: ffffffff81b9e080 R15: ffffffff811594a0
FS:  0000000000000000(0000) GS:ffff880028050000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Call Trace:
 [<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
 [<ffffffff810a4592>] smp_call_function+0x22/0x30
 [<ffffffff8106f334>] on_each_cpu+0x24/0x50
 [<ffffffff8115c46f>] do_tune_cpucache+0x12f/0x630
 [<ffffffff8115cb4b>] enable_cpucache+0x3b/0xf0
 [<ffffffff814c5d3f>] setup_cpu_cache+0x22f/0x340
 [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8115ac92>] ? kmem_cache_alloc+0x182/0x190
 [<ffffffff8115d7da>] kmem_cache_create+0x3fa/0x580
 [<ffffffff81bfd1b7>] flow_cache_init+0x4a/0x1af
 [<ffffffff81bfd16d>] ? flow_cache_init+0x0/0x1af
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff81bbd884>] kernel_init+0x29d/0x2f9
 [<ffffffff8100c1ca>] child_rip+0xa/0x20
 [<ffffffff8100b393>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8100bb1d>] ? retint_restore_args+0x5/0x6
 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1]
Modules linked in:
CPU 0:
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.32-131.0.15.el6.x86_64 #1
RIP: e030:[<ffffffff810a4516>]  [<ffffffff810a4516>] smp_call_function_many+0x1b6/0x210
RSP: e02b:ffff8807dc96bc10  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff880028061160 RCX: 000000000000003c
RDX: 0000000000000004 RSI: 0000000000000004 RDI: ffff8807ffe7dc00
RBP: ffff8807dc96bc50 R08: ffff8807ffe7dc00 R09: 0000000000000000
R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff81b9e080 R14: ffffffff81b9e080 R15: ffffffff811594a0
FS:  0000000000000000(0000) GS:ffff880028050000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a25000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Call Trace:
 [<ffffffff811594a0>] ? do_ccupdate_local+0x0/0x40
 [<ffffffff810a4592>] smp_call_function+0x22/0x30
 [<ffffffff8106f334>] on_each_cpu+0x24/0x50
 [<ffffffff8115c46f>] do_tune_cpucache+0x12f/0x630
 [<ffffffff8115cb4b>] enable_cpucache+0x3b/0xf0
 [<ffffffff814c5d3f>] setup_cpu_cache+0x22f/0x340
 [<ffffffff81007b3f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8115ac92>] ? kmem_cache_alloc+0x182/0x190
 [<ffffffff8115d7da>] kmem_cache_create+0x3fa/0x580
 [<ffffffff81bfd1b7>] flow_cache_init+0x4a/0x1af
 [<ffffffff81bfd16d>] ? flow_cache_init+0x0/0x1af
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff81bbd884>] kernel_init+0x29d/0x2f9
 [<ffffffff8100c1ca>] child_rip+0xa/0x20
 [<ffffffff8100b393>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8100bb1d>] ? retint_restore_args+0x5/0x6
 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
Comment 6 Ben Howard 2011-06-01 16:35:48 EDT
The four stack traces are from four separate hosts.
Comment 7 Ben Howard 2011-06-01 18:40:45 EDT
Dom0 in all cases is RHEL 5.3 x86_64. We are observing this accross multiple vendors. 

Also, it look like the launch failure is 50%. 15% of the hosts that do launch seem to suffer some sort of death into their life cycle. 

The m2.2xlarge has more failures than the m2.4xlarge hosts.
Comment 8 Ben Howard 2011-06-01 18:46:25 EDT
Correction -- "multiple vendors" should read "multple hardware vendors".
Comment 9 wes hayutin 2011-06-01 19:44:30 EDT
I'm testing a m2.4xlarge x86_64 RHEL 6.1 and I'm having trouble recreating.
As far as I understand it the recreate is..

1. bring up a RHEL 6.1, x86_64 w/ m2.2xlarge or m2.4xlarge
2. wait roughly 60 minutes
3. 50% of those instances fail

Questions:
1. Are there any operations running on the host while its up?
2. How many guests should be started to recreate, roughly.. ?
Comment 10 Dor Laor 2011-06-02 04:00:51 EDT
Can you please provide the output of cat /proc/cpuinfo on the guest?
Comment 11 Chris Morgan 2011-06-02 07:10:12 EDT
Created attachment 502501 [details]
uname -a for instance in apac-sing

Uname -a output from m2.4xlarge in apac-sing.
Comment 12 Chris Morgan 2011-06-02 07:12:38 EDT
Created attachment 502502 [details]
/proc/cpuinfo from apac-sing

This is output from a running m2.4xlarge instance in apac-sing.
Comment 13 Chris Morgan 2011-06-02 07:15:11 EDT
Much like Wes, I cannot repeat the issue with one instance, but given my usage, this falls into the stats supplied by Ben.  I can try other instances, if needed, but the inconsistency is frustrating.  It should also be noted that the error appears not to occur with RHEL 5.5, 5.6, and 6.0.
Comment 14 Chris Morgan 2011-06-02 07:25:41 EDT
Wanted to add that my response time is slow in the instance, but that could be more of a factor of connecting from Raleigh to Singapore.  The instance is still going after 22 minutes of uptime.
Comment 15 wes hayutin 2011-06-02 09:00:07 EDT
Created attachment 502528 [details]
cpu proc, uptime data for 10 instances across all regions

FYI.. these instances are also m2.4xlarge.. I'll bring up 10 m2.2xlarge right now to test
Comment 16 wes hayutin 2011-06-02 09:03:09 EDT
Another Question:
If the instances does soft crash.. is it still listed in the ec2 webui console after the crash?
Comment 17 Laszlo Ersek 2011-06-02 09:45:24 EDT
Chris,

(In reply to comment #13)
> Much like Wes, I cannot repeat the issue with one instance, but given my usage,
> this falls into the stats supplied by Ben.  I can try other instances, if
> needed, but the inconsistency is frustrating.  It should also be noted that the
> error appears not to occur with RHEL 5.5, 5.6, and 6.0.

could you please elaborate a bit more on the last sentence? Are you saying that this is a guest regression from 6.0 to 6.1? Or do you mean that you could not reproduce the problem either with 6.0 or with 6.1? Thanks!
Comment 18 Chris Morgan 2011-06-02 09:54:21 EDT
Laszlo,

Amazon told us that they only experience the problem themselves with 6.1 images.  None of the 5.5, 5.6 or 6.0 images have the issue.  So it would imply that some sort of incompatibility began between 6.0 and 6.1 errata.
Comment 19 wes hayutin 2011-06-02 11:26:20 EDT
Created attachment 502568 [details]
eu-west x86_64 m2.2xlarge crash console

Found one that did indeed crash.. console log attached
Comment 20 wes hayutin 2011-06-02 11:28:17 EDT
Info for the above crash

AMI: RHEL-6.1-Starter-EBS-x86_64-1-Hourly (ami-4efcca3a)	Zone: eu-west-1a
Security Groups: default	Type: m2.2xlarge
Status: running	Owner: 673500695950
VPC ID: -	Subnet ID: -
Source/Dest. Check:	Virtualization: paravirtual
Placement Group:	Reservation: r-6ae7481c
RAM Disk ID: -	Platform: -	Kernel ID: aki-a90a3ddd
Monitoring: basic	AMI Launch Index: 0
Elastic IP: -	Root Device:
Root Device Type: ebs	Tenancy: default
Lifecycle: normal
Block Devices:
sda
Public DNS: ec2-79-125-54-178.eu-west-1.compute.amazonaws.com
Private DNS: ip-10-230-55-234.eu-west-1.compute.internal
Private IP Address: 10.230.55.234
Launch Time: 2011-06-02 09:49 EDT
Comment 21 wes hayutin 2011-06-02 11:36:32 EDT
Created attachment 502574 [details]
ap-southeast-x86_64_rhel61_m2.2xlargeCRASH
Comment 22 wes hayutin 2011-06-02 11:37:16 EDT
RHEL-6.1-Starter-EBS-x86_64-1-Hourly (ami-aefe87fc)	Zone: ap-southeast-1b
Security Groups: default	Type: m2.2xlarge
Status: running	Owner: 673500695950
VPC ID: -	Subnet ID: -
Source/Dest. Check:	Virtualization: paravirtual
Placement Group:	Reservation: r-9718b5c2
RAM Disk ID: -	Platform: -	Kernel ID: aki-82235ad0
Monitoring: basic	AMI Launch Index: 0
Elastic IP: -	Root Device:
Root Device Type: ebs	Tenancy: default
Lifecycle: normal
Block Devices:
sda
Public DNS: ec2-175-41-173-14.ap-southeast-1.compute.amazonaws.com
Private DNS: ip-10-130-227-206.ap-southeast-1.compute.internal
Private IP Address: 10.130.227.206
Launch Time: 2011-06-02 09:16 EDT
Comment 23 wes hayutin 2011-06-02 11:40:42 EDT
two out of the ten instances across the regions have crash with in two hours.
Comment 24 wes hayutin 2011-06-02 11:43:16 EDT
actually to clarify...

I have two instances w/ m2.4xlarge and m2.2xlarge in each region.
one crash in eu-west, one crash in ap-southeast

m2.4xlarge 0/10 crashed after 10+ hours
m2.2xlarge 2/10 crashed after 2+ hours
Comment 25 Andrew Jones 2011-06-02 12:04:12 EDT
OK, I looked at the boot logs, traces, and the cpuinfo and have a *guess* at what the problem could be. The cpuinfo given to me, which I assume is a from a working guest on the problem host, has nonstop_tsc set in it. This is a risky feature to have on a guest. Furthermore, the 6.0 -> 6.1 regression that Laszlo confirmed with Chris adds another clue, because the Intel cpuidle driver was added during that development phase. The boot messages generally showed the crash shortly after printing

cpuidle: using governor ladder
cpuidle: using governor menu

and traces appear to involve that driver (intel_idle is on the stack) and that's tied to nonstop_tsc because intel_idle_cpuidle_devices_init() introduced with that patchset does NOT set the tsc unstable if nonstop_tsc is present.

I think the best way to proceed is to try a trial guest kernel with nonstop_tsc masked out. I can create one now.
Comment 26 Laszlo Ersek 2011-06-02 12:15:06 EDT
$ addr2line -pife \
    /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux \
    <<< ffffffff812bb828
__monitor at /usr/src/debug/kernel-2.6.32-131.0.15.el6/linux-2.6.32-131.0.15.el6.x86_64/arch/x86/include/asm/processor.h:751
 (inlined by) intel_idle at /usr/src/debug/kernel-2.6.32-131.0.15.el6/linux-2.6.32-131.0.15.el6.x86_64/drivers/idle/intel_idle.c:192


[drivers/idle/intel_idle.c]

168  static int intel_idle(struct cpuidle_device *dev, struct cpuidle_state *state)
169  {
170    unsigned long ecx = 1; /* break on interrupt flag */
171    unsigned long eax = (unsigned long)cpuidle_get_statedata(state);
172    unsigned int cstate;
173    ktime_t kt_before, kt_after;
174    s64 usec_delta;
175    int cpu = smp_processor_id();
176  
177    cstate = (((eax) >> MWAIT_SUBSTATE_SIZE) & MWAIT_CSTATE_MASK) + 1;
178  
179    local_irq_disable();
180  
181    if (!(lapic_timer_reliable_states & (1 << (cstate))))
182      clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, &cpu);
183  
184    kt_before = ktime_get_real();
185  
186    stop_critical_timings();
187  #ifndef MODULE
188    trace_power_start(POWER_CSTATE, (eax >> 4) + 1, cpu);
189  #endif
190    if (!need_resched()) {
191  
192      __monitor((void *)&current_thread_info()->flags, 0, 0); /* HERE */
193      smp_mb();
194      if (!need_resched())
195        __mwait(eax, ecx);
196    }
197  
198    start_critical_timings();
199  
200    kt_after = ktime_get_real();
201    usec_delta = ktime_to_us(ktime_sub(kt_after, kt_before));
202  
203    local_irq_enable();
204  
205    if (!(lapic_timer_reliable_states & (1 << (cstate))))
206      clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, &cpu);
207  
208    return usec_delta;
209  }

[arch/x86/include/asm/processor.h]

747  static inline void __monitor(const void *eax, unsigned long ecx,
748             unsigned long edx)
749  {
750    /* "monitor %eax, %ecx, %edx;" */
751    asm volatile(".byte 0x0f, 0x01, 0xc8;"
752           :: "a" (eax), "c" (ecx), "d"(edx));
753  }
Comment 30 wes hayutin 2011-06-02 17:57:39 EDT
[root@ip-10-230-53-220 ~]# uname -a
Linux ip-10-230-53-220 2.6.32-131.0.15.el6mask_nonstop_tsc.x86_64 #1 SMP Thu Jun 2 13:10:15 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@ip-10-230-53-220 ~]# cat /proc/cpuinfo |grep nonstop_tsc
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm


patch is not good.. dont distribute.
Comment 32 Andrew Jones 2011-06-03 01:32:31 EDT
(In reply to comment #30)
> [root@ip-10-230-53-220 ~]# uname -a
> Linux ip-10-230-53-220 2.6.32-131.0.15.el6mask_nonstop_tsc.x86_64 #1 SMP Thu
> Jun 2 13:10:15 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> [root@ip-10-230-53-220 ~]# cat /proc/cpuinfo |grep nonstop_tsc
> flags  : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht
> syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni
> ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm

Argh... This is disappointing. I don't know why it didn't work... I'll find a machine with nonstop_tsc to fix and test it on before spinning another rpm.
Comment 35 Andrew Jones 2011-06-03 02:12:33 EDT
Created attachment 502732 [details]
mask nonstop_tsc
Comment 36 Andrew Jones 2011-06-03 02:39:01 EDT
Ah, here's the issue. Sigh, I shouldn't have looked at this code closer when I was grepping for NONSTOP_TSC

in early_init_intel() we have

        if (c->x86_power & (1 << 8)) {
                set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
                set_cpu_cap(c, X86_FEATURE_NONSTOP_TSC);
                if (!check_tsc_unstable())
                        sched_clock_stable = 1;
        }

also in early_init_amd() we have a corresponding

        if (c->x86_power & (1 << 8)) {
                set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
                set_cpu_cap(c, X86_FEATURE_NONSTOP_TSC);
        }

so it's getting flipped back on again. I should probably just clear all of x86_power, which is 8000_0007 edx. The patch I tried before only cleared the one NONSTOP_TSC bit, I'll roll another one that zeros out all of edx.
Comment 37 Andrew Jones 2011-06-03 02:44:11 EDT
Created attachment 502735 [details]
zero all x86_power feature bits
Comment 41 wes hayutin 2011-06-03 11:12:28 EDT
[root@ip-10-58-159-28 ~]# cat /root/BEFORE_INSTALL | grep nonstop_tsc
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
[root@ip-10-58-159-28 ~]# 
[root@ip-10-58-159-28 ~]# uname -a
Linux ip-10-58-159-28 2.6.32-131.0.15.el6zero_x86_power.x86_64 #1 SMP Fri Jun 3 03:09:47 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@ip-10-58-159-28 ~]# 
[root@ip-10-58-159-28 ~]# cat /proc/cpuinfo | grep nonstop_tsc
Comment 47 wes hayutin 2011-06-04 08:47:54 EDT
OK.. I'm going to give my response in two parts.

First off.. yes.. the patch fixes the issue. After 15 hours 10/10 of the x86_64 6.1 eu-west-1 ami's are still running. So.. yes the patch is a PASS

Just to note 3 ami's crashed before I could get the patch installed, so I'm in a region that can reproduce the issue.

I am run kernel-qe beaker tier1/2 tests on 2 of the 10 boxes with the patch. I will post the results I have thus far, but the tests are still running and I do see some failures.  Something to be checked by the kernel-qe team for sure.
Comment 48 wes hayutin 2011-06-04 08:56:16 EDT
Created attachment 502982 [details]
beaker tier1/2 kernel qe tests.. tests still running no complete
Comment 49 wes hayutin 2011-06-04 14:30:02 EDT
Comment on attachment 502982 [details]
beaker tier1/2 kernel qe tests.. tests still running no complete

have full logs.. tests finished
Comment 50 wes hayutin 2011-06-04 14:35:16 EDT
Created attachment 503004 [details]
tier1 & tier2 kernel qe tests
Comment 51 Andrew Jones 2011-06-05 02:08:50 EDT
(In reply to comment #47)
> First off.. yes.. the patch fixes the issue. After 15 hours 10/10 of the x86_64
> 6.1 eu-west-1 ami's are still running. So.. yes the patch is a PASS
> 
> Just to note 3 ami's crashed before I could get the patch installed, so I'm in
> a region that can reproduce the issue.
>

This is good news.

> I am run kernel-qe beaker tier1/2 tests on 2 of the 10 boxes with the patch. I
> will post the results I have thus far, but the tests are still running and I do
> see some failures.  Something to be checked by the kernel-qe team for sure.

This is bad news. It sounds like I still shouldn't post the patch until we've determined whether or not it introduced these other test failures. Although I don't imagine these tests were run before on this exact setup, since these setups weren't booting before, so there's a good chance these are different issues 

We should do the following

a) run the test kernel through all the regression tests here in house on our test machines and ensure all PASS (or determine the patch introduces regressions)

b) investigate the failures (get debug logs, etc) and use Igor's test machines for experiments - we would do this work under separate bugzillas for each issue.
Comment 54 Chris Morgan 2011-06-05 07:04:41 EDT
I received an update from Amazon yesterday:

"Sorry, I thought I had sent a confirmation. The patch appears to fix the problem.

Thanks,
Ben"
Comment 56 wes hayutin 2011-06-06 09:49:53 EDT
I just tested the attachment.
1. download the attachment
 1003  bzip2 -d tests.tar.bz2 
 1004  ls -ltra
 1005  tar -xvf tests.tar 

should get two dirs.. 


drwxr-xr-x. 24 whayutin whayutin      4096 Jun  4 14:25 Tests_x86_64_01
drwxr-xr-x. 24 whayutin whayutin      4096 Jun  4 14:30 Tests_x86_6402

[whayutin@localhost Downloads]$ tar -tvf tests.tar  | more
drwxr-xr-x whayutin/whayutin 0 2011-06-04 14:25 Tests_x86_64_01/
drwxr-xr-x whayutin/whayutin 0 2011-06-04 14:24 Tests_x86_64_01/operational/
-rw------- whayutin/whayutin 715 2011-06-04 14:24 Tests_x86_64_01/operational/tm
p.klT1c1
-rw-r--r-- whayutin/whayutin 113 2011-06-04 14:24 Tests_x86_64_01/operational/op
erational.error.log
-rw-r--r-- whayutin/whayutin   0 2011-06-04 14:24 Tests_x86_64_01/operational/dm
esg_operational.log
-rw-r--r-- whayutin/whayutin 336 2011-06-04 14:24 Tests_x86_64_01/operatio
Comment 57 Andrew Jones 2011-06-06 11:35:17 EDT
Now that we know turning off nonstop_tsc resolves the issue, I've done another round of code analysis to see why, and if the patch I proposed would introduce other problems or not.

I believe the 'why' is because with nonstop_tsc turned off, sched_clock_stable stays off. With sched_clock_stable turned on the scheduler assumes sched_clock() is synchronized between processors. However, it's not synchronized with RHEL6 xenpv guests because it's been overridden with xen_sched_clock(), which attempts to measure unstolen time (which is different per vcpu). Upstream dropped this idea about a year ago with the following patch

commit 8a22b9996b001c88f2bfb54c6de6a05fc39e177a
Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Date:   Mon Jul 12 11:49:59 2010 -0700

    xen: drop xen_sched_clock in favour of using plain wallclock time

Likely the correct fix for this issue is to backport this patch, rather than to mask the cpuid features. If Fedora 15 instances didn't have problems running on the same hosts, then that would be some confirmation of this theory.

OTOH, the patch I've already created looks to be safe from the second part of my analysis, and it's a conservative/defensive patch to have in place. A clear advantage to using it over a backport of the patch above is that we don't have to switch sched_clock() now at the last minute, which could introduce regressions. Masking cpuid bits of unneeded features is in general safe, and it reduces potential for other regressions as well.

So, I guess I've convinced myself to post the masking patch for now. I'll open a new bug to consider what to do during 6.2 development.
Comment 59 Andrew Jones 2011-06-06 13:08:31 EDT
(In reply to comment #57)
> commit 8a22b9996b001c88f2bfb54c6de6a05fc39e177a
> Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> Date:   Mon Jul 12 11:49:59 2010 -0700
> 
>     xen: drop xen_sched_clock in favour of using plain wallclock time
> 

The mail thread related to this patch is here
http://lists.xensource.com/archives/html/xen-devel/2010-07/msg00738.html
Comment 67 wes hayutin 2011-06-09 08:42:31 EDT
If there are any rhts/beaker based tests that can specifically test this issue, please let cloud-qe know.

Thank you!
Comment 68 Aristeu Rozanski 2011-06-09 10:10:12 EDT
Patch(es) available on kernel-2.6.32-156.el6
Comment 72 Tomas Capek 2011-06-17 03:52:26 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Xen guests cannot make use of all CPU features, and in some cases they are even risky to be advertised. One such feature is CONSTANT_TSC. This feature prevents the TSC (Time Stamp Counter) from being marked as unstable, which allows the sched_clock_stable option to be enabled. Having the sched_clock_stable option enabled is problematic for Xen PV guests because the sched_clock() function has been overridden with the xen_sched_clock() function, which is not synchronized between virtual CPUs. This update provides a patch, which sets all x86_power features to 0 as a preventive measure against other potentially dangerous assumptions the kernel could make based on the features, fixing this issue.
Comment 73 wes hayutin 2011-06-21 14:06:16 EDT
verified per.. https://bugzilla.redhat.com/show_bug.cgi?id=710609
Comment 74 errata-xmlrpc 2011-12-06 08:08:46 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2011-1530.html

Note You need to log in before you can comment on or make changes to this bug.