This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Does any other guest boot on this host? Can you get a stack with 

/usr/lib64/xen/bin/xenctx -s <rhel6-kernel-symfile> <domain#> [cpu#]

Yes, these Xen Hosts are running many HVM guests.

# /usr/lib64/xen/bin/xenctx -s symvers-2.6.32-19.el6.x86_64.gz 63
rip: ffffffff81038bf6 
rsp: ffffffff81705df0
rax: 00000001	rbx: 00000001	rcx: 00000100	rdx: 00000016
rsi: ffffffff81705e50	rdi: ffffffff81705e54	rbp: ffffffff81705df8
 r8: ffffffff81705e4c	 r9: ffffffff81705e48	r10: ffff88003e5b2748	r11: 00000070
r12: da7a8dcc	r13: ffffffff81705e54	r14: ffffffff81705e50	r15: ffffffff81705e4c
 cs: 00000010	 ds: 00000000	 fs: 00000000	 gs: 00000000

I'm not very familiar with symbol tables, I think this is what you want. 

I think that the host's 'xm info' may be useful as well. These systems are getting a little old, so I'm ok if the answer is "upgrade your Hypervisor". I just want to make sure that is the case.

host                   : ####
release                : 2.6.18-128.1.6.el5xen
version                : #1 SMP Wed Apr 1 09:53:14 EDT 2009
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2000
hw_caps                : bfebfbff:28100800:00000000:00000140:009ce3bd:00000000:00000001
total_memory           : 12279
free_memory            : 2620
node_to_cpu            : node0:0-7
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-128.1.6.el5
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Wed Apr  1 09:00:24 EDT 2009
xend_config_format     : 2

Actually I meant System.map-2.6.32-19.el6.x86_64, sorry for that confusion.

You're correct though that the answer is likely just to update your HV. The RIP you captured is pointing at a cpuid call, and we're far enough along in the boot that we're most likely at identify_boot_cpu. I'm guessing RHEL6 is trying to probe or enable some of the hardware on your system that isn't supported, or hidden, by the old HV.

Let's try to get another stack, since I'm curious what exactly we're choking on. Then you can update to 5.5 and test again, which will very likely work.

I find it interesting that I see something different between when I attach to the console during the boot sequence, and when I don't. 

If I create the VM with out attaching to the console, I get the following.
# /usr/lib64/xen/bin/xenctx -s System.map-2.6.32-19.el6.x86_64 67
rip: ffffffff81317977 io_serial_out+0x17
rsp: ffffffff81705d78
rax: 00000030	rbx: ffffffff81c05460	rcx: 00000000	rdx: 000003f8
rsi: 00000000	rdi: ffffffff81c05460	rbp: ffffffff81705d78
 r8: ffffffff81892b80	 r9: 00000000	r10: 0000007c	r11: 000024c0
r12: 00000030	r13: 0000004e	r14: 0000000c	r15: ffffffff81318000
 cs: 00000010	 ds: 00000000	 fs: 00000000	 gs: 00000000

Stack:
failed to map page.

If I create the VM with the '-c' flag to attach to the console, after it crashes, I exit the console, and check, and get the following.
# /usr/lib64/xen/bin/xenctx -s System.map-2.6.32-19.el6.x86_64 68
rip: ffffffff81038bf6 native_cpuid+0x16
rsp: ffffffff81705df0
rax: 00000001	rbx: 00000001	rcx: 00000100	rdx: 00000010
rsi: ffffffff81705e50	rdi: ffffffff81705e54	rbp: ffffffff81705df8
 r8: ffffffff81705e4c	 r9: ffffffff81705e48	r10: ffff88003e5b2748	r11: 00000070
r12: 00aa9c71	r13: ffffffff81705e54	r14: ffffffff81705e50	r15: ffffffff81705e4c
 cs: 00000010	 ds: 00000000	 fs: 00000000	 gs: 00000000

Stack:
failed to map page.

I'm not quite sure what the output is supposed to look like, but this out put doesn't look right either.

hmm, I was hoping that passing in the wrong file, as was done for comment 4, was the reason we didn't have a backtrace. It looks like even with the correct file we don't get any more information on how we got here.

Possibly you ran the xenctx command too soon for the case where you didn't try connecting the console? Or, how many vcpus does the guest have? If it's more than one than you should run the xenctx command once per vcpu, where the last parameter of the command is the vcpu id, e.g. 0,1,2... If one vcpu was doing io while the other attempted a cpuid that triggered an unsupported code path and killed the guest, then that would explain the different outputs (i.e. guest boot (1) had serial io on vcpu 0 and then try (2) had the cpuid).

Only 1 vcpu. With the console vs. non-console output, it looks like I was a little impatient. I let it sit a lot longer this time, and I get the native_cpuid rip on the first line. 

I'll just have to wait till our next iteration of the production environment is ready. It does run RHEL6 in testing. 

Thanks for your time.

This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release. It has
been denied for the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **

Setting to 6.1. Hopefully we'll get more information for this bug, otherwise we'll have to close it as not-enough-info.

Drew

Haven't heard anything new on this bug in quite a while, so I'll close it as not-enough-info. If more information becomes available, then please reopen.

Sorry I couldn't give you more information. With the release of RHELv6 we aren't interested in solving this problem on our existing 5.3 Xen servers.